Multi-dimensional collaborative AI training processor benchmark evaluation device and method

Abstract

The invention discloses a multi-dimensional collaborative AI training processor benchmark evaluation device and method. According to end-to-end performance evaluation, TTA and throughput indexes are obtained through a standardized training process; the operator support degree evaluation adopts a dynamic parameter generation technology to evaluate the speed-up ratio and the floating point performance utilization rate of calculation-intensive and memory-access-intensive operators; the energy consumption evaluation calculates a power consumption delay product based on steady-state power consumption monitoring. And finally, integrating multi-dimensional indexes through an analytic hierarchy process model and a fuzzy theory, realizing level mapping by adopting a four-section membership function, and finishing multi-level aggregation evaluation in combination with matrix operation. According to the method, three dimensions of performance, operator support degree and energy efficiency are creatively integrated, the problems of single evaluation index and high environment dependence of the heterogeneous AI processor are solved through automatic container deployment, dynamic parameter generation and steady-state power consumption monitoring technologies, and a comprehensive and quantifiable evaluation basis is provided for processor model selection.

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic digital data processing, and further relates to a multi-dimensional collaborative AI training processor benchmark evaluation device and method in the technical field of computer AI processor testing. The present invention can be used for various AI processor platforms, and examines the adaptation and optimization degree of AI processors through representative models and core operators in various fields of deep learning. Background Art

[0002] In recent years, the vigorous development of deep learning has largely benefited from the continuous innovation of AI processor architectures and computing power. In the field of artificial intelligence, high-performance acceleration chips for training, as the core computing power foundation, are regarded as the key link to achieve technological autonomy and industrial upgrading. Benchmark evaluation is the core method for evaluating the performance of AI training processors. The existing benchmark evaluations of deep learning AI processors can generally be divided into the following three categories:

[0003] The first category uses end-to-end training as the benchmark, and trains on real tasks using public datasets and complete neural network models to evaluate the overall performance of AI processors. A typical representative is the paper "MLPerf Training Benchmark" published by Mattson et al. in "Proceedings of Machine Learning and Systems" in 2020, which proposed a complete end-to-end evaluation framework. This evaluation framework uses seven typical neural network architectures such as ResNet-50, Transformer, and BERT as benchmark models, uses standardized datasets such as ImageNet, COCO, and LibriSpeech, and fully realizes the entire training process from data loading, forward propagation, backward propagation to parameter update, and mainly measures key performance indicators such as the training time required to reach the target accuracy and the system throughput.

[0004] The second category is based on the operator level and mainly focuses on the performance of individual basic operators. Represented by the open-source tool "DeepBench: Benchmarking Deep Learning Operations on Different Hardware" published by Baidu Research Institute on its GitHub repository in 2017, it systematically designs performance test schemes for three core operators: general matrix multiplication (GEMM), convolution (CONV), and recurrent neural network (RNN), covering three numerical precisions of FP32, FP16, and INT8, constructs a complete parameter space from batch size 16 to 4096 and input feature map size 32×32 to 1024×1024, and focuses on measuring performance indicators such as the end-to-end calculation latency and peak floating-point operation throughput of the operators.

[0005] The third category is based on energy consumption as a benchmark to examine the energy efficiency of AI processors in processing various AI training tasks. The paper "EAIBench: An Energy Efficiency Benchmark for AI Training" by Zhang et al. published in the "IEEE International Symposium on Performance Analysis of Systems and Software" in 2019. This benchmark innovatively collects the power consumption data of the processor core, storage unit, and interconnect bus in real-time through a board-level current sensor with an I2C interface, establishes an energy efficiency evaluation model based on the power consumption-performance trade-off, and defines the Energy-Delay Product (EDP) as the core energy efficiency indicator.

[0006] The above technical solutions have formed the following key problems during implementation. The fundamental problem is that the three types of benchmarks adopt completely separate evaluation dimensions: the end-to-end benchmark only focuses on the overall training efficiency, the operator benchmark isolates the test of the computing unit, and the energy efficiency benchmark measures the power consumption characteristics separately. This fragmentation makes it impossible to comprehensively evaluate the actual effectiveness of AI processors. The specific manifestations are as follows: (1) The overall evaluation method of the end-to-end benchmark "MLPerf Training Benchmark" (Mattson et al., 2020) makes the performance of the computing unit unobservable. It mixes the testing of computationally intensive layers and memory-intensive layers, making the optimization direction of AI processors ambiguous; (2) The vacuum test environment of the operator benchmark "DeepBench" (Baidu Research Institute, 2017) causes system-level performance distortion. Although it details the performance of a single operator, it ignores the system-level influencing factors in the real training scenario; (3) The energy efficiency benchmark "EAIBench" (Zhang et al., 2019) lacks co-design with the other two types of benchmarks, resulting in difficulty in cross-analyzing the computing performance and energy consumption indicators.

[0007] More critically, the existing benchmarks are deeply bound to AI processor architectures such as Turing and Ampere: "MLPerf Training Benchmark" relies on the NCCL communication library and cuDNN acceleration library, "DeepBench" is implemented based on the CUDA core instruction set, and "EAIBench" uses the NVML power management interface. This architecture dependence makes these benchmarks unable to be directly applied to heterogeneous platforms with different computing architectures. Summary of the Invention

[0008] The objective of the present invention is to address the deficiencies of the above-mentioned existing technologies by proposing a multi-dimensional collaborative AI training processor benchmark evaluation device and method, aiming to solve two key technical problems existing in the existing benchmark evaluation system: First, the end-to-end training benchmark, operator performance benchmark, and energy efficiency benchmark are mutually disjointed and cannot provide a comprehensive evaluation for multi-index collaborative optimization; Second, the existing benchmarks deeply rely on AI processor architectures such as Turing and Ampere and cannot be directly adapted to architectures such as Da Vinci and ASIC.

[0009] The idea for achieving the objective of the present invention is that the evaluation system of the present invention constructs a multi-dimensional evaluation framework through four dimensions: end-to-end performance evaluation, operator support degree evaluation, energy consumption evaluation, and comprehensive evaluation. The specific implementation path is as follows: (1) End-to-end performance evaluation: On a standard data set, by recording the cumulative time for the model to reach a preset accuracy threshold and the sample processing rate during the stable training phase, the comprehensive performance of the hardware in real training tasks is evaluated. This evaluation is based on two core indicators, TTA and throughput, covering the fields of computer vision, natural language processing, and recommendation systems, and comprehensively considering three types of loads: dense computing, sparse computing, and hybrid computing. (2) Operator support degree evaluation: For five core operators such as convolution, fully connected, and multi-head attention, the peak floating-point utilization rate and speedup ratio are calculated. Through random parameter generation and legality verification, covering calculation modes from dense matrix multiplication to sparse look-up tables, the performance bottlenecks of the hardware in compute-intensive and memory-intensive operators are identified. (3) Energy consumption evaluation: The PDP indicator is used to synchronously collect power consumption and execution time data during the second training cycle, calculate the power-delay product per training step, and quantify the potential of the hardware in green computing. (4) Multi-dimensional comprehensive evaluation: By fusing the data from the first three evaluations, a weighted scoring model for end-to-end performance, operator efficiency, and energy consumption is constructed, and a radar chart is generated to intuitively compare the advantages and disadvantages of different hardware. The evaluation method of the present invention realizes the organic unity of key indicators in three dimensions through comprehensive evaluation by establishing a collaborative evaluation mechanism for end-to-end training, operator performance, and system energy efficiency, effectively solving the problem of disjointed dimensions in traditional benchmark evaluation. The evaluation system of the present invention supports multi-operating system environments and is adapted to heterogeneous hardware such as X86, ARM architectures, GPUs, and NPUs. Through containerization technology, the consistency of deep learning framework versions and dependent libraries is ensured, eliminating the interference of system environment differences on evaluation results.

[0010] The fuzzy comprehensive evaluation device for the AI training system proposed by the present invention includes an end-to-end evaluation module, an operator support degree evaluation module, an energy consumption evaluation module, and a comprehensive evaluation module; among which:

[0011] The end-to-end evaluation module is used to fix the random seed for the load to be tested, perform data standardization preprocessing, and generate a training set and a test set; configure standardized training parameters and TTA thresholds for the load, train each epoch, calculate the throughput metric, and stop training when the accuracy of the test set reaches the TTA threshold. All the training times are combined to form the TTA metric.

[0012] The operator support degree evaluation module is used to dynamically generate random test parameters for key operators of compute-intensive and memory-intensive types. After performing warm-up iterations for each valid configuration, it conducts formal tests and records the median execution time; calculates two core metrics: the average speedup ratio and the peak floating-point performance utilization.

[0013] The energy consumption evaluation module uses the steady-state power consumption monitoring method to start power consumption acquisition when the load training enters the second epoch, and records the power consumption data in real time through the hardware interface until the epoch ends; in the data processing stage, after removing the transitional data in the initial and end stages to eliminate noise interference, it calculates the average power consumption in the steady state to obtain the power-delay product of the core energy efficiency evaluation metric.

[0014] The comprehensive evaluation module is used to construct a three-layer analytic hierarchy model based on the three criterion dimensions of end-to-end performance, operators, and energy consumption; uses the speedup ratio of each index of the processor to be tested relative to the benchmark processor as the input, and realizes standardized mapping through a four-segment membership function to obtain the level of comprehensive evaluation; automatically matches the piecewise linear conversion strategy for index data of different dimensions, constructs a three-dimensional scoring matrix M to store the membership degree scores of each index, and then realizes hierarchical aggregation through matrix multiplication; uses the weighted average method for defuzzification, calculates the weighted sum of the scores of each level and the representative values of the corresponding levels, and takes the maximum value as the final evaluation level. Outputs a comprehensive performance score containing multi-dimensional indexes.

[0015] The specific steps of the multi-dimensional collaborative AI training processor benchmark evaluation method of the present invention are as follows:

[0016] Step 1, deploy a containerized adaptation environment:

[0017] Step 1.1, identify the processor architecture type and dynamically load the corresponding driver interface;

[0018] Step 1.2, retrieve the container image repository based on the architecture characteristics, intelligently match the benchmark container image of the load to be tested optimized for the processor architecture, and inject platform-specific environment variables and runtime configurations;

[0019] Step 1.3, after verifying the integrity of the hardware acceleration function in the isolated container environment, inject the load to be tested;

[0020] Step 2, perform end-to-end performance evaluation:

[0021] The end-to-end evaluation module fixes a random seed for the load to be tested. After performing data standardization preprocessing, it generates a training set and a test set; configures standardized training parameters and TTA thresholds for the load, trains each epoch and calculates the throughput metric, and stops training when the accuracy of the test set reaches the TTA threshold. All the training times are combined to form the TTA metric.

[0022] Step 3, implement the operator support degree evaluation:

[0023] The operator support degree evaluation module dynamically generates random test parameters for key operators that are computationally intensive and memory access intensive. After performing warm-up iterations for each valid configuration, it conducts formal tests and records the median execution time; calculates two core metrics: the average speedup and the peak floating-point performance utilization.

[0024] Step 4, perform energy consumption evaluation:

[0025] The energy consumption evaluation module uses the steady-state power consumption monitoring method. It starts power consumption acquisition when the load training enters the second epoch and records the power consumption data in real time through the hardware interface until the epoch ends; in the data processing stage, after removing the transitional data in the initial and ending stages to eliminate noise interference, it calculates the average power consumption in the steady state to obtain the power-delay product, which is the core energy efficiency evaluation index.

[0026] Step 5, conduct comprehensive evaluation and analysis:

[0027] The comprehensive evaluation module constructs a three-layer hierarchical analysis model based on the three criterion dimensions of end-to-end performance, operators, and energy consumption; uses the speedup of each index of the processor to be tested relative to the benchmark processor as the input, and realizes standardized mapping through a four-segment membership function to obtain the level of comprehensive evaluation; automatically matches the piecewise linear conversion strategy for index data with different dimensions, constructs a three-dimensional scoring matrix M to store the membership degree scores of each index, and then realizes hierarchical aggregation through matrix multiplication; uses the weighted average method for defuzzification, calculates the weighted sum of the scores of each level and the representative values of the corresponding levels, and takes the maximum value as the final evaluation level.

[0028] Furthermore, the container image repository includes differentiated software stacks pre-set for different computing architectures; for the GPU architecture, it provides a deep learning framework and an acceleration library collection that natively support the CUDA computing ecosystem; for the NPU and other dedicated acceleration architectures, it provides a software stack with the same function implemented through secondary compilation of the underlying instruction set, and integrates the dedicated operator library and runtime components of the corresponding computing platform.

[0029] Further, the optimized load to be tested is implemented based on the GPU architecture with a standard implementation aligned with the native interfaces of mainstream deep learning frameworks, and operator mapping is completed based on the default precision mode; for NPUs and other heterogeneous computing architectures, equivalent functions to the native interfaces of mainstream deep learning frameworks are implemented through their programming specifications, and a mixed-precision strategy is adopted according to the hardware characteristics.

[0030] Further, the steps for dynamically generating random test parameters, performing warm-up iterations for each valid configuration, and then conducting formal tests and recording the median execution time are as follows:

[0031] First step, based on the computational graph topological features of the target operator, randomly generate a multi-dimensional parameter combination including data shape and computational topology parameters; among them, the data shape includes tensor dimensions, data types, and batch sizes; the computational topology parameters include input dimensions, convolution kernel sizes, and sliding strides.

[0032] Second step, perform a legality check on the parameter combination according to the video memory capacity threshold of the current processor, and eliminate invalid configurations that exceed the hardware resource limits.

[0033] Third step, generate a set of test parameters that meet the video memory occupancy threshold through a uniform distribution method.

[0034] Fourth step, perform 10 warm-up iterations for each valid configuration to eliminate initialization overhead.

[0035] Fifth step, perform 50 formal tests in a stable state, and record the effective execution time of each single iteration after each iteration.

[0036] Sixth step, select the median value of the statistical distribution feature set as the benchmark test result.

[0037] Further, the calculation of the average speedup ratio and peak floating-point performance utilization rate metrics, where the average speedup ratio is: the execution time of the operator on the source processor divided by the operator time on the current processor; the peak floating-point performance utilization rate is obtained by the following formula:

[0038] F_util = F_theoretical / (Π × T_avg)

[0039] where F_util represents the peak floating-point performance utilization rate, F_theoretical represents the theoretical floating-point operation count, which is determined according to the operator algorithm logic and input data dimensions, Π represents the peak floating-point performance of the processor determined according to the processor hardware specification parameters, and T_avg represents the average execution time of the operator obtained from the test results, with the unit of seconds.

[0040] Further, the steps of the steady-state power consumption monitoring method are as follows:

[0041] In the first step, when the load training enters the second epoch, power consumption acquisition starts and continues until the end of the current training cycle;

[0042] In the second step, the original power consumption data is subjected to time-domain alignment and noise filtering, and the transitional data in the first 10% duration of the start phase and the last 10% duration of the end phase of the training cycle is removed;

[0043] In the third step, the average power consumption P_avg is calculated within the steady-state interval, unit: watt;

[0044] In the fourth step, the power-delay product PDP is calculated:

[0045] PDP = P_avg × T_epoch

[0046] Where: P_avg represents the average power consumption of the processor in the steady-state phase, and T_epoch represents the effective execution time of power consumption acquisition for a single training cycle, unit: second.

[0047] Further, the construction of the three-layer hierarchical analysis model refers to the target layer, the scheme layer, and the criterion layer. The target layer is the comprehensive evaluation result of the processor. The criterion layer consists of end-to-end performance, operator support level, and energy consumption. The scheme layer is the quantitative evaluation indicators under each criterion layer; a tree-like topology is formed through a doubly linked list between levels. Among them, the parent node stores the weight distribution strategy, and the child node binds the training throughput, peak floating-point utilization rate, and power-delay product index data.

[0048] Further, the level of the comprehensive evaluation refers to the result classification after standardization mapping through a four-segment membership function with the acceleration ratio u of each index of the processor to be measured relative to the reference processor as the input: Excellent level v1: When u ≤ 1.0, the membership degree is 0, and it linearly increases to 1 in the interval u ∈ (1.0, 1.2); Good level v2: It is a unimodal membership function in the interval u ∈ (0.8, 1.0], and reaches the peak value of 1 at u = 0.9; Medium level v3: It is a unimodal membership function in the interval u ∈ (0.4, 0.6], and reaches the peak value of 1 at u = 0.5; Poor level v4: When u ≤ 0.4, the membership degree is 1, and it linearly decreases to 0 in the interval u ∈ (0.4, 0.6).

[0049] Further, the steps of constructing the three-dimensional scoring matrix M to store the membership degree scores of each index and then realizing hierarchical aggregation through matrix multiplication are as follows:

[0050] The first step is to implement multi-dimensional membership mapping based on the three-dimensional scoring matrix M, where: the first dimension is the set of evaluation indicators, binding the training throughput, peak floating-point performance utilization, and power-delay product indicators; the second dimension is the benchmark workloads to be tested that cover optimizations in the fields of computer vision, natural language processing, and recommendation systems; the third dimension is the evaluation grade division, including four membership grades: excellent, good, medium, and poor.

[0051] The second step is to construct a judgment matrix based on the 1-9 scale, calculate the eigenvector and perform the CR consistency test. For matrices that do not satisfy CR < 0.1, adjust the judgment matrix, and finally obtain the weight vectors of the secondary indicators and the primary criterion dimension respectively.

[0052] The third step is to perform sub-index layer aggregation, multiply M by the weight vector W2 of the secondary indicators to obtain the load-level scoring matrix.

[0053] The fourth step is to perform load-level aggregation, taking the average of all load scores under the same evaluation item.

[0054] The fifth step is to perform criterion layer aggregation, multiplying the result matrix by the primary weight vector.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] First, the evaluation method of the present invention establishes a collaborative evaluation mechanism for end-to-end training, operator performance, and system energy efficiency. Through comprehensive evaluation, it realizes the organic unity of key indicators in three dimensions, effectively solves the problem of fragmented dimensions in traditional benchmark evaluations, and overcomes the limitation of the prior art that can only provide single-dimensional performance data. This enables the present invention to evaluate the adaptability of hardware to real business scenarios, examines the adaptation and optimization degree of hardware for the core operators required by mainstream deep learning frameworks, provides a more comprehensive reference basis for the optimization decision of AI processors, and at the same time, the evaluation method of the present invention can achieve the power consumption and energy efficiency ratio under different computing loads, providing a reference for green computing and large-scale deployment.

[0057] Second, the device of the present invention, through the AI processor abstraction layer and containerization technology, not only fully compatible with the characteristics of computing architectures such as Turing and Ampere, but also seamlessly adapts to new heterogeneous computing architectures, such as Da Vinci and ASIC, etc., overcoming the technical problems of difficult cross-architecture platform deployment and incomparable test results of traditional benchmark evaluation devices. This cross-architecture support ability makes the present invention the first technical solution that truly realizes unified evaluation of different computing architectures. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic structural diagram of the evaluation device of the present invention;

[0059] Figure 2It is the implementation flowchart of the end-to-end evaluation in the embodiments of the present invention;

[0060] Figure 3 It is the implementation flowchart of the operator support degree evaluation in the embodiments of the present invention;

[0061] Figure 4 It is the parameter diagram of the operator support degree evaluation in the embodiments of the present invention;

[0062] Figure 5 It is the implementation flowchart of the energy consumption evaluation method in the embodiments of the present invention;

[0063] Figure 6 It is the hierarchical analysis model diagram constructed in the embodiments of the present invention. Detailed implementation manners

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] Refer to Figure 1 to further describe the evaluation device of the invention.

[0066] The fuzzy comprehensive evaluation device of the AI training system of the present invention includes an end-to-end performance evaluation module, an operator support degree evaluation module, an energy consumption evaluation module, a fuzzification processing module, and a comprehensive evaluation module. Among them:

[0067] Refer to Figure 2 to further describe the end-to-end evaluation in the embodiments of the present invention. The end-to-end performance evaluation module covers three types of AI core computing modes: computer vision (dense computing: Resnet-50), natural language processing (hybrid computing: BiLSTM, AlBert), and recommendation system (sparse computing: DLRM). The evaluation process includes three standardized steps: ① In the training parameter configuration stage, set the benchmark hyperparameter group and the task-specific accuracy threshold, and in the dataset and model initialization stage, use a fixed random seed to ensure data consistency (TOP-1 / ACC / AUC); ② In the TTA measurement stage, record the first epoch time when the preset verification index is reached; ③ In the throughput statistics stage, calculate the single-epoch sample processing rate (Throughput = N / Tepoch). The evaluation index adopts a two-dimensional system: TTA comprehensively reflects the training speed and convergence efficiency, and the throughput (samples / s) quantifies the data processing ability of the AI processor.

[0068] Refer to Figure 3 to further describe the implementation process of the operator support degree evaluation in the embodiments of the present invention.

[0069] The operator support level evaluation module focuses on evaluating five types of key operators, such as: ① Computation-intensive operators (convolution / full connection / LSTM / multi-head attention); ② Memory-access-intensive operators (embedding). The evaluation method adopts a three-stage process: ① In the random configuration generation stage, 200 groups of parameter configurations that are multiples of 2 are generated, such as Figure 4 as shown. Among them, for the convolution operator, parameters such as the input image size, input / output channel numbers, padding method, stride, convolution kernel size, and input / output amplitude need to be configured; for the full connection operator, the input / output dimensions and input / output speeds are involved; for the long short-term memory network (LSTM) operator, the input / hidden layer dimensions, sequence length, number of stacked layers, and whether it is bidirectional need to be set; for the multi-head attention operator, the sequence length, embedding dimension, and number of attention heads are included; while for the embedding operator, which is a memory-access-intensive operator, the vocabulary size, sequence length, and embedding dimension need to be defined; ② In the legality verification stage, abnormal configurations such as dimension mismatches are eliminated; ③ In the iterative test stage, 10 warm-up runs + 50 formal measurements (taking the time median) are performed. The evaluation system includes: ① The peak floating-point performance utilization rate reveals the theoretical computing power utilization rate; ② The average acceleration ratio reflects the performance differences across AI processors.

[0070] Refer to Figure 5 to further describe the implementation process of the energy consumption evaluation of the embodiments of the present invention.

[0071] The energy consumption evaluation module takes PDP (power-delay product) as the core indicator, and evaluates the energy efficiency performance by monitoring the product of the steady-state power consumption (Pavg) and the execution time (Tepoch) in the second training epoch. A power monitoring script is used to collect data in real time to avoid noise interference in the initialization stage, providing a standardized energy efficiency evaluation method for heterogeneous AI processors. Its specific implementation idea is: when the training enters the second epoch, power consumption monitoring is started, and power consumption data is collected at 100ms intervals through interfaces such as NVML / RAPL, and continuously recorded until the epoch ends; the average power consumption Pavg in the steady-state stage (excluding the first 10% and the last 10% of the data) is calculated, and then multiplied by the epoch execution time Tepoch to obtain PDP (power-delay product) = Pavg × Tepoch; at the same time, the maximum / minimum power consumption fluctuation range is statistically analyzed, and finally an energy consumption analysis report including the time-series power consumption curve, PDP value, and temperature data is generated.

[0072] The comprehensive evaluation module adopts a hierarchical fuzzy decision-making architecture. Based on the AHP algorithm, a three-level weight system is constructed, and the index tree structure is stored through a doubly linked list.

[0073] Refer to Figure 6 to further describe the hierarchical analysis model constructed in the embodiments of the present invention.

[0074] The first level is the overall goal layer; the second level includes three major criterion layers: end-to-end evaluation, operator support degree evaluation, and energy consumption evaluation; the third level is the specific index layer. Among them, the end-to-end evaluation includes two indicators: TTA and throughput. The operator support degree evaluation has two sub-indicators: peak floating-point utilization rate and speedup ratio. The energy consumption evaluation includes the PDP index. Furthermore, the weights of each level are dynamically calculated (CR < 0.1). The membership function library provides a four-segment conversion template to convert heterogeneous data such as TTA and FLOPSutil into standardized scores. And through the matrix operation of "membership degree × weight", multi-level aggregation is realized, and finally a weighted average comprehensive score is output. The specific implementation idea is as follows: First, establish an index tree mapping relationship to classify and collect the original test data (such as TTA, FLOPSutil, etc.) into three major criterion dimensions: end-to-end performance, operator efficiency, and energy consumption, and use a doubly linked list to store the index level relationship; then dynamically construct a judgment matrix based on the 1-9 scale, calculate the eigenvector and perform a CR consistency test. Adjust the judgment matrix for the matrix that does not meet CR < 0.1, and finally obtain the weight vectors of the secondary indicators (such as TTA and throughput) and the first-level criterion dimensions respectively. Secondly, build a four-segment membership function library inside. Taking the speedup ratio u of each index of the processor to be tested relative to the benchmark processor as the input, the results after standardized mapping are classified through the four-segment membership function. For example, for the excellent level (v1), when u ≤ 1.0, the membership degree is 0, 1.0

Claims

1. A multi-dimensional collaborative AI training processor benchmark evaluation device, characterized in that, It includes an end-to-end evaluation module, an operator support level evaluation module, an energy consumption evaluation module, and a comprehensive evaluation module; among them: The end-to-end evaluation module is used to fix the random seed for the load to be tested, generate a training set and a test set after data standardization preprocessing; configure standardized training parameters and TTA thresholds for the load, train each epoch and calculate the throughput metric, and stop training when the test set accuracy reaches the TTA threshold, and form the TTA metric with all the training times; The operator support level evaluation module is used to dynamically generate random test parameters for key operators of compute-intensive and memory-intensive types, perform warm-up iterations for each valid configuration, conduct formal tests and record the median execution time; calculate two core metrics of average speedup and peak floating-point performance utilization; The energy consumption evaluation module adopts a steady-state power consumption monitoring method, starts power consumption collection when the load training enters the second epoch, and records power consumption data in real time through the hardware interface until the epoch ends; in the data processing stage, after removing the transitional data in the initial and end stages to eliminate noise interference, calculate the average power consumption in the steady-state stage to obtain the power-delay product of the core energy efficiency evaluation metric; The comprehensive evaluation module is used to construct a three-layer analytic hierarchy model based on the three criterion dimensions of end-to-end performance, operators, and energy consumption; use the speedup ratio of each index of the processor to be tested relative to the benchmark processor as the input, and achieve standardized mapping through a four-segment membership function to obtain the level of comprehensive evaluation; automatically match the piecewise linear conversion strategy for index data of different dimensions, construct a three-dimensional scoring matrix M to store the membership degree scores of each index, and then achieve hierarchical aggregation through matrix multiplication; use the weighted average method for defuzzification, calculate the weighted sum of the scores of each level and the representative value of the corresponding level, and take the maximum value as the final evaluation level, and output a comprehensive performance score including multi-dimensional indexes.

2. A multi-dimensional collaborative AI training processor benchmark evaluation method for the benchmark evaluation device according to claim 1, wherein, The specific steps of this evaluation method are as follows: Step 1, deploy a containerized adaptation environment: Step 1.1, identify the processor architecture type and dynamically load the corresponding driver interface; Step 1.2, retrieve the container image repository based on the architecture characteristics, intelligently match the benchmark container image of the load to be tested optimized for the processor architecture, and inject platform-specific environment variables and runtime configurations; Step 1.3, after verifying the integrity of the hardware acceleration function in the isolated container environment, inject the load to be tested; Step 2, perform end-to-end performance evaluation: The end-to-end evaluation module fixes the random seed for the load to be tested, generates a training set and a test set after data standardization preprocessing; configures standardized training parameters and TTA thresholds for the load, trains each epoch and calculates the throughput metric, and stops training when the test set accuracy reaches the TTA threshold, and forms the TTA metric with all the training times; Step 3, implement operator support level evaluation: The operator support evaluation module dynamically generates random test parameters for key computationally intensive and memory-intensive operators, performs warm-up iterations for each valid configuration, conducts formal testing, and records the median execution time; calculates two core indicators: average acceleration ratio and peak floating-point performance utilization; Step 4: Perform energy consumption evaluation: The energy consumption evaluation module adopts a steady-state power consumption monitoring method. When the load training enters the second epoch, it starts power consumption collection and records power consumption data in real time through the hardware interface until the end of the epoch. In the data processing stage, after removing the transition period data in the initial and final stages to eliminate noise interference, the average power consumption in the steady-state stage is calculated to obtain the power consumption delay product of the core energy efficiency evaluation indicator. Step 5: Conduct comprehensive evaluation and analysis: The comprehensive evaluation module constructs a three-layer hierarchical analysis model based on the three criteria of end-to-end performance, operators and energy consumption; takes the acceleration ratio of each indicator of the processor to be tested relative to the benchmark processor as input, realizes standardized mapping through a four-segment membership function, and obtains the comprehensive evaluation level; automatically matches the piecewise linear conversion strategy for indicator data of different dimensions, constructs a three-dimensional scoring matrix M to store the membership score of each indicator, and then realizes hierarchical aggregation through matrix multiplication; calculates the scoring vector of each level of the hierarchical analysis, and after obtaining the criterion layer scoring vector, takes the maximum membership as the final evaluation level.

3. The benchmark evaluation method according to claim 2, characterized in that, The container image repository described in step 1.2 includes differentiated software stacks pre-installed for different computing architectures. For GPU architecture, it provides a deep learning framework and acceleration library set that natively supports the CUDA computing ecosystem. For NPU and other dedicated acceleration architectures, it provides a software stack implementation with the same function that has been recompiled from the underlying instruction set, and integrates the dedicated operator library and runtime components of the corresponding computing platform.

4. The benchmark evaluation method according to claim 3, wherein The optimized load to be tested described in step 1.2 is for the GPU architecture, using a standard implementation aligned with the native interface of the mainstream deep learning framework, and completing operator mapping based on the default precision mode; for NPU and other heterogeneous computing architectures, equivalent functions to the native interface of the mainstream deep learning framework are implemented through their programming specifications, and a mixed precision strategy is used according to the hardware characteristics.

5. The benchmark evaluation method according to claim 2, characterized in that The steps for dynamically generating random test parameters as described in step 3, performing warm-up iterations for each valid configuration, and then performing formal testing and recording the median execution time are as follows: In the first step, based on the computational graph topology characteristics of the target operator, a multi-dimensional parameter combination including data form and computational topology parameters is randomly generated; the data form includes tensor dimension, data type, and batch size; the computational topology parameters include input dimension, convolution kernel size, and sliding step size; The second step is to check the validity of the parameter combination according to the video memory capacity threshold of the current processor, and eliminate invalid configurations that exceed the hardware resource limit; The third step is to generate a test parameter set that meets the video memory occupancy threshold through a uniform distribution method; In the fourth step, 10 warm-up iterations are performed for each valid configuration to eliminate the initialization overhead; Step 5: Perform 50 formal tests in a stable state and record the effective execution time of a single iteration after each iteration. Step 6: Select the median value of the statistical distribution feature set as the benchmark test result.

6. The benchmark evaluation method according to claim 2, wherein The average speedup ratio and peak floating-point performance utilization index are calculated in step 3, wherein the average speedup ratio is: the source processor operator execution time divided by the current processor operator time; the peak floating-point performance utilization is obtained by the following formula: F_util=F_theoretical / (Π×T_avg) Among them, F_util represents the peak floating-point performance utilization, F_theoretical represents the theoretical number of floating-point operations, which is determined according to the operator algorithm logic and the input data dimension, Π represents the peak floating-point performance of the processor determined according to the processor hardware specification parameters, and T_avg represents the average execution time of the operator obtained from the test results, in seconds.

7. The benchmark evaluation method according to claim 2, wherein The steps of the steady-state power consumption monitoring method described in step 4 are as follows: The first step is to start power consumption collection when the load training enters the second epoch until the current training cycle ends; The second step is to perform time domain alignment and noise filtering on the original power consumption data, and remove the transition period data of the first 10% duration of the initial stage of the training cycle and the last 10% duration of the final stage; The third step is to calculate the average power consumption P_avg in the steady-state range, in watts; The fourth step is to calculate the power delay product PDP: PDP=P_avg×T_epoch Where: P_avg represents the average power consumption of the processor in the steady state stage, T_epoch represents the effective execution time of power consumption collection in a single training cycle, unit: second.

8. The benchmark evaluation method according to claim 2, wherein The three-layer hierarchical analysis model constructed in step 5 refers to the target layer, the solution layer, and the criterion layer. The target layer is the comprehensive evaluation result of the processor, the criterion layer is composed of end-to-end performance, operator support level and energy consumption, and the solution layer is the quantitative evaluation index under each criterion layer; A tree topology is formed between the levels through a bidirectional linked list, in which the parent node stores the weight allocation strategy and the child node is bound to the training throughput, peak floating point utilization and power consumption delay product indicator data.

9. The benchmark evaluation method according to claim 2, wherein The level of comprehensive evaluation described in step 5 refers to the classification of the results after standardized mapping by using the acceleration ratio u of each indicator of the processor to be tested relative to the benchmark processor as input through a four-stage membership function: excellent level v1: when u≤1.0, the membership is 0, and it increases linearly to 1 in the interval u∈(1.0,1.2); good level v2: it presents a single-peak membership function in the interval u∈(0.8,1.0], and reaches a peak value of 1 when u=0.9; medium level v3: it presents a single-peak membership function in the interval u∈(0.4,0.6], and reaches a peak value of 1 when u=0.5; poor level v4: when u≤0.4, the membership is 1, and it decreases linearly to 0 in the interval u∈(0.4,0.6).

10. The benchmark evaluation method according to claims 6 to 9, characterized in that, The steps of constructing a three-dimensional scoring matrix M to store the membership scores of each indicator in step 5 and then realizing hierarchical aggregation through matrix multiplication are as follows: The first step is to implement multi-dimensional membership mapping based on the three-dimensional scoring matrix M, where: the first dimension is the set of evaluation indicators, binding the training throughput, peak floating-point performance utilization, and power-delay product indicators; the second dimension is the benchmark workloads to be tested covering optimizations in the fields of computer vision, natural language processing, and recommendation systems; the third dimension is the evaluation grade division, including four membership grades: excellent, good, medium, and poor. The second step is to construct a judgment matrix based on the 1-9 scale, calculate the eigenvector and conduct a consistency test. The matrix with a consistency greater than 0.1 meets the consistency test. For the matrix with a consistency less than 0.1, adjust the judgment matrix. Finally, obtain the weight vectors of the secondary indicators and the primary criterion dimension respectively. The third step is to perform sub-index layer aggregation. Multiply M by the secondary indicator weight vector W2 to obtain the load-level scoring matrix. The fourth step is to perform load-level aggregation. Take the average of all load scores under the same evaluation item. The fifth step is to perform criterion layer aggregation. Multiply the result matrix of the load-level aggregation by the primary weight vector.

Images