AI device evaluation method and device based on stepped pressure, equipment and medium

Abstract

This application provides an AI device evaluation method, apparatus, device, and medium based on stepped pressure. The method includes: for each AI device to be evaluated, obtaining at least one type of test load model based on AI device domain requirements and AI device parameters; running the AI ​​device to be evaluated in each test load model; applying multiple levels of fixed stepped pressure to each test load model; when the test load model is stable under each level of fixed stepped pressure; collecting multi-dimensional performance parameters of the AI ​​device to be evaluated; determining cross-vendor multi-dimensional normalized reference values ​​of the test load model based on historical standard parameters of the reference AI device; calculating a comprehensive score for the AI ​​device to be evaluated based on the multi-dimensional performance parameters of each test load model, AI device domain requirements, and cross-vendor multi-dimensional normalized reference values ​​of each test load model; and evaluating the AI ​​device with the highest comprehensive score as the optimal device that meets the AI ​​device domain requirements, thereby improving the accuracy of the evaluation results.

Description

Technical Field

[0001] This application relates to the field of performance evaluation technology for artificial intelligence devices, specifically to an AI device evaluation method, apparatus, equipment, and medium based on stepped pressure. Background Technology

[0002] Currently, AI devices are diverse in type and architecture, with different manufacturers employing differentiated technical approaches in areas such as computing power design, power consumption control, and task scheduling. This results in a lack of unified measurement standards for device performance parameters. As AI applications such as large models, multimodal inference, and autonomous driving become increasingly prevalent, actual operation often faces complex conditions involving dynamic load fluctuations and alternating high and low intensity loads. Traditional evaluation methods often rely on single fixed loads or simple stress tests, failing to accurately represent the stepped load variations in real-world scenarios. This leads to significant discrepancies between evaluation results and actual usage performance. Furthermore, existing evaluation systems often focus on single performance indicators, lacking a normalized comparison mechanism across vendors. This results in biased performance assessments, weak horizontal comparability, and an inability to provide comprehensive, objective, and practically relevant data for device selection. Therefore, it is evident that existing evaluation methods suffer from limitations in reproducing realistic stepped load conditions and insufficient horizontal comparison capabilities, leading to low evaluation accuracy. Summary of the Invention

[0003] To address the problems in the existing technology, this application provides an AI device evaluation method, apparatus, equipment, and medium based on stepped pressure, which can solve the problem of inconsistent parameter systems of AI devices from different manufacturers and the difficulty in making horizontal comparisons, thereby improving the accuracy of evaluation results.

[0004] To solve at least one of the above problems, this application provides the following technical solution: Firstly, this application provides an AI device evaluation method based on stepped pressure, comprising: acquiring multiple AI devices to be evaluated and AI device domain requirements; for each AI device to be evaluated, acquiring at least one type of test load model from a preset test load library based on the AI ​​device domain requirements and the AI ​​device parameters of the AI ​​device to be evaluated; running the AI ​​device to be evaluated in each type of test load model; adjusting the task load control parameters of each test load model based on the AI ​​device parameters and combining random load fluctuations to apply multi-level fixed stepped pressure to each test load model to simulate the multi-level stepped pressure scenario of each test load model; and applying multi-level fixed stepped pressure to each test load model. When the test load model operates stably in each of the stepped pressure scenarios, multi-dimensional performance parameters of the AI ​​device under evaluation are collected in each stepped pressure scenario. Based on the historical standard parameters of the reference AI device, the cross-vendor multi-dimensional normalized reference value of the test load model is determined. Based on the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the stepped pressure scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference value of each of the test load models, a comprehensive score is calculated to determine whether the AI ​​device under evaluation meets the domain requirements of the AI ​​device. The AI ​​device with the highest comprehensive score is evaluated as the optimal device that meets the domain requirements of the AI ​​device.

[0005] In some embodiments, the step of obtaining at least one type of test load model from a preset test load library for each of the AI ​​devices to be evaluated, based on the domain requirements of the AI ​​device and the AI ​​device parameters of the AI ​​device to be evaluated, includes: obtaining the AI ​​device parameters of the AI ​​device to be evaluated, parsing the domain requirements of the AI ​​device, determining the AI ​​device domain, and if the AI ​​device domain is the entire field of artificial intelligence, then obtaining at least six types of test load models matching the AI ​​device parameters from the preset test load library, wherein the six types of test load models are respectively a large language model type load model, a multimodal type load model, a computer vision type model, a recommendation type model, and a basic operator pressure type. Force-type models and system-level stress-type models; if the AI ​​device domain is a local domain of artificial intelligence, then at least one type of test load model that matches the local domain of artificial intelligence and the parameters of the AI ​​device is obtained from the preset test load library. The correspondence between the local domain of artificial intelligence and the test load model is as follows: natural language processing domain corresponds to the large language model type load model, video processing domain corresponds to the multimodal type load model, autonomous driving domain corresponds to the computer vision type model, recommendation system domain corresponds to the recommendation type model, hardware performance testing domain corresponds to the basic operator stress-type model, and multi-card heterogeneous computing domain corresponds to the system-level stress-type model.

[0006] In some embodiments, the task load control parameters include batch size, number of concurrent tasks, sequence length, and parallelism. The step of adjusting the task load control parameters of each test load model based on the AI ​​device parameters and combining this with random load fluctuations to apply multi-level fixed-step pressure to each test load model to simulate multi-level stepped pressure scenarios includes: for each test load model, while ensuring that the controlled task load control parameters conform to the AI ​​device parameters, adjusting at least one parameter among the task load control parameters to apply a first fixed-step pressure to the test load model; and applying random load fluctuations to the test load model while applying the first fixed-step pressure. The system simulates the test load model under a first-level pressure scenario. After the test load model has been under the first-level pressure scenario for a preset duration and its operating state meets a preset stable state, it performs another operation. Based on ensuring that the controlled task load control parameters meet the AI ​​device parameters, it adjusts at least one parameter of the task load control parameters to apply a second fixed-level pressure to the test load model. When applying the second fixed-level pressure, random load fluctuations are simultaneously superimposed to simulate the second-level pressure scenario. This process is repeated until multiple levels of fixed-level pressure are applied to the test load model, at which point the adjustment of the task load control parameters ends. The multiple levels of fixed-level pressure are applied sequentially from low to high and cannot be skipped.

[0007] In some embodiments, the multi-dimensional performance parameters include computing power indicators, energy efficiency indicators, stability indicators, and system indicators. When the test load model operates stably in each of the stepped pressure scenarios, the multi-dimensional performance parameters of the AI ​​device under evaluation are collected in each of the stepped pressure scenarios. This includes: for each stepped pressure scenario, when the duration of the test load model in the stepped pressure scenario meets a preset duration, and the fluctuation value of the multi-dimensional performance parameters is less than a preset fluctuation, it indicates that the test load model is operating stably in the stepped pressure scenario, and the computing power indicators, energy efficiency indicators, stability indicators, and system indicators of the AI ​​device under evaluation in the stepped pressure scenario are collected.

[0008] In some embodiments, determining the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device includes: selecting a reference AI device from multiple AI devices based on computing power, computing accuracy, multi-card parallelism, and domain-wide acceptance; obtaining a set of historical multi-dimensional performance parameters of the reference AI device when running in each of the stepped stress scenarios of the test load model; averaging the performance parameters of the same dimension in the historical multi-dimensional performance parameter sets of each stepped stress scenario to obtain the normalized reference value of each dimension; and constructing the cross-vendor multi-dimensional normalized reference value of the test load model based on the normalized reference values ​​of each dimension.

[0009] In some embodiments, the step of calculating a comprehensive score for the AI ​​device under evaluation based on multi-dimensional performance parameters of the AI ​​device under evaluation in each of the tiered stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, includes: determining multi-dimensional weights based on the domain requirements of the AI ​​device; determining the tiered stress weights of each of the tiered stress scenarios based on the stress values ​​of each of the tiered stress scenarios in the multi-level tiered stress scenarios; and, for each of the test load models, combining the tiered stress weights to evaluate the AI ​​device under evaluation in each of the test load models. In each of the aforementioned stepped pressure scenarios, the parameters of the same dimension in the multi-dimensional performance parameters are weighted and summed to obtain the multi-scenario dimension values ​​of each dimension. For each dimension, based on the cross-vendor normalized reference value of the dimension in the cross-vendor multi-dimensional normalized reference value of the test load model, the multi-scenario dimension value of the dimension is normalized to obtain the normalized dimension value of the dimension. The normalized dimension values ​​of each dimension are summed using the multi-dimensional weights to obtain the single-model total evaluation value of the test load model. The single-model total evaluation values ​​of each test load model are added together to obtain the comprehensive score of the AI ​​device under evaluation as meeting the requirements of the AI ​​device domain.

[0010] In some embodiments, determining the multi-dimensional weights based on the requirements of the AI ​​device domain includes: if the AI ​​device domain is the entire field of artificial intelligence, the computing power indicator corresponds to the first weight, the energy efficiency indicator corresponds to the second weight, the stability indicator corresponds to the third weight, and the system indicator corresponds to the fourth weight, wherein the first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is equal to the fourth weight, and the first weight, the second weight, the third weight, and the fourth weight constitute the multi-dimensional weights; if the AI ​​device domain is a local field of artificial intelligence, the multi-dimensional weights corresponding to the multi-dimensional performance parameters are determined based on the performance priority of the local field of artificial intelligence.

[0011] Secondly, this application provides an AI device evaluation apparatus based on stepped pressure, comprising: an acquisition unit, configured to acquire multiple AI devices to be evaluated and AI device domain requirements; for each AI device to be evaluated, based on the AI ​​device domain requirements and the AI ​​device parameters of the AI ​​device to be evaluated, acquire at least one type of test load model from a preset test load library; run the AI ​​device to be evaluated in each type of test load model; adjust the task load control parameters of each test load model based on the AI ​​device parameters and combine random load fluctuations to apply multi-level fixed stepped pressure to each test load model to simulate the multi-level stepped pressure scenario of each test load model; and a collection unit, configured to collect data for each of the test load models. The load model, when the test load model is running stably in each of the stepped pressure scenarios, collects multi-dimensional performance parameters of the AI ​​device under test in each of the stepped pressure scenarios, and determines the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device; the evaluation unit is used to calculate the comprehensive score of the AI ​​device under test in meeting the requirements of the AI ​​device domain based on the multi-dimensional performance parameters of the AI ​​device under test in each of the stepped pressure scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference value of each of the test load models, and evaluates the AI ​​device under test with the highest comprehensive score as the optimal device that meets the requirements of the AI ​​device domain.

[0012] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the AI ​​device evaluation method based on stepped pressure.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the AI ​​device evaluation method based on stepped pressure.

[0014] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the AI ​​device evaluation method based on stepped pressure.

[0015] As can be seen from the above technical solution, this application provides an AI device evaluation method, apparatus, equipment, and medium based on stepped pressure. By combining the needs of the AI ​​device field with the device parameters, it accurately matches the corresponding test load model from a preset test load library, and uses a multi-level fixed stepped pressure combined with random load fluctuations to simulate real and complex pressure scenarios. This can better fit actual application conditions and comprehensively test the operational stability and performance of AI devices under different load intensities. By collecting multi-dimensional performance parameters and introducing historical standard parameters of reference AI devices to construct cross-vendor multi-dimensional normalized reference values, it effectively solves the problem of inconsistent parameter systems and difficulty in horizontal comparison of AI devices from different manufacturers, improving the objectivity and universality of the evaluation results. Finally, based on the comprehensive calculation of scores according to the field needs, measured performance, and normalization standards, it can quickly and accurately select the device most suitable for the corresponding AI device field needs. This not only ensures the scientificity and comparability of the evaluation results, but also provides a reliable basis for cross-vendor AI device selection, significantly improving the accuracy of AI device evaluation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the AI ​​device evaluation method based on stepped pressure in the embodiments of this application; Figure 2 This is a structural diagram of the AI ​​device evaluation device based on stepped pressure in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0020] In view of the problems existing in the prior art, this application provides an AI device evaluation method, apparatus, equipment, and medium based on stepped pressure. By combining the needs of the AI ​​device field with the device parameters, it accurately matches the corresponding test load model from a preset test load library, and uses a multi-level fixed stepped pressure combined with random load fluctuations to simulate real and complex pressure scenarios. This can better fit actual application conditions and comprehensively test the operational stability and performance of AI devices under different load intensities. By collecting multi-dimensional performance parameters and introducing historical standard parameters of reference AI devices to construct cross-vendor multi-dimensional normalized reference values, it effectively solves the problem of inconsistent parameter systems of AI devices from different manufacturers and the difficulty in horizontal comparison, improving the objectivity and universality of the evaluation results. Finally, based on the comprehensive calculation of scores according to the field needs, measured performance, and normalization standards, it can quickly and accurately select the device most suitable for the corresponding AI device field needs. This not only ensures the scientificity and comparability of the evaluation results, but also provides a reliable basis for cross-vendor AI device selection, significantly improving the accuracy of AI device evaluation.

[0021] To address the shortcomings of existing AI device evaluation methods, such as inability to reproduce realistic stepped load conditions, insufficient horizontal comparison capabilities, and low evaluation accuracy, and to effectively improve the accuracy of evaluation results and achieve fair and quantitative horizontal comparisons of AI devices from different manufacturers, this application provides an embodiment of an AI device evaluation method based on stepped pressure. See [link to embodiment]. Figure 1 The AI ​​device evaluation method based on stepped pressure specifically includes the following: Step S110: Obtain multiple AI devices to be evaluated and AI device domain requirements. For each AI device to be evaluated, based on the AI ​​device domain requirements and the AI ​​device parameters of the AI ​​device to be evaluated, obtain at least one type of test load model from the preset test load library. Run the AI ​​device to be evaluated in each type of test load model. Based on the AI ​​device parameters, adjust the task load control parameters of each test load model and combine random load fluctuations to apply multi-level fixed step pressure to each test load model to simulate the multi-level step pressure scenario of each test load model.

[0022] AI (Artificial Intelligence) refers to machines or systems created by humans that possess certain intelligent behaviors, capable of simulating human perception, reasoning, learning, judgment, and language comprehension, and automatically completing tasks such as information processing, pattern recognition, and decision optimization. AI devices to be evaluated refer to AI hardware devices that require performance testing and comparative evaluation, such as AI accelerator cards, servers, edge computing boxes, and smart chips. AI device domain requirements refer to the actual application scenarios in which AI devices are used, such as large model inference, multimodal generation, autonomous driving perception, cloud training, and industrial real-time inference. AI device parameters refer to the hardware parameters of the AI ​​device itself, such as computing power, video memory size, RAM size, power consumption limit, computing accuracy, number of cores, bandwidth, and heat dissipation specifications. The pre-built test load library refers to a collection of standardized AI test tasks that are pre-built and stored. This library is uniformly encapsulated using containerization or scripting methods, compatible with multiple deep learning frameworks such as PyTorch, TensorFlow, ONNX, and MindSpore. It covers various types of test load models and supports continuous expansion to include other types. Each test load model in the library is equipped with a pre-built standard version with unified input size, dataset, and accuracy. It also provides a user-defined interface, allowing testers to upload their own models and datasets, providing reusable, scalable, and configurable standardized task resources for the tiered stress evaluation of AI devices. Test load models refer to specific models selected and loaded from the pre-built test load library, i.e., specific AI test task instances. Test load models can be run directly on the AI ​​device under evaluation. Different intensity loads can be applied to the test load models by adjusting task load control parameters, and random fluctuations can be used to simulate real multi-tiered load application scenarios. Task load control parameters refer to parameters that control the pressure of test tasks, such as concurrency, batch size, input data volume, model size, inference frame rate, and request frequency. By adjusting these parameters, the test load model can be controlled to operate under different load states. Random load fluctuation refers to adding small, irregular random load changes on top of the baseline load pressure determined by the task load control parameters. This simulates non-steady-state conditions such as fluctuations in user request numbers and business traffic jitter in real application scenarios, avoiding an overly idealized test environment and thus improving the consistency between evaluation results and actual operating performance. Multi-level fixed tiered pressure refers to setting multiple fixed load pressures. By adjusting the task load control parameters in a gradually increasing direction, the test load model is sequentially operated under each of these fixed load pressure levels, thus forming a "tiered" pressure increase.The multi-level fixed step pressure includes 10%, 30%, 50%, 70%, 90%, and 100% load pressure, covering the full range of pressure from light load to full load. It fully adapts to the performance testing needs of different AI devices, accurately simulates the actual operating conditions of the device under different load intensities, and realizes the simulation of multi-level step pressure scenarios of the device from light load to heavy load, with one fixed step pressure corresponding to one step pressure scenario.

[0023] Specifically, in step S110 above, the step of obtaining at least one type of test load model from a preset test load library for each of the AI ​​devices to be evaluated, based on the domain requirements of the AI ​​device and the AI ​​device parameters of the AI ​​device to be evaluated, includes: obtaining the AI ​​device parameters of the AI ​​device to be evaluated, parsing the domain requirements of the AI ​​device, determining the AI ​​device domain, and if the AI ​​device domain is the entire field of artificial intelligence, then obtaining at least six types of test load models matching the AI ​​device parameters from the preset test load library, wherein the six types of test load models are respectively a large language model load model, a multimodal load model, a computer vision model, a recommendation model, and a basic model. Operator-level stress models and system-level stress models; if the AI ​​device domain is a local domain of artificial intelligence, then at least one type of test load model that matches the local domain of artificial intelligence and the parameters of the AI ​​device is obtained from the preset test load library. The correspondence between the local domain of artificial intelligence and the test load model is as follows: natural language processing domain corresponds to the large language model type load model, video processing domain corresponds to the multimodal type load model, autonomous driving domain corresponds to the computer vision type model, recommendation system domain corresponds to the recommendation type model, hardware performance testing domain corresponds to the basic operator stress model, and multi-card heterogeneous computing domain corresponds to the system-level stress model.

[0024] The AI ​​device field is divided into two main areas: the overall AI field and specific AI fields. The overall AI field encompasses all typical AI application scenarios, including large language models, multimodal processing, computer vision, scientific computing (recommendation processing), basic operator verification, and system-level heterogeneous parallelism. It is not limited to a single business direction and focuses on a comprehensive evaluation of the general capabilities of AI devices. Specific AI fields, on the other hand, focus on a single or specific sub-field of AI application scenarios, such as large language models, multimodal processing, computer vision, scientific computing (recommendation processing), basic operator verification, and system-level heterogeneous parallelism. These fields emphasize targeted evaluation of the specific capabilities of AI devices.

[0025] For example, the large language model workload can be any model such as Llama-3-8B, Llama-3-70B, Qwen2-72B, or the GPT series, supporting application scenarios such as pre-training, supervised fine-tuning, inference, key-value caching, and hybrid expert models. The multimodal workload can be any model employing a stable diffusion model, a video latent diffusion model, a Sora-like video generation model, LLAVA, or a Qwen visual multimodal model, supporting image and text generation and understanding, and video generation and understanding. Computer vision models, also known as autonomous driving models, can be any model such as YOLOv10, bird's-eye view, Transformer models, Transformer fusion detection models, or sparse perception autonomous driving models, supporting multi-sensor fusion, real-time inference, and end-to-end training. Recommendation models also include scientific computing models, which can be any model such as an Alpha 3-fold protein structure prediction model, deep accelerated reinforcement learning human feedback training, or a deep neural network recommendation model. The basic operator stress test models can be any of the following: mixed-precision general matrix multiplication, fast attention operator 2nd generation, global reduction communication for NVIDIA aggregate communication testing, all-to-all communication stress test, CUDA computational flow graph, etc. The system-level stress test models can be any of the following: multi-GPU parallelism (data parallelism + tensor parallelism + pipelined parallelism), heterogeneous load of CPU + GPU, bandwidth stress test of memory + high-speed serial bus + NVIDIA high-speed interconnect bus, etc.

[0026] For example, if the AI ​​device to be evaluated is an AI accelerator card, then the computing power, memory capacity, power consumption limit, bandwidth, and other parameters of the AI ​​accelerator card are obtained. If it is determined that the AI ​​device domain requirement is to comprehensively verify the versatility of the AI ​​accelerator card, then the AI ​​device domain is determined to be the entire field of artificial intelligence. In this case, six types of models are simultaneously selected from the preset test load library: large language model load models, multimodal load models, computer vision models, recommendation models, basic operator stress models, and system-level stress models. If it is determined that the AI ​​device domain requirement is for autonomous driving perception, then the AI ​​device domain is determined to be a partial field of autonomous driving. And according to the corresponding relationship, computer vision models that can be used to test autonomous driving are selected from the preset test load library.

[0027] Therefore, this solution can intelligently match the corresponding test load model based on the AI ​​device parameters and the actual application requirements of the AI ​​device. It can comprehensively select six types of loads for evaluation across all scenarios, or precisely match a single type of model for specific scenarios, achieving a high degree of adaptability between the test scenario, device capabilities, and application requirements. This improves the relevance, comprehensiveness, practicality, flexibility, and accuracy of the evaluation. In addition, each type of test load model in the preset test load library is equipped with a pre-set standard version with unified input size, dataset, and accuracy, providing a unified, fair, and practically applicable evaluation basis for AI devices from different manufacturers and with different positioning.

[0028] Specifically, in step S110, the task load control parameters include batch size, number of concurrent tasks, sequence length, and parallelism. Adjusting the task load control parameters of each test load model based on the AI ​​device parameters and combining this with random load fluctuations to apply multi-level fixed-step pressure to each test load model to simulate multi-level stepped pressure scenarios includes: for each test load model, while ensuring that the controlled task load control parameters conform to the AI ​​device parameters, adjusting at least one parameter among the task load control parameters to apply a first fixed-step pressure to the test load model; while applying the first fixed-step pressure to the test load model, applying random... The system simulates the test load model under a first-level pressure scenario by adjusting the machine load fluctuations. After the test load model has been under the first-level pressure scenario for a preset duration and its operating state meets a preset stable state, at least one parameter of the task load adjustment parameters is adjusted again, based on ensuring that the adjusted task load adjustment parameters meet the AI ​​device parameters, to apply a second fixed-level pressure to the test load model. Random load fluctuations are simultaneously superimposed when applying the second fixed-level pressure to simulate the second-level pressure scenario. This process is repeated until multiple levels of fixed-level pressure are applied to the test load model, at which point the adjustment of the task load adjustment parameters ends. The multiple levels of fixed-level pressure are applied sequentially from low to high and cannot be skipped.

[0029] Batch size refers to the number of samples processed in a single forward and backward propagation of the model. Concurrent task count refers to the number of AI tasks running simultaneously on the AI ​​device, determining the overall concurrent pressure of the model. Sequence length refers to the length of the input text or data, directly affecting computational load and GPU memory usage. Parallelism refers to the degree of task splitting and collaborative execution among multiple GPUs and nodes, such as data parallelism, tensor parallelism, and pipeline parallelism. First fixed-step pressure and second fixed-step pressure refer to the fixed load intensities of each level in a multi-level fixed-step pressure system, arranged sequentially. The first fixed-step pressure is the first level of fixed-step pressure in the multi-level fixed-step pressure system, i.e., the fixed-step pressure with the smallest pressure value. The second fixed-step pressure is the second level of fixed-step pressure, only greater than the first fixed-step pressure. Preset duration refers to the minimum time required for the model to run stably under each level of fixed-step pressure, ensuring the validity of the collected data. Preset stable state refers to the state where the fluctuations of the AI ​​device's operating indicators are within a reasonable range. Applying pressure step by step and not skipping steps means that the pressure applied to the test load model must be applied sequentially from low to high, and cannot be skipped. In other words, the first fixed step pressure must be applied to the test load model before the second fixed step pressure can be applied.

[0030] For example, for each test load model, without exceeding the hardware capabilities corresponding to the AI ​​device parameters of the AI ​​device under evaluation, at least one or more of the batch size, number of concurrent tasks, sequence length, and parallelism are adjusted to apply a first fixed-step pressure to the test load model. Based on the first fixed-step pressure, random load fluctuations are simultaneously superimposed to simulate the first-step pressure scenario in a real-world environment. When the test load model reaches the preset duration in the first-step pressure scenario and its operating state stabilizes, a second fixed-step pressure is applied to the test load model, with fluctuations superimposed, simulating the second-step pressure scenario in a real-world environment, again without exceeding the hardware capabilities corresponding to the AI ​​device parameters of the AI ​​device under evaluation. When the test load model reaches the preset duration in the second-step pressure scenario and its operating state stabilizes... Following the aforementioned method of applying fixed-step pressure to the test load model, the third, fourth, fifth, and sixth fixed-step pressures are applied sequentially to the test load model until all multi-level fixed-step pressure tests are completed, at which point the load control ends.

[0031] For example, when the fluctuation of the multi-dimensional performance parameters of the AI ​​device under test in the test load model is less than the preset fluctuation value, or when the fluctuation of the stability evaluation index of the test load model is less than the preset fluctuation value, the operating state of the test load model is considered to meet the preset stable state. The stability evaluation index can be the number of abnormal task interruptions, inference frame drop rate, service timeouts, device restarts, computing power drop rate, number of abnormal memory fluctuations, number of system errors, and number of process crashes.

[0032] Therefore, this solution uses AI device parameters as the baseline for dynamically adjusting load control parameters such as batch size, number of concurrent tasks, sequence length, and parallelism. Within this baseline, it applies fixed-step pressure to the test load model by adjusting load control parameters, and then superimposes random load fluctuations on top of these fixed-step pressures, realistically reproducing complex and variable real-world operating environments. Simultaneously, it employs a stable control method of gradually increasing pressure from low to high without skipping steps, ensuring that each pressure level meets preset durations and stabilizes before moving to the next, guaranteeing a safe and reliable testing process and accurate and effective data. Furthermore, by covering the full range of pressure from light load to full load, it comprehensively adapts to the performance testing needs of different AI devices, accurately simulating the actual operating conditions of devices under different load intensities, filling the performance data gaps in non-full load ranges, providing a scientific basis for selection in actual production environments, and significantly improving the authenticity, stability, and reference value of the evaluation results.

[0033] Step S120: For each of the test load models, when the test load model is running stably in each of the stepped pressure scenarios, collect the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the stepped pressure scenarios, and determine the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device.

[0034] The multi-dimensional performance parameters refer to the collected operational indicators of the AI ​​device under evaluation, such as computing power, energy efficiency, stability, and system metrics. Each dimension has multiple sub-indicators. For example, computing power includes sub-indicators such as trillions of floating-point operations per second, tokens processed per second, images processed per second, and frames per second. Energy efficiency includes sub-indicators such as power consumption in watts, trillions of floating-point operations per watt, energy consumption per token, and energy consumption per image. Stability includes sub-indicators such as P99 latency, latency jitter, task failure rate, and device temperature variation curve. System metrics include sub-indicators such as graphics processor utilization, video memory utilization, NVLink high-speed interconnect bandwidth utilization, PCIe bus bandwidth utilization, and CPU utilization. The reference AI device refers to a universally recognized standard AI device for comparison, used to unify the evaluation benchmark. Historical standard parameters refer to the standard performance data obtained from multiple tests of the reference AI device under the same test load and the same tiered stress. Cross-vendor multi-dimensional normalized reference values ​​refer to standardized conversion values ​​that are based on the historical standard parameters of reference AI devices and are used to uniformly convert various performance indicators of devices from different manufacturers. This is used to achieve fair and comparable horizontal comparisons between devices from different manufacturers.

[0035] Specifically, in the aforementioned step S120, the multi-dimensional performance parameters include computing power indicators, energy efficiency indicators, stability indicators, and system indicators. When the test load model is running stably in each of the stepped pressure scenarios, the multi-dimensional performance parameters of the AI ​​device under evaluation are collected in each of the stepped pressure scenarios. This includes: for each stepped pressure scenario, when the duration of the test load model in the stepped pressure scenario meets a preset duration, and the fluctuation value of the multi-dimensional performance parameters is less than a preset fluctuation, it indicates that the test load model is running stably in the stepped pressure scenario, and the computing power indicators, energy efficiency indicators, stability indicators, and system indicators of the AI ​​device under evaluation in the stepped pressure scenario are collected.

[0036] Among them, computing power indicators refer to parameters reflecting the computing capabilities of AI devices. Energy efficiency indicators refer to parameters reflecting the relationship between computing power and power consumption. Stability indicators refer to parameters reflecting the operational reliability of AI devices. System indicators refer to parameters reflecting the operational status of the AI ​​device's hardware system. Indicator fluctuation value: the range of fluctuation of each performance parameter over a period of time. Preset fluctuation refers to the maximum allowable fluctuation range of the indicator; exceeding this range is considered as unstable operation.

[0037] Therefore, this solution judges the operational stability of AI devices by setting both the duration and the fluctuation of indicators, ensuring that multi-dimensional performance data is collected only under stable conditions. This effectively avoids collection errors caused by instantaneous fluctuations or unstable operating conditions, and guarantees the authenticity and reliability of computing power, energy efficiency, stability, and system indicators. By collecting multi-dimensional parameters under various pressure levels, covering the full range of pressure from light load to full load, the solution accurately simulates the actual operating conditions of AI devices under different load intensities, filling the performance data gap in the non-full load range and providing a scientific basis for selection in actual production environments.

[0038] Specifically, in the aforementioned step S120, determining the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device includes: selecting a reference AI device from multiple AI devices based on computing power, computing accuracy, multi-card parallel capability, and domain-wide acceptance; obtaining the historical multi-dimensional performance parameter set of the reference AI device when running in each of the stepped stress scenarios of the test load model; averaging the performance parameters of the same dimension in the historical multi-dimensional performance parameter set of each stepped stress scenario to obtain the normalized reference value of each dimension; and forming the cross-vendor multi-dimensional normalized reference value of the test load model based on the normalized reference values ​​of each dimension.

[0039] Among them, the reference AI device refers to a standard device selected from numerous AI devices as a unified evaluation benchmark, used to provide a consistent comparison standard for devices from different manufacturers. Computing power refers to the AI ​​device's theoretical computing power, actual effective computing power, peak computing performance, and other computing capabilities. Computing accuracy refers to the AI ​​device's ability and efficiency in supporting half-precision floating-point, single-precision floating-point, 8-bit integer, and unidirectional floating-point operations. Multi-GPU parallel capability refers to the efficiency of multi-GPU interconnection, multi-machine communication, and parallel expansion. Domain-wide acceptance refers to the AI ​​device's versatility, maturity, market application breadth, and recognized representativeness within the industry. The historical multi-dimensional performance parameter set refers to the collection of multi-dimensional performance parameters such as computing power, energy efficiency, stability, and system performance obtained from multiple tests of the reference AI device running under various stress scenarios and test load models. In other words, the historical multi-dimensional performance parameter set corresponds one-to-one with the test load model and stress scenarios. Mean averaging involves taking the average of multiple historical data points for the same dimension to eliminate single-test errors and obtain a more stable standard value. The normalized reference values ​​for each dimension are based on the reference device and are standard reference values ​​determined for each performance dimension, used to uniformly convert the scores of different AI devices. The cross-vendor multi-dimensional normalized reference values ​​are a unified scoring benchmark composed of the normalized reference values ​​of each dimension, which can realize fair horizontal comparison of AI devices from different manufacturers.

[0040] Furthermore, the performance parameters of the same dimension in the historical multi-dimensional performance parameter set of each stepped pressure scenario are averaged to obtain normalized reference values ​​for each dimension. This includes: for each dimension, averaging the data of the dimension in the historical multi-dimensional performance parameter set of the same stepped pressure scenario to obtain the sub-dimensional mean; and weighting and summing the sub-dimensional means corresponding to each stepped pressure scenario according to the stepped pressure weights to obtain the normalized reference value corresponding to the dimension, thereby obtaining the normalized reference value for each dimension.

[0041] For example, considering comprehensive computing power, computational accuracy, multi-GPU parallel processing capabilities, and general industry recognition, a representative and industry-recognized reference AI device is selected from multiple AI devices. Historical performance data of this reference AI device is retrieved from multiple tests under various tiered stress scenarios within the current test load model. The average value is calculated for the historical performance parameters of each tiered stress scenario and each dimension, yielding a normalized reference value for that dimension. The normalized reference values ​​for all dimensions are combined to form a cross-vendor multi-dimensional normalized reference value corresponding to this test load model, serving as a unified standard for subsequent scoring.

[0042] Therefore, this solution generates scientifically unified, multi-dimensional, normalized reference values ​​across manufacturers by comprehensively screening reference AI devices and averaging historical performance parameters. This not only solves the problem of difficulty in comparing AI devices from different manufacturers due to their different parameter systems, but also accurately simulates the actual operating conditions of the devices by combining full-range stress testing scenarios, ensuring the practicality and authority of the normalized reference values. At the same time, relying on a unified normalization standard, AI devices from different positioning and manufacturers can be evaluated under the same benchmark, greatly improving the fairness, accuracy, and reference value of the evaluation, and providing a reliable basis for the performance comparison and selection of AI devices.

[0043] Step S130: Based on the multi-dimensional performance parameters of the AI ​​device under test in each of the stepped stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, calculate the comprehensive score of the AI ​​device under test to meet the domain requirements of the AI ​​device, and evaluate the AI ​​device under test with the highest comprehensive score as the optimal device that meets the domain requirements of the AI ​​device.

[0044] The comprehensive score refers to the quantitative total score calculated by combining measured performance, domain requirement weights, and normalization standards. It is used to evaluate the degree to which the AI ​​device under evaluation is adapted to the domain requirements of AI devices.

[0045] The device that meets the needs of the AI ​​device field has the highest overall score among all participating devices and is the most suitable AI device for actual use in this scenario.

[0046] Specifically, in step S130 above, the step of calculating a comprehensive score for the AI ​​device under evaluation based on the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the tiered stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, includes: determining multi-dimensional weights based on the domain requirements of the AI ​​device; determining the tiered stress weights of each of the tiered stress scenarios based on the stress values ​​of each of the tiered stress scenarios in the multi-level tiered stress scenarios; and, for each of the test load models, combining the tiered stress weights to evaluate the AI ​​device under evaluation in each of the test load models. In each of the stepped pressure scenarios of the test load model, the parameters of the same dimension in the multi-dimensional performance parameters are weighted and summed to obtain the multi-scenario dimension values ​​of each dimension. For each dimension, based on the cross-vendor normalized reference value of the dimension in the cross-vendor multi-dimensional normalized reference value of the test load model, the multi-scenario dimension value of the dimension is normalized to obtain the normalized dimension value of the dimension. The normalized dimension values ​​of each dimension are summed using the multi-dimensional weights to obtain the single-model total evaluation value of the test load model. The single-model total evaluation values ​​of each test load model are added together to obtain the comprehensive score of the AI ​​device under evaluation as meeting the requirements of the AI ​​device domain.

[0047] Among them, multi-dimensional weights refer to the importance coefficients assigned to different performance dimensions such as computing power, energy efficiency, stability, and system performance based on the needs of the AI ​​device field. Tiered stress weights refer to the weighting coefficients assigned to each tier of stress scenarios based on the stress level, reflecting the importance of different load intensities in the evaluation. Normalized dimension values ​​refer to the standard scores of each dimension after conversion using a unified benchmark. The single-model total evaluation value refers to the quantitative score of the overall performance of the AI ​​device under a single test load model.

[0048] For example, the pressure weight of each step in a multi-level stepped pressure scenario can be determined based on the pressure value of each step in the multi-level stepped pressure scenario, as shown in the following formula: , Where r = {0.1, 0.3, 0.5, 0.7, 0.9, 1.0}, r is the set of pressure values ​​corresponding to the multi-level stepped pressure scenario, and k represents the k-th stepped pressure scenario. This represents the k-th value in the set of pressure values. This represents the step pressure weight corresponding to the k-th step pressure scenario.

[0049] For example, based on the cross-vendor normalized reference value of the dimension in the cross-vendor multi-dimensional normalized reference value of the test load model, the multi-scenario dimension value of the dimension is normalized to obtain the normalized dimension value of the dimension, as shown in the following formula: S= , Where S is the normalized dimension value of the dimension. Multi-scenario dimension values. The cross-vendor normalized reference value is for the dimension. This means taking 100 and... The minimum value in.

[0050] Furthermore, it should be noted that if a weighted summation is performed on the stability indicators of the stability dimension, the negative values ​​of each collected data point in the stability dimension are weighted and summed using the respective step pressure weights of the stability dimension. If an indicator of a dimension includes multiple sub-indicators, the average value of the collected data for each sub-indicator is calculated, and this average value is used as the indicator value for that dimension. When calculating the average value of the collected data for multiple sub-indicators, the collected data can be weighted and summed according to the sub-indicator weights before calculating the average value.

[0051] Therefore, this solution achieves differentiated weighting of evaluation indicators and load intensity by combining multi-dimensional weights based on domain requirements and allocating tiered pressure weights according to pressure levels, making the evaluation results more aligned with actual application priorities. By weighted summation of parameters under each level of pressure scenario and cross-vendor normalization, it ensures reasonable data integration under different load intensities and solves the problem of inconsistent indicator systems and difficulty in horizontal comparison between different manufacturers' equipment. At the same time, the evaluation process covers the full range of pressure from light load to full load, fully adapting to the performance testing needs of different AI devices, accurately simulating the actual operating conditions of devices under different load intensities, making the performance data more complete and closer to real deployment scenarios. The final comprehensive score is scientific, objective, and highly comparable, and can accurately select the AI ​​devices most suitable for the corresponding domain requirements, significantly improving the practicality and reference value of the evaluation results.

[0052] Furthermore, the determination of multi-dimensional weights based on the needs of the AI ​​device domain includes: if the AI ​​device domain is the entire field of artificial intelligence, computing power corresponds to a first weight, energy efficiency corresponds to a second weight, stability corresponds to a third weight, and system indicators correspond to a fourth weight, wherein the first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is equal to the fourth weight, and the first weight, the second weight, the third weight, and the fourth weight constitute the multi-dimensional weights; if the AI ​​device domain is a local field of artificial intelligence, then the multi-dimensional weights corresponding to the multi-dimensional performance parameters are determined based on the performance priority of the local field of artificial intelligence.

[0053] Performance priority refers to the ranking of core performance dimensions that are adapted to a specific AI domain. For example, if the AI ​​domain is autonomous driving, the priority is given to stability indicators, followed by computing power indicators, then system indicators, and finally energy efficiency indicators.

[0054] Therefore, this solution achieves precise and personalized evaluation by combining the needs of the AI ​​device field and setting targeted multi-dimensional weights: For the entire field of artificial intelligence, a fixed weight ranking is adopted to ensure that the performance of all scenarios can be comprehensively evaluated; for specific fields, the performance priority and corresponding multi-dimensional weights are determined according to the core needs of the scenario, so that the multi-dimensional weights have the ability to adapt to the field. This ensures both the comprehensiveness of the evaluation across the entire field and the specificity of the evaluation in specific fields, making the comprehensive score more in line with the actual application needs, improving the scientificity, rationality and practicality of the evaluation results, and providing a reliable basis for the accurate selection of AI devices.

[0055] In summary, this solution combines the needs of the AI ​​device domain with the device parameters, accurately matches the corresponding test load model from a pre-set test load library, and simulates real-world complex pressure scenarios using a combination of multi-level fixed-step pressure and random load fluctuations. This approach better reflects actual application conditions and comprehensively tests the operational stability and performance of AI devices under different load intensities. By collecting multi-dimensional performance parameters and introducing historical standard parameters from reference AI devices to construct cross-vendor multi-dimensional normalized reference values, it effectively solves the problem of inconsistent parameter systems and difficulty in horizontal comparison among AI devices from different manufacturers, improving the objectivity and universality of the evaluation results. Finally, based on domain requirements, measured performance, and normalization standards, a comprehensive score is calculated, which can quickly and accurately select the most suitable device for the corresponding AI device domain requirements. This ensures the scientific rigor and comparability of the evaluation results and provides a reliable basis for cross-vendor AI device selection, significantly improving the accuracy of AI device evaluation.

[0056] To address the shortcomings of existing AI device evaluation methods, such as inability to reproduce realistic stepped load conditions, insufficient horizontal comparison capabilities, and low evaluation accuracy, and to effectively improve the accuracy of evaluation results and achieve fair and quantitative horizontal comparisons of AI devices from different manufacturers, this application provides an embodiment of an AI device evaluation device based on stepped pressure, which implements all or part of the aforementioned AI device evaluation method based on stepped pressure. See [link to embodiment]. Figure 2 The AI ​​device evaluation device based on stepped pressure specifically includes the following components: The acquisition unit 10 is used to acquire multiple AI devices to be evaluated and AI device domain requirements. For each AI device to be evaluated, based on the AI ​​device domain requirements and the AI ​​device parameters of the AI ​​device to be evaluated, at least one type of test load model is acquired from a preset test load library. The AI ​​device to be evaluated is run in each type of test load model. Based on the AI ​​device parameters, the task load control parameters of each test load model are adjusted and combined with random load fluctuations to apply multi-level fixed step pressure to each test load model to simulate the multi-level step pressure scenario of each test load model.

[0057] The acquisition unit 20 is used to acquire multi-dimensional performance parameters of the AI ​​device under evaluation in each of the test load scenarios when the test load model is running stably in each of the step pressure scenarios, and determine the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device.

[0058] Evaluation unit 30 is used to calculate a comprehensive score for the AI ​​device under evaluation based on the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the stepped stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, and evaluate the AI ​​device under evaluation with the highest comprehensive score as the optimal device that meets the domain requirements of the AI ​​device.

[0059] As described above, the AI ​​device evaluation device based on stepped pressure provided in this application can accurately match the corresponding test load model from a preset test load library by combining the needs of the AI ​​device domain with the device parameters. It simulates real and complex pressure scenarios by using a combination of multi-level fixed stepped pressure and random load fluctuations, which can better fit actual application conditions and comprehensively test the operational stability and performance of AI devices under different load intensities. By collecting multi-dimensional performance parameters and introducing reference AI device historical standard parameters to construct cross-vendor multi-dimensional normalized reference values, it effectively solves the problem of inconsistent parameter systems and difficulty in horizontal comparison of AI devices from different manufacturers, and improves the objectivity and universality of the evaluation results. Finally, based on the domain needs, measured performance and normalization standards, a comprehensive score is calculated, which can quickly and accurately select the device most suitable for the corresponding AI device domain needs. This not only ensures the scientificity and comparability of the evaluation results, but also provides a reliable basis for cross-vendor AI device selection, significantly improving the accuracy of AI device evaluation.

[0060] To further illustrate this solution, this application also provides a specific application example of using the aforementioned AI device evaluation device based on stepped pressure to implement the AI ​​device evaluation method based on stepped pressure, which specifically includes the following: In some embodiments, when the device is used to obtain at least one type of test load model from a preset test load library for each of the AI ​​devices to be evaluated based on the domain requirements of the AI ​​device and the AI ​​device parameters of the AI ​​device to be evaluated, it is specifically used to: obtain the AI ​​device parameters of the AI ​​device to be evaluated, parse the domain requirements of the AI ​​device, determine the AI ​​device domain, and if the AI ​​device domain is the entire field of artificial intelligence, obtain at least six types of test load models matching the AI ​​device parameters from the preset test load library, wherein the six types of test load models are respectively a large language model load model, a multimodal load model, a computer vision model, a recommendation model, etc. The test load models include basic operator stress models and system-level stress models. If the AI ​​device domain is a local domain of artificial intelligence, at least one type of test load model that matches the local domain of artificial intelligence and the parameters of the AI ​​device is obtained from the preset test load library. The correspondence between the local domain of artificial intelligence and the test load models is as follows: natural language processing corresponds to the large language model type load model, video processing corresponds to the multimodal type load model, autonomous driving corresponds to the computer vision type model, recommendation system corresponds to the recommendation type model, hardware performance testing corresponds to the basic operator stress model, and multi-card heterogeneous computing corresponds to the system-level stress model.

[0061] In some embodiments, when the device is used to adjust the task load control parameters of each test load model based on the AI ​​device parameters and combine them with random load fluctuations to apply multi-level fixed-step pressure to each test load model to simulate a multi-level fixed-step pressure scenario for each test load model, the device is specifically used to: for each test load model, while ensuring that the controlled task load control parameters conform to the AI ​​device parameters, adjust at least one parameter among the task load control parameters to apply a first fixed-step pressure to the test load model; when applying the first fixed-step pressure to the test load model, apply... Random load fluctuations are added to simulate the test load model under the first-level pressure scenario. After the test load model is under the first-level pressure scenario for a preset duration and the running state meets the preset stable state, at least one parameter of the task load control parameters is adjusted again to apply a second fixed-level pressure to the test load model, based on ensuring that the controlled task load control parameters meet the AI ​​device parameters. Random load fluctuations are simultaneously superimposed when applying the second fixed-level pressure to simulate the second-level pressure scenario. This process is repeated until multiple levels of fixed-level pressure are applied to the test load model, at which point the adjustment of the task load control parameters ends. The multiple levels of fixed-level pressure are applied step by step from low to high and cannot be skipped.

[0062] In some embodiments, the device is used to collect the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the tiered pressure scenarios when the test load model is running stably in each of the tiered pressure scenarios. Specifically, for each of the tiered pressure scenarios, when the duration of the test load model in the tiered pressure scenario meets a preset duration and the fluctuation value of the multi-dimensional performance parameters is less than a preset fluctuation, it indicates that the test load model is running stably in the tiered pressure scenario, and the device collects the computing power index, the energy efficiency index, the stability index, and the system index of the AI ​​device under evaluation in the tiered pressure scenario.

[0063] In some embodiments, when the device is used to determine the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device, it is specifically used to: select a reference AI device from multiple AI devices based on computing power, computing accuracy, multi-card parallelism, and domain-wide acceptance; obtain a set of historical multi-dimensional performance parameters of the reference AI device when it runs in each of the stepped stress scenarios of the test load model; perform mean processing on the performance parameters of the same dimension in the historical multi-dimensional performance parameter sets of each stepped stress scenario to obtain the normalized reference value of each dimension; and form the cross-vendor multi-dimensional normalized reference value of the test load model based on the normalized reference values ​​of each dimension.

[0064] In some embodiments, when the device is used to calculate a comprehensive score for the AI ​​device under evaluation to meet the AI ​​device domain requirements based on the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the stepped pressure scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, the device specifically performs the following: determining multi-dimensional weights based on the AI ​​device domain requirements; determining each stepped pressure weight of each stepped pressure scenario based on the pressure value of each stepped pressure scenario in the multi-level stepped pressure scenarios; and, for each of the test load models, applying the stepped pressure weights to the AI ​​device under evaluation in the test load models. In the multi-dimensional performance parameters of the test load model under each stepped pressure scenario, the parameters of the same dimension are weighted and summed to obtain the multi-scenario dimension values ​​of each dimension. For each dimension, based on the cross-vendor normalized reference value of the dimension in the cross-vendor multi-dimensional normalized reference value of the test load model, the multi-scenario dimension value of the dimension is normalized to obtain the normalized dimension value of the dimension. The normalized dimension values ​​of each dimension are summed using the multi-dimensional weights to obtain the single-model total evaluation value of the test load model. The single-model total evaluation values ​​of each test load model are added together to obtain the comprehensive score of the AI ​​device under evaluation as meeting the requirements of the AI ​​device domain.

[0065] In some embodiments, when the device is used to determine the multi-dimensional weights based on the requirements of the AI ​​device domain, it is specifically used as follows: if the AI ​​device domain is the entire field of artificial intelligence, the computing power indicator corresponds to the first weight, the energy efficiency indicator corresponds to the second weight, the stability indicator corresponds to the third weight, and the system indicator corresponds to the fourth weight, wherein the first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is equal to the fourth weight, and the first weight, the second weight, the third weight, and the fourth weight constitute the multi-dimensional weights; if the AI ​​device domain is a local field of artificial intelligence, the multi-dimensional weights corresponding to the multi-dimensional performance parameters are determined based on the performance priority of the local field of artificial intelligence.

[0066] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the AI ​​device evaluation method based on stepped pressure.

[0067] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described AI device evaluation method based on stepped pressure.

[0068] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described AI device evaluation method based on stepped pressure.

[0069] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating AI devices based on stepped pressure, characterized in that, The method includes: Multiple AI devices to be evaluated and their domain requirements are obtained. For each AI device to be evaluated, at least one type of test load model is obtained from a preset test load library based on the domain requirements and the AI ​​device parameters. The AI ​​devices to be evaluated are run in each of the test load models. The task load control parameters of each test load model are adjusted based on the AI ​​device parameters and combined with random load fluctuations to apply multi-level fixed step pressure to each test load model to simulate the multi-level step pressure scenario of each test load model. For each of the aforementioned test load models, when the test load model is running stably in each of the aforementioned stepped stress scenarios, the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the aforementioned stepped stress scenarios are collected, and the cross-vendor multi-dimensional normalized reference value of the test load model is determined based on the historical standard parameters of the reference AI device. Based on the multi-dimensional performance parameters of the AI ​​device under test under each of the stepped stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models, a comprehensive score is calculated to determine whether the AI ​​device under test meets the domain requirements of the AI ​​device. The AI ​​device under test with the highest comprehensive score is evaluated as the optimal device that meets the domain requirements of the AI ​​device.

2. The AI ​​device evaluation method based on stepped pressure according to claim 1, characterized in that, For each of the AI ​​devices to be evaluated, at least one test load model is obtained from a preset test load library based on the domain requirements of the AI ​​device and the AI ​​device parameters of the AI ​​device to be evaluated, including: The AI ​​device parameters of the AI ​​device to be evaluated are obtained, the domain requirements of the AI ​​device are analyzed, and the AI ​​device domain is determined. If the AI ​​device domain is the entire field of artificial intelligence, at least six types of test load models that match the AI ​​device parameters are obtained from the preset test load library. The six types of test load models are large language model load model, multimodal load model, computer vision model, recommendation model, basic operator stress model and system-level stress model. If the AI ​​device domain is a local domain of artificial intelligence, then at least one type of test load model that matches the local domain of artificial intelligence and the parameters of the AI ​​device is obtained from the preset test load library. The correspondence between the local domain of artificial intelligence and the test load model is as follows: natural language processing domain corresponds to the large language model class load model, video processing domain corresponds to the multimodal class load model, autonomous driving domain corresponds to the computer vision class model, recommendation system domain corresponds to the recommendation class model, hardware performance testing domain corresponds to the basic operator stress class model, and multi-card heterogeneous computing domain corresponds to the system-level stress class model.

3. The AI ​​device evaluation method based on stepped pressure according to claim 1, characterized in that, The task load control parameters include batch size, number of concurrent tasks, sequence length, and parallelism. The process of adjusting the task load control parameters of each test load model based on the AI ​​device parameters and combining this with random load fluctuations to apply multi-level fixed-step pressure to each test load model to simulate multi-level step pressure scenarios includes: For each of the aforementioned test load models, while ensuring that the adjusted task load adjustment parameters conform to the AI ​​device parameters, at least one parameter among the task load adjustment parameters is adjusted to apply a first fixed step pressure to the test load model. When the first fixed step pressure is applied to the test load model, random load fluctuations are applied to the test load model to simulate the test load model being under the first step pressure scenario; After the test load model has been in the first-level pressure scenario for a preset duration and its operating state meets the preset stable state, the process is repeated to adjust at least one parameter of the task load control parameters to apply a second fixed-level pressure to the test load model, based on ensuring that the controlled task load control parameters meet the AI ​​device parameters. Random load fluctuations are simultaneously superimposed when the second fixed-level pressure is applied to simulate the second-level pressure scenario. This process is repeated until all levels of fixed-level pressure have been applied to the test load model, at which point the adjustment of the task load control parameters ends. The multiple levels of fixed-level pressure are applied sequentially from low to high and cannot be skipped.

4. The AI ​​device evaluation method based on stepped pressure according to claim 1, characterized in that, The multi-dimensional performance parameters include computing power, energy efficiency, stability, and system performance. When the test load model operates stably under each of the tiered pressure scenarios, the multi-dimensional performance parameters of the AI ​​device under evaluation are collected under each of the tiered pressure scenarios, including: For each of the aforementioned tiered pressure scenarios, when the duration of the test load model in the tiered pressure scenario meets the preset duration and the fluctuation value of the multi-dimensional performance parameters is less than the preset fluctuation, it indicates that the test load model is operating stably in the tiered pressure scenario, and the computing power index, energy efficiency index, stability index, and system index of the AI ​​device under evaluation in the tiered pressure scenario are collected.

5. The AI ​​device evaluation method based on stepped pressure according to claim 1, characterized in that, The cross-vendor, multi-dimensional normalized reference values ​​for the test load model are determined based on historical standard parameters of the reference AI device, including: Reference AI devices were selected from multiple AI devices based on computing power, computational accuracy, multi-card parallel processing capabilities, and general acceptance in the field. Obtain the historical multi-dimensional performance parameter set of the reference AI device when it runs in each of the stepped pressure scenarios of the test load model. Perform mean processing on the performance parameters of the same dimension in the historical multi-dimensional performance parameter set of each stepped pressure scenario to obtain the normalized reference value of each dimension. Based on the normalized reference value of each dimension, form the cross-vendor multi-dimensional normalized reference value of the test load model.

6. The AI ​​device evaluation method based on stepped pressure according to claim 1, characterized in that, The comprehensive score for the AI ​​device under evaluation, based on its multi-dimensional performance parameters under each stepped stress scenario of each test load model, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each test load model, is calculated to determine whether the AI ​​device meets the domain requirements. This score includes: Based on the needs of the AI ​​device field, multi-dimensional weights are determined, and the pressure weights of each step pressure scenario are determined based on the pressure values ​​of each step pressure scenario in the multi-level step pressure scenario. For each of the aforementioned test load models, the parameters of the same dimension in the multi-dimensional performance parameters of the AI ​​device under test in each of the aforementioned step pressure scenarios of the test load model are weighted and summed in combination with the aforementioned step pressure weights to obtain the multi-scenario dimension values ​​of each dimension. For each of the aforementioned dimensions, based on the cross-vendor normalized reference value of the dimension in the cross-vendor multi-dimensional normalized reference value of the test load model, the multi-scenario dimension value of the dimension is normalized to obtain the normalized dimension value of the dimension. The normalized dimensional values ​​of each dimension are summed using the multi-dimensional weights to obtain the total evaluation value of a single model of the test load model. The total evaluation values ​​of each single model of the test load model are then added together to obtain a comprehensive score indicating whether the AI ​​device under test meets the requirements of the AI ​​device domain.

7. The AI ​​device evaluation method based on stepped pressure according to claim 6, characterized in that, The multi-dimensional weights determined based on the needs of the AI ​​device field include: If the AI ​​device field is the entire field of artificial intelligence, the computing power index corresponds to the first weight, the energy efficiency index corresponds to the second weight, the stability index corresponds to the third weight, and the system index corresponds to the fourth weight. The first weight is greater than the second weight, the second weight is greater than the third weight, and the third weight is equal to the fourth weight. The first weight, the second weight, the third weight, and the fourth weight constitute a multi-dimensional weight. If the AI ​​device domain is a local domain of artificial intelligence, then the multi-dimensional weights corresponding to the multi-dimensional performance parameters are determined based on the performance priority of the local domain of artificial intelligence.

8. An AI device evaluation device based on stepped pressure, characterized in that, The device includes: The acquisition unit is used to acquire multiple AI devices to be evaluated and the domain requirements of AI devices. For each AI device to be evaluated, based on the domain requirements of AI devices and the AI ​​device parameters of the AI ​​devices to be evaluated, it acquires at least one type of test load model from a preset test load library. The AI ​​devices to be evaluated are run in each type of test load model. Based on the AI ​​device parameters, the task load control parameters of each test load model are adjusted and combined with random load fluctuations to apply multi-level fixed step pressure to each test load model to simulate the multi-level step pressure scenario of each test load model. The data acquisition unit is used to collect multi-dimensional performance parameters of the AI ​​device under evaluation in each of the step pressure scenarios when the test load model is running stably in each of the step pressure scenarios, and to determine the cross-vendor multi-dimensional normalized reference value of the test load model based on the historical standard parameters of the reference AI device. The evaluation unit is used to calculate a comprehensive score for the AI ​​device under evaluation based on the multi-dimensional performance parameters of the AI ​​device under evaluation in each of the stepped stress scenarios of each of the test load models, the domain requirements of the AI ​​device, and the cross-vendor multi-dimensional normalized reference values ​​of each of the test load models. The AI ​​device under evaluation with the highest comprehensive score is evaluated as the optimal device that meets the domain requirements of the AI ​​device.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the AI ​​device evaluation method based on stepped pressure as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the AI ​​device evaluation method based on stepped pressure as described in any one of claims 1 to 7.

Images