AI accelerator performance fault diagnosis method

Abstract

The invention relates to the technical field of artificial intelligence calculation hardware, in particular to an AI accelerator performance fault diagnosis method, which comprises the following steps of: acquiring an actual measurement performance index of an AI accelerator through an MLPerf test suite, and calculating effective peak calculation performance and effective memory bandwidth of the AI accelerator; obtaining the theoretical calculation amount and the theoretical memory access amount of the target AI model, and calculating the theoretical calculation strength of the target AI model; judging a theoretical bottleneck type of the target AI model, and predicting a theoretical performance upper limit of the target AI model by using a Roofline model; calculating the actual calculation strength and the actual operation performance of the target AI model based on the acquired operation data of the AI accelerator, and judging the actual operation bottleneck type of the target AI model; and performing comprehensive judgment and diagnosis on the performance of the AI accelerator to obtain a performance fault diagnosis result of the AI accelerator. According to the method, the accuracy of performance diagnosis of the AI accelerator can be improved, and the problems that the performance fault diagnosis of the AI accelerator is not accurate enough and hardware abnormity and normal bottleneck are difficult to distinguish are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence computing hardware technology, and in particular to a method for diagnosing performance faults of an AI accelerator. Background Art

[0002] Artificial intelligence has become a crucial engine driving scientific and technological progress and social development. From image recognition and natural language understanding to complex scientific computing and industrial automation control, the application scope of AI models continues to expand, and model complexity (e.g., number of parameters and computational effort) has also increased dramatically. To efficiently execute these computationally intensive tasks, purpose-built AI accelerators have become the industry's mainstream choice. These include GPUs (graphics processing units) and neural network processing units (NPUs), such as the Huawei Ascend series and Google TPUs. These AI accelerators, leveraging massively parallel computing architectures and optimized compute units (such as Tensor Cores and Da Vinci architecture Cube Units), deliver AI computing performance far exceeding that of traditional CPUs. However, successfully deploying AI models on these high-performance AI accelerators is only the first step. Maintaining sustained, stable, and efficient performance over the long term presents numerous challenges. As complex hardware systems, the performance of AI accelerators is impacted by numerous factors. First, hardware itself has a physical lifecycle. Long-term high-load operation can cause silicon-based device aging, reduced heat dissipation efficiency, adjustments to power management policies (such as dynamic frequency reduction and power wall limiting), and even occasional or permanent failures of local hardware units (such as compute core failures and high-bandwidth memory (HBM) subsystem errors). Second, changes in the operating environment, such as temperature fluctuations and unstable power supply in the data center, can directly affect the actual operating status and performance of the hardware. Furthermore, frequent updates and iterations of the software stack that supports AI accelerators (including device drivers, firmware, runtime libraries such as CUDA or Huawei CANN, and AI frameworks such as TensorFlow, PyTorch, and MindSpore) may bring performance optimizations and new features, but they can also introduce unexpected performance degradation or compatibility issues. In real-world applications, especially those with extremely high performance and reliability requirements, any unexpected performance degradation of AI accelerators can lead to service delays, reduced prediction accuracy, and even trigger chain reactions, resulting in serious economic losses or security risks. Therefore, effective performance monitoring and timely fault diagnosis of AI accelerators are crucial.

[0003] Existing system monitoring methods have limitations. Traditional operating system-based monitoring tools (such as top, htop, and vmstat) primarily focus on CPU and system memory resources and fail to provide a deep understanding of the internal operating status and performance bottlenecks of AI accelerators. While specialized monitoring tools provided by accelerator vendors (such as NVIDIA's nvidia-smi or Huawei's npu-smi) can provide metrics such as hardware utilization, power consumption, temperature, and video memory usage, these metrics are often macroscopic and fail to reveal the specific causes of performance degradation. For example, observing a drop in utilization doesn't directly indicate whether it's due to an input data flow bottleneck, model scheduling issues, or a hardware failure. In summary, existing technologies for AI accelerator performance fault diagnosis lack a systematic diagnostic approach that can integrate the hardware's actual load capacity (rather than theoretical peak values), distinguish normal performance bottlenecks from hardware anomalies, and provide clues to the root cause. This is especially true for AI accelerators like Huawei's Ascend, which have specific architectures (such as the Da Vinci architecture) and complex software stacks. General-purpose monitoring and analysis methods may not be sufficient for accurate diagnosis. Summary of the Invention

[0004] In order to solve the technical problems existing in the existing technology, the present invention proposes a method for diagnosing AI accelerator performance faults, which can solve the problems in the existing technology that AI accelerator performance fault diagnosis is not accurate enough and it is difficult to distinguish between hardware anomalies and normal bottlenecks.

[0005] The purpose of the present invention can be achieved by taking the following technical solutions:

[0006] A method for diagnosing performance faults of an AI accelerator, the method comprising:

[0007] S1. Obtain the measured performance indicators of the AI ​​accelerator using the MLPerf benchmark model test suite and calculate the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator.

[0008] S2. Analyze the theoretical characteristics of the target AI model, obtain the theoretical computational load and theoretical memory access capacity of the target AI model, and calculate the theoretical computational intensity of the target AI model;

[0009] S3. Determine the theoretical bottleneck type of the target AI model and use the Roofline model to predict the theoretical performance upper limit of the target AI model;

[0010] S4. Run the target AI model on the AI ​​accelerator and collect the operating data of the AI ​​accelerator. Based on the collected operating data of the AI ​​accelerator, calculate the actual computing intensity and actual operating performance of the target AI model, and determine the actual operating bottleneck type of the target AI model.

[0011] S5. Perform a comprehensive judgment and diagnosis on the performance of the AI ​​accelerator based on the predicted theoretical bottleneck type, actual operation bottleneck type, actual operation performance, and predicted theoretical performance upper limit to obtain the performance fault diagnosis result of the AI ​​accelerator.

[0012] Specifically, the step S1 includes:

[0013] S11. Run the test suite of MLPerf benchmark models on an AI accelerator. The MLPerf benchmark models in the test suite include a compute-intensive model and a memory-intensive or hybrid model.

[0014] S12. Run a computationally intensive model to test the computing capability of the AI ​​accelerator, record the measured benchmark model throughput, and calculate the effective peak computing performance of the AI ​​accelerator based on the measured benchmark model throughput;

[0015] S13. Run a memory access or hybrid model to monitor the memory access status of the AI ​​accelerator at runtime, and calculate the effective memory bandwidth of the AI ​​accelerator based on the measured bandwidth data.

[0016] Specifically, step S3 includes:

[0017] S31. Calculate the balance point computing intensity of the AI ​​accelerator based on the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator;

[0018] S32. Compare the theoretical computational intensity of the target AI model with the computational intensity of the equilibrium point to determine the theoretical bottleneck type of the target AI model;

[0019] S33. Based on the accelerator's effective memory bandwidth, effective peak computing performance, and the theoretical computing intensity of the target AI model, the Roofline model is used to predict the theoretical performance upper limit of the target AI model.

[0020] Specifically, the step of comparing the theoretical computational intensity of the target AI model with the computational intensity of the equilibrium point to determine the theoretical bottleneck type of the target AI model includes:

[0021] When the theoretical computing intensity of the target AI model is greater than the computing intensity of the equilibrium point, the theoretical bottleneck type of the target AI model is determined to be a computing bottleneck;

[0022] When the theoretical computing intensity of the target AI model is less than the computing intensity of the equilibrium point, the theoretical bottleneck type of the target AI model is determined to be a bandwidth bottleneck;

[0023] When the theoretical computing intensity of the target AI model is equal to the computing intensity of the equilibrium point, it is judged that the theoretical bottleneck type of the target AI model is in a balanced state.

[0024] Specifically, the step S4 includes:

[0025] S41. Run the target AI model on the AI ​​accelerator, monitor the actual running status of the target AI model, and collect actual running performance data of the target AI model, where the actual running performance data includes: actual throughput, actual computing power, and actual memory access volume;

[0026] S42. Calculate the actual computing intensity of the target AI model based on the actual memory access amount and the actual computing amount, and calculate the actual operating performance of the target AI model based on the actual computing amount;

[0027] S43. Compare the actual computing intensity of the target AI model with the computing intensity of the equilibrium point to determine the actual bottleneck type of the target AI model.

[0028] Specifically, comparing the actual computing intensity of the target AI model with the equilibrium point computing intensity to determine the actual bottleneck type of the target AI model includes:

[0029] When the actual computing intensity is greater than the equilibrium point computing intensity, it is determined that the actual bottleneck type of the target AI model during actual operation is a computing bottleneck;

[0030] When the actual computing intensity is less than the equilibrium point computing intensity, it is determined that the actual bottleneck type of the target AI model during actual operation is the bandwidth bottleneck;

[0031] When the actual computing intensity is equal to the equilibrium point computing intensity, it is judged that the target AI model is in a balanced state during actual operation.

[0032] Specifically, step S5 includes:

[0033] S51. Calculate the performance achievement rate based on the actual performance prediction and the theoretical performance upper limit, set a performance degradation threshold, and determine that the AI ​​accelerator performance has significantly degraded when the performance achievement rate is less than the performance degradation threshold; and determine that the AI ​​accelerator performance has not significantly degraded when the performance achievement rate is greater than or equal to the performance degradation threshold, thereby obtaining a performance degradation analysis result.

[0034] S52. Determine whether there is a bottleneck mismatch based on whether the theoretical prediction and the actual operation bottleneck types are consistent, and obtain a bottleneck mismatch analysis result; if the theoretical bottleneck type is inconsistent with the actual operation bottleneck type, determine that there is a bottleneck mismatch; otherwise, determine that the bottleneck matches;

[0035] S53. Combine the performance degradation analysis results and the bottleneck mismatch analysis results to comprehensively diagnose the performance of the AI ​​accelerator and obtain the performance fault diagnosis results of the AI ​​accelerator.

[0036] Specifically, the step S53 includes:

[0037] When the AI ​​accelerator performance significantly declines and there is a bottleneck mismatch, it is considered a serious performance failure;

[0038] When the AI ​​accelerator performance significantly declines but the bottleneck is matched, it is considered a moderate performance failure;

[0039] When the AI ​​accelerator performance does not show significant degradation but there is a bottleneck mismatch, it is judged to be in a warning state;

[0040] When the AI ​​accelerator performance does not show significant degradation and the bottleneck is matched, it is considered to be operating normally.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] The present invention provides a method for diagnosing performance faults of AI accelerators. The method obtains the measured performance indicators of the AI ​​accelerator through the test suite of the MLPerf benchmark model, combines the bottleneck mismatch with Roofline analysis, effectively distinguishes hardware anomalies from normal bottlenecks, and improves the accuracy of AI accelerator performance fault diagnosis. Based on the MLPerf standard, the diagnostic results are comparable. According to the performance degradation mode and the bottleneck mismatch type, the performance of the AI ​​accelerator is comprehensively judged and diagnosed to obtain the performance fault diagnosis result of the AI ​​accelerator. This can solve the problem in the prior art that the performance fault diagnosis of AI accelerators is not accurate enough and it is difficult to distinguish hardware anomalies from normal bottlenecks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0044] Figure 1 This is an overall flow chart of a method for diagnosing AI accelerator performance faults in an embodiment of the present invention;

[0045] Figure 2 1 is a schematic diagram of a hardware performance benchmark calibration process in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the Roofline model coordinates in an embodiment of the present invention;

[0047] Figure 4 4 is a performance fault diagnosis logic diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is obvious that the embodiments described are only some embodiments of the present invention, not all embodiments, and the implementation of the present invention is not limited to these. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] Example 1:

[0050] like Figure 1 As shown, the method for diagnosing AI accelerator performance faults according to the present invention includes the following steps:

[0051] S1. Obtain the measured performance indicators of the AI ​​accelerator through the test suite of the MLPerf benchmark model, and calculate the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator.

[0052] S11. Run the test suite of MLPerf benchmark models on AI accelerators. The MLPerf benchmark models in the test suite include a compute-intensive model and a memory-access or hybrid model.

[0053] The standard MLPerf benchmark suite is run on the target AI accelerator to benchmark the hardware performance of the AI ​​accelerator and establish an effective performance baseline that reflects its actual capabilities. The MLPerf benchmark suite uses a selection of compute-intensive models: ResNet and BERT; and memory-intensive / hybrid models: DLRM and 3D-UNet. One compute-intensive model and one memory-intensive or hybrid model are used. ResNet (Residual Network) represents a compute-intensive model in MLPerf, primarily used for image classification tasks. BERT (Bidirectional Encoder Representation) is a compute-intensive model in natural language processing. BERT relies on the massively parallel attention mechanism of the Transformer architecture to support tasks such as question answering and text classification. DLRM (Deep Learning Recommendation Model) represents a memory-intensive and hybrid workload, used in personalized recommendation systems. 3D-UNet represents a hybrid model, used for medical image segmentation (such as 3D voxel processing in MRIs). By testing both compute-intensive and memory-intensive models, the MLPerf benchmark reveals optimization potential for AI accelerators in multiple dimensions, including computing power, storage, and energy efficiency. Running a test suite containing at least one compute-intensive and one memory-intensive (or mixed) MLPerf benchmark model aims to expose optimization potential for AI accelerators in multiple dimensions, such as computing power, storage, and energy efficiency, by testing both compute-intensive and memory-intensive models. This allows for a more comprehensive benchmarking of AI accelerator performance. Specifically, this tests various hardware metrics of AI accelerators, such as computing power, memory, and memory bandwidth.

[0054] S12. Run the computationally intensive model ResNet50 to test the computing capability of the AI ​​accelerator, record the measured benchmark model throughput, and calculate the effective peak computing performance of the AI ​​accelerator based on the measured benchmark model throughput.

[0055] The purpose of running the computationally intensive ResNet50 model is to test the computing power of the AI ​​accelerator. During the test, the measured benchmark model throughput is recorded. The ResNet-50 model is used as the core benchmark for image classification tasks to evaluate computing performance because it involves multi-layer matrix operations in deep convolutional networks.

[0056] like Figure 2The figure shows a schematic diagram of the hardware performance benchmark calibration process. In this embodiment, the MLPerf benchmark test suite is periodically run on the target AI accelerator Ascend 910, using the computationally intensive model ResNet50 to test computing power and record the measured benchmark model throughput. ResNet-50 involves multi-layer matrix operations of deep convolutional networks and is the core benchmark for image classification tasks. The measured benchmark model throughput is recorded by combining the recorded measured benchmark model throughput with the known theoretical computational load of a single inference of the benchmark model, and is recorded as T bench , unit: samples / sec, the theoretical computational amount of a single inference of this benchmark model is recorded as FLOPs bench-inf Specifically, the effective peak computing performance = the measured benchmark model throughput × the single inference computing amount of the benchmark model.

[0057] The effective peak computing performance is calculated based on the measured benchmark model throughput, and the baseline P eff The effective peak computing performance that reflects the actual capabilities of the AI ​​accelerator, the baseline P eff The calculation formula is expressed as:

[0058] P eff =T bench ×FLOPs bench-inf ;

[0059] S13. Run a memory access or hybrid model to monitor the memory access status of the AI ​​accelerator at runtime, and calculate the effective memory bandwidth of the AI ​​accelerator based on the measured bandwidth data.

[0060] The purpose of running memory access or hybrid model DLRM is to test the hybrid capabilities of AI accelerators, especially their memory access related performance. During the test, the DLRM model will be run on the target AI accelerator to monitor its runtime memory access. Monitoring runtime memory access refers to the memory bandwidth (HBM, DDR) of the AI ​​accelerator, and combined with the measured bandwidth data to establish the "effective memory bandwidth" baseline BW eff , effective memory bandwidth reflects the efficiency of AI accelerators in accessing memory in actual applications. These baseline data (P eff ,BW eff ) is stored together with the software and hardware environment information during the test (such as driver version, firmware version, test temperature, etc.) as a benchmark for subsequent comparisons.

[0061] S2. Analyze the theoretical characteristics of the target AI model, obtain the theoretical computing capacity and theoretical memory access capacity of the target AI model, and calculate the theoretical computing intensity of the target AI model.

[0062] Specifically, static analysis is performed on the target AI model or a model analysis tool is used to obtain the theoretical total computational capacity and theoretical total memory access capacity of a single inference or single iteration of the target AI model. Static analysis refers to analyzing the structure of the model, analyzing the computation and memory access of a forward or backward propagation, so as to obtain the theoretical total computational capacity and theoretical total memory access capacity of a single iteration. The theoretical total memory access capacity includes weights, activations, inputs and outputs, etc. Weights refer to the parameters of each layer of the model (such as convolution kernels, fully connected layer weight matrices, etc.); activations refer to the intermediate outputs of each layer (such as feature maps after convolution); inputs and outputs refer to the input data and final output content of the model.

[0063] In this embodiment, a specific LSTM model for power load forecasting is analyzed. LSTM is a long short-term memory network model, which is a special recurrent neural network (RNN). It is only used as an example model here and has no special features. It can be replaced with other models. The theoretical total computing capacity and theoretical total memory access capacity of the LSTM model are obtained. The theoretical computing intensity of the target AI model is calculated based on the theoretical total computing capacity and theoretical total memory access capacity. The theoretical computing intensity of the target AI model is AI. theory The calculation formula can be expressed as:

[0064]

[0065] Among them, FLOPs model-total is the theoretical total computational capacity, Bytes model-total The theoretical total memory access volume. The theoretical computational intensity of the model reveals the hardware performance bottleneck of the model and is the core analysis indicator of the roofline model.

[0066] S3. Determine the theoretical bottleneck type of the target AI model and use the Roofline model to predict the theoretical performance upper limit of the target AI model.

[0067] S31. Calculate the balance point computing intensity of the AI ​​accelerator based on the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator.

[0068] Specifically, the balance point computing intensity = effective peak computing performance / effective memory bandwidth, the balance point computing intensity of computing hardware AI balance Expressed as:

[0069]

[0070] S32. Compare the theoretical computing intensity of the target AI model with the computing intensity of the equilibrium point to determine the theoretical bottleneck type of the target AI model.

[0071] Specifically, the theoretical computational intensity AI of the target model theory Calculate strength AI with balance point balance For comparison:

[0072] When the theoretical computing intensity of the target AI model is greater than the computing intensity of the equilibrium point, that is, AI theory >AI balance , the theoretical bottleneck type of the target AI model is determined to be compute-bound; the program performance has reached the upper limit of the hardware computing power and cannot be further improved by optimizing the computing intensity. At this time, the hardware computing power utilization is close to 100%.

[0073] When the theoretical computing intensity of the target AI model is less than the computing intensity of the equilibrium point, that is, AI theory <AI balance , then the theoretical bottleneck type of the target AI model is determined to be bandwidth bottleneck (Memory-Bound); program performance is linearly positively correlated with computing intensity, and limited by memory bandwidth, the hardware computing power utilization is low at this time

[0074] When the theoretical computing intensity of the target AI model is equal to the computing intensity of the equilibrium point, AI theory ≈AI balance , then the theoretical bottleneck type of the target AI model is judged to be in a balanced state.

[0075] S33. Based on the accelerator's effective memory bandwidth, effective peak computing performance, and the theoretical computing intensity of the target AI model, the Roofline model is used to predict the theoretical performance upper limit of the target AI model.

[0076] Specifically, the theoretical performance upper limit P of the target AI model thoery-max It represents the theoretical maximum computing performance that the target model can achieve under the currently calibrated hardware baseline capabilities. As a theoretical performance analysis tool, the Roofline model establishes a relationship between computing performance and computing intensity. By combining the peak computing power and peak memory bandwidth of the hardware (AI accelerator), it can visually determine whether a computing task is limited by the processing speed of the computing unit or the data transmission speed of the memory system. The Roofline model addresses the question of: what is the theoretical performance ceiling that can be achieved by a target AI model with a computational load of A and a memory access load of B on an AI accelerator (computing platform) with a peak computing power of C and a peak memory bandwidth of D?

[0077] Theoretical performance upper limit P of the target AI model thoery-max is expressed by the following formula:

[0078] P thoery-max =min(P eff ,AItheory ×BW eff );

[0079] BW eff The effective memory bandwidth of AI accelerator, P eff The effective peak computing performance of AI accelerators, AI theory is the theoretical computational strength of the target AI model.

[0080] like Figure 3 The following figure shows a coordinate diagram of the Roofline model. The horizontal axis, I, represents the model's computational intensity (FLOP / Byte); the vertical axis, P, represents the number of floating-point operations per second (FLOP / S). The theoretical model performance, β and π, represent the bandwidth and computing power of the AI ​​accelerator, respectively. Calibration is performed here (using the MLPerf model). I_max refers to the maximum computational intensity of the AI ​​accelerator (hardware platform), where I_max = π / β. By calculating the theoretical computational intensity (horizontally), we can determine whether the vertical axis is memory-bound or computer-bound.

[0081] S4. Run the target AI model on the AI ​​accelerator and collect the operating data of the AI ​​accelerator. Based on the collected operating data of the AI ​​accelerator, calculate the actual computing intensity and actual operating performance of the target AI model, and determine the bottleneck type during the actual operation of the target AI model.

[0082] S41. Run the target AI model on the AI ​​accelerator, monitor the actual operating status of the target AI model, and collect actual operating performance data of the target AI model. The actual operating performance data includes: actual throughput, actual computing power, and actual memory access volume.

[0083] In this embodiment, the AI ​​accelerator uses Huawei's Ascend series NPU. When the target AI model is actually running online, the actual operation performance data is collected in real time or periodically using the Ascend Profiler tool and MindStudio Profiling tool provided by Huawei, or through the npu-smi command and AscendCL API. The actual operation performance data includes: actual throughput T actual (For example: QPS-Queries Per Second or Samples / Second), average inference latency actual , actual computing capacity FLOPs actual-executed , actual memory access size Bytes actual-access , AI accelerator real-time status data. AI accelerator real-time status data includes: core temperature Temp corePower consumption usage , operating frequency Freq actual wait.

[0084] S42. Calculate the actual computing intensity of the target AI model based on the actual memory access amount and the actual computing amount, and calculate the actual operating performance of the target AI model based on the actual computing amount.

[0085] According to the actual memory access amount and actual computing amount, the actual computing intensity of the target AI model in actual operation is calculated. actual It can be expressed as:

[0086]

[0087] Among them, FLOPs actual-executed Indicates the actual amount of calculation, Bytes actual-access Represents the actual memory access amount.

[0088] Calculate the actual operating performance of the target AI model based on the actual computing amount. The actual operating performance P actual It can be expressed as:

[0089]

[0090] FLOPs actual-executed Indicates the actual amount of calculation, and Execution Time indicates the actual time consumed by the target model to complete a specific amount of calculation during actual operation.

[0091] S43. Compare the actual computing intensity of the target AI model with the computing intensity of the equilibrium point to determine the actual bottleneck type of the target AI model.

[0092] Specifically, the actual computing intensity AI actual Calculate strength AI with balance point balance Compare and determine the actual bottleneck type of the target AI model:

[0093] When the actual computing intensity is greater than the equilibrium point computing intensity, it is judged that the actual bottleneck type of the target AI model during actual operation is the computing bottleneck, that is, AI actual >AI balance , then the actual operation is computationally limited;

[0094] When the actual computing intensity is less than the equilibrium point computing intensity, it is judged that the actual bottleneck type of the target AI model during actual operation is the bandwidth bottleneck, that is, AI actual <AI balance , then the actual operation is limited by memory;

[0095] When the actual computing intensity is equal to the equilibrium point computing intensity, it is judged that the target AI model is in a balanced state during actual operation, that is, AI actual ≈AI balance , the actual operation is in a balanced state.

[0096] S5. Perform a comprehensive judgment and diagnosis on the performance of the AI ​​accelerator based on the predicted theoretical bottleneck type, actual operation bottleneck type, actual performance, and predicted theoretical performance upper limit to obtain a performance fault diagnosis result of the AI ​​accelerator.

[0097] like Figure 4 As shown, the performance fault diagnosis logic diagram performs a comprehensive judgment and diagnosis on the performance of the AI ​​accelerator based on the predicted theoretical bottleneck type, the actual operation bottleneck type, the actual performance, and the predicted theoretical performance upper limit, including: performing a performance degradation analysis (Performance Degradation Analysis), comparing the actual operation performance with the theoretical performance upper limit or performance baseline. If it is lower than the preset first threshold, it is determined that there is performance degradation. Performing a bottleneck mismatch analysis (Bottleneck Mismatch Analysis), comparing the theoretical bottleneck type with the actual bottleneck type. If the bottleneck type is inconsistent, it is determined that there is a bottleneck mismatch. A comprehensive judgment is made based on the degree of performance degradation and the bottleneck mismatch situation to obtain the performance fault diagnosis result of the AI ​​accelerator. The performance fault diagnosis result of the AI ​​accelerator is graded according to the fault characteristics, for example, into normal, warning, moderate fault, severe fault and other levels.

[0098] S51. Performance degradation analysis: calculate the performance achievement rate based on the actual performance prediction and the theoretical performance upper limit, set a performance degradation threshold, and when the performance achievement rate is less than the performance degradation threshold, it is determined that there is a significant performance degradation of the AI ​​accelerator; when the performance achievement rate is greater than or equal to the performance degradation threshold, it is determined that there is no significant performance degradation of the AI ​​accelerator, and the performance degradation analysis result is obtained.

[0099] Specifically, the actual performance P actual Divide by the predicted theoretical upper performance limit P theory-max Get the performance achievement rate η P , performance achievement rate η P Expressed as:

[0100]

[0101] Among them, P actual Indicates actual performance, P theory-max Represents the theoretical upper bound of the predicted performance.

[0102] A performance degradation threshold is set. When the performance achievement rate is less than the performance degradation threshold, it is determined that the AI ​​accelerator performance has a significant performance degradation. In this embodiment, a performance degradation threshold θ is set. degrade , for example, setting θ degrade =0.85, when the performance achievement rate η p Less than the performance degradation threshold θ degrade , that is, η P <0.85, it is judged that the AI ​​accelerator performance has significantly declined; when the performance achievement rate η P Greater than or equal to the performance degradation threshold θ degrade , that is, η P ≥θ degrade , after determining that there is no significant performance degradation in the AI ​​accelerator.

[0103] S52. Bottleneck mismatch analysis: Determine whether there is a bottleneck mismatch based on whether the theoretical prediction is consistent with the actual operation bottleneck type; when the theoretical bottleneck type is inconsistent with the actual operation bottleneck type, determine that there is a bottleneck mismatch; otherwise, determine that the bottleneck matches, and obtain the bottleneck mismatch analysis result.

[0104] Compare the theoretical bottleneck types predicted in step 3 (based on AI theory vs AI balance ) and the actual operation bottleneck type determined in step 5 (based on AI actual vs AI balance When the theoretical bottleneck type and the actual bottleneck type do not match, a bottleneck mismatch is determined. Otherwise, a bottleneck match is determined, meaning the theoretical bottleneck type and the actual bottleneck type match. For example, if the theoretical prediction is compute-bound, but the actual runtime is memory-bound, this may indicate inefficient compute units or unexpected latency / bandwidth issues in the memory system.

[0105] S53. Combine the performance degradation analysis results and the bottleneck mismatch analysis results to comprehensively diagnose the performance of the AI ​​accelerator and obtain the performance fault diagnosis results of the AI ​​accelerator.

[0106] S531: When the AI ​​accelerator performance significantly declines and there is a bottleneck mismatch (for example, theoretical computing limitation becomes actual memory limitation), it is determined to be a serious performance failure, and the possible cause is a computing unit failure or efficiency problem.

[0107] S532: When the AI ​​accelerator performance shows a significant decline but the bottleneck matches (for example, both theoretically and practically it is memory-limited), it is judged as a moderate performance failure. The possible reasons are that the actual available value of the hardware subsystem related to the bottleneck (such as memory bandwidth) is lower than the calibration baseline, or there are other unmodeled system overheads.

[0108] S533, when the AI ​​accelerator performance does not show significant degradation (η P ≥θ degrade ) However, if there is a bottleneck mismatch, it is judged to be in a warning state. The possible reasons are that there is room for optimization in the model implementation, runtime library or scheduling strategy, or there is a slight change in the hardware status.

[0109] S534: When the AI ​​accelerator performance does not show significant degradation and the bottleneck is matched, it is determined to be operating normally.

[0110] After the diagnosis results are obtained, the diagnosis results including normal operation, warning status, moderate performance failure, severe performance failure, and possible causes are output to the monitoring system or log.

[0111] In conclusion, the present invention provides a method for diagnosing performance faults of AI accelerators. By obtaining the measured performance indicators of the AI ​​accelerator through the test suite of the MLPerf benchmark model, and combining it with Roofline to analyze the bottleneck mismatch, it can effectively distinguish between hardware anomalies and normal bottlenecks, thereby improving the accuracy of AI accelerator performance fault diagnosis. Based on the MLPerf standard, the diagnostic results are comparable. According to the performance degradation mode and the bottleneck mismatch type, the performance of the AI ​​accelerator is comprehensively judged and diagnosed to obtain the performance fault diagnosis results of the AI ​​accelerator, which can solve the problem in the prior art that the performance fault diagnosis of the AI ​​accelerator is not accurate enough and it is difficult to distinguish between hardware anomalies and normal bottlenecks.

[0112] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for diagnosing AI accelerator performance faults, characterized in that: The following steps are involved: S1. Obtain the measured performance indicators of the AI ​​accelerator using the MLPerf benchmark model test suite and calculate the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator. S2. Analyze the theoretical characteristics of the target AI model, obtain the theoretical computational load and theoretical memory access capacity of the target AI model, and calculate the theoretical computational intensity of the target AI model; S3. Determine the theoretical bottleneck type of the target AI model and use the Roofline model to predict the theoretical performance upper limit of the target AI model; S4. Run the target AI model on the AI ​​accelerator and collect the operating data of the AI ​​accelerator. Based on the collected operating data of the AI ​​accelerator, calculate the actual computing intensity and actual operating performance of the target AI model, and determine the actual operating bottleneck type of the target AI model. S5. Perform a comprehensive judgment and diagnosis on the performance of the AI ​​accelerator based on the predicted theoretical bottleneck type, actual operation bottleneck type, actual operation performance, and predicted theoretical performance upper limit to obtain the performance fault diagnosis result of the AI ​​accelerator.

2. The method for diagnosing AI accelerator performance faults according to claim 1, characterized in that: The step S1 includes: S11. Run the test suite of MLPerf benchmark models on an AI accelerator. The MLPerf benchmark models in the test suite include a compute-intensive model and a memory-intensive or hybrid model. S12. Run a computationally intensive model to test the computing capability of the AI ​​accelerator, record the measured benchmark model throughput, and calculate the effective peak computing performance of the AI ​​accelerator based on the measured benchmark model throughput; S13. Run a memory access or hybrid model to monitor the memory access status of the AI ​​accelerator at runtime, and calculate the effective memory bandwidth of the AI ​​accelerator based on the measured bandwidth data.

3. The method for diagnosing AI accelerator performance faults according to claim 1, characterized in that: The calculation formula for the theoretical computing strength of the target AI model is: Among them, AI theory is the theoretical computing intensity, FLOPs model-total is the theoretical total computational capacity, Bytes model-total is the theoretical total memory access amount.

4. The method for diagnosing AI accelerator performance faults according to claim 1, wherein: The step S3 comprises: S31. Calculate the balance point computing intensity of the AI ​​accelerator based on the effective peak computing performance and effective memory bandwidth of the AI ​​accelerator; S32. Compare the theoretical computational intensity of the target AI model with the computational intensity of the equilibrium point to determine the theoretical bottleneck type of the target AI model; S33. Based on the accelerator's effective memory bandwidth, effective peak computing performance, and the theoretical computing intensity of the target AI model, the Roofline model is used to predict the theoretical performance upper limit of the target AI model.

5. The method for diagnosing AI accelerator performance faults according to claim 4, characterized in that: The step of comparing the theoretical computational strength of the target AI model with the computational strength of the equilibrium point to determine the theoretical bottleneck type of the target AI model includes: When the theoretical computing intensity of the target AI model is greater than the computing intensity of the equilibrium point, the theoretical bottleneck type of the target AI model is determined to be a computing bottleneck; When the theoretical computing intensity of the target AI model is less than the computing intensity of the equilibrium point, the theoretical bottleneck type of the target AI model is determined to be a bandwidth bottleneck; When the theoretical computing intensity of the target AI model is equal to the computing intensity of the equilibrium point, it is judged that the theoretical bottleneck type of the target AI model is in a balanced state.

6. The method for diagnosing AI accelerator performance faults according to claim 4, characterized in that: The theoretical performance upper limit of the target AI model is expressed by the following formula: P thoery-max =min(P eff ,AI theory ×BW eff ); BW eff The effective memory bandwidth of AI accelerator, P eff The effective peak computing performance of AI accelerators, AI theory is the theoretical computational strength of the target AI model.

7. The method for diagnosing AI accelerator performance faults according to claim 1, characterized in that: The step S4 comprises: S41. Run the target AI model on the AI ​​accelerator, monitor the actual running status of the target AI model, and collect actual running performance data of the target AI model, where the actual running performance data includes: actual throughput, actual computing power, and actual memory access volume; S42. Calculate the actual computing intensity of the target AI model based on the actual memory access amount and the actual computing amount, and calculate the actual operating performance of the target AI model based on the actual computing amount; S43. Compare the actual computing intensity of the target AI model with the computing intensity of the equilibrium point to determine the actual bottleneck type of the target AI model.

8. The method for diagnosing AI accelerator performance faults according to claim 7, characterized in that: Comparing the actual computing intensity of the target AI model with the equilibrium point computing intensity to determine the actual bottleneck type of the target AI model includes: When the actual computing intensity is greater than the equilibrium point computing intensity, it is determined that the actual bottleneck type of the target AI model during actual operation is a computing bottleneck; When the actual computing intensity is less than the equilibrium point computing intensity, it is determined that the actual bottleneck type of the target AI model during actual operation is the bandwidth bottleneck; When the actual computing intensity is equal to the equilibrium point computing intensity, it is judged that the target AI model is in a balanced state during actual operation.

9. The method for diagnosing AI accelerator performance faults according to claim 1, characterized in that: The step S5 comprises: S51. Calculate the performance achievement rate based on the actual performance prediction and the theoretical performance upper limit, set a performance degradation threshold, and determine that the AI ​​accelerator performance has significantly degraded when the performance achievement rate is less than the performance degradation threshold; and determine that the AI ​​accelerator performance has not significantly degraded when the performance achievement rate is greater than or equal to the performance degradation threshold, thereby obtaining a performance degradation analysis result. S52. Determine whether there is a bottleneck mismatch based on whether the theoretical prediction and the actual operation bottleneck types are consistent, and obtain a bottleneck mismatch analysis result; if the theoretical bottleneck type is inconsistent with the actual operation bottleneck type, determine that there is a bottleneck mismatch; otherwise, determine that the bottleneck matches; S53. Combine the performance degradation analysis results and the bottleneck mismatch analysis results to comprehensively diagnose the performance of the AI ​​accelerator and obtain the performance fault diagnosis results of the AI ​​accelerator.

10. The method for diagnosing AI accelerator performance faults according to claim 9, characterized in that: The step S53 includes: When the AI ​​accelerator performance significantly declines and there is a bottleneck mismatch, it is considered a serious performance failure; When the AI ​​accelerator performance significantly declines but the bottleneck is matched, it is considered a moderate performance failure; When the AI ​​accelerator performance does not show significant degradation but there is a bottleneck mismatch, it is judged to be in a warning state; When the AI ​​accelerator performance does not show significant degradation and the bottleneck is matched, it is considered to be operating normally.

Images