Active testing methods, devices, equipment, media and products for AI computing clusters

Abstract

The embodiments of the present invention disclose active testing methods, devices, equipment, media and products for AI computing power clusters. The method includes: when it is determined that the time for active testing has arrived, predicting the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type; based on the fault probability of each node, determining the test node from the AI ​​computing power cluster; based on the possible fault type of the test node, determining a test subset matching the possible fault type from a test set containing N test tasks, the test subset containing M test tasks, where M and N are both positive integers, and M is less than or equal to N; based on the test subset, performing a test on the test node. Perform active testing before a fault occurs to improve the overall stability and efficiency of the AI ​​computing power cluster. Targetedly select test subsets based on prediction results to reduce the amount of test tasks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence (AI) technology, and more specifically, to active testing methods, devices, equipment, media and products for AI computing power clusters. Background Art

[0002] ‌AI computing cluster‌ is a collection of multiple nodes that are interconnected through a network to form a unified computing resource pool that can perform AI-related computing tasks. Each node can contain a high-performance processor, a large amount of memory, and a fast storage system, allowing the entire cluster to process large amounts of data and complex algorithms in parallel, thereby accelerating the training and reasoning process of AI models‌.

[0003] At present, fault detection is usually performed after an abnormality occurs in the AI ​​computing cluster to determine the fault type. However, this passive method of triggering detection after an abnormality occurs may affect the stability and efficiency of the AI ​​computing cluster (for example, affecting the real-time and continuity of the computing tasks being processed by the AI ​​computing cluster). Summary of the invention

[0004] The present invention proposes an active testing method, device, equipment, medium and product for an AI computing power cluster, which helps to improve the stability and efficiency of the AI ​​computing power cluster.

[0005] The technical solution of the embodiment of the present invention is as follows:

[0006] An active testing method for an AI computing cluster, comprising:

[0007] When it is determined that the time for active testing has arrived, predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type;

[0008] Based on the failure probability of each node, determining a test node from the AI ​​computing power cluster;

[0009] Based on the possible fault type of the test node, determine a test subset matching the possible fault type from a test set including N test tasks, the test subset including M test tasks, where M and N are both positive integers, and M is less than or equal to N;

[0010] Based on the test subset, a test is performed on the test node.

[0011] In one embodiment, the determining that the active test opportunity has arrived includes at least one of the following:

[0012] Determining that the AI ​​computing cluster is started for the first time;

[0013] Determining that the topology of the AI ​​computing power cluster has changed;

[0014] Determining that an active test instruction triggered by a user is received;

[0015] Determine that a predetermined active detection time point has been reached.

[0016] In one embodiment, the predicting of the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type includes:

[0017] Based on the fault prediction model of each node, predict the possible fault type of each node and the fault probability corresponding to the possible fault type;

[0018] The generation method of the fault prediction model of each node includes at least one of the following:

[0019] Based on the historical detection data of each node, a fault prediction model of each node is generated in a predetermined statistical manner;

[0020] Generate a fault prediction model for each node based on historical detection data of each node and a predetermined machine learning algorithm;

[0021] Based on the training data set of each node and a predetermined deep learning algorithm, an artificial neural network is trained to obtain a fault prediction model for each node.

[0022] In one embodiment, the fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution;

[0023] The fault prediction model based on each node predicts the possible fault type of each node and the fault probability corresponding to the possible fault type, including:

[0024] Real-time detection data of each node is acquired, and the real-time detection data is input into a fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the fault probability corresponding to the possible fault type based on the real-time detection data.

[0025] In one embodiment, determining a test node from the AI ​​computing power cluster based on the failure probability of each node includes:

[0026] Compare the failure probabilities of all possible failure types of each node with a predetermined first threshold value respectively;

[0027] Based on the comparison result, a test node is determined from the AI ​​computing power cluster, wherein the failure probability of at least one possible failure type of the test node is greater than or equal to the first threshold.

[0028] In one embodiment, the determining, based on the possible fault type of the test node, a test subset matching the possible fault type from a test set including N test tasks comprises:

[0029] The test subset is determined from the test set by a greedy algorithm, wherein the coverage of the test subset for the possible fault type meets a predetermined second threshold and the total running time of the test subset is minimized.

[0030] In one embodiment, performing a test on the test node based on the test subset includes:

[0031] Execute the M test tasks included in the test subset to obtain M test results corresponding to the M test tasks;

[0032] For each of the M test results, the similarity with the offline standard of the corresponding test task is calculated, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, it is determined that the corresponding test task has been passed, and when the similarity is less than the third threshold, it is determined that the corresponding test task has not been passed.

[0033] In one embodiment, it includes:

[0034] When the AI ​​computing power cluster is in the offline standard learning period, the test set including the N test tasks is executed in each node in the AI ​​computing power cluster, and based on the test results of the test set executed by all nodes, unsupervised offline standard learning is performed on each of the N test tasks to determine the offline standard of each test task.

[0035] In one embodiment, the performing unsupervised offline standard learning on each of the N test tasks based on the test results of all nodes executing the test set to determine the offline standard of each test task includes:

[0036] Remove erroneous test results from the test results of all nodes executing the same test task;

[0037] Based on the remaining test results after removing the erroneous test results, an offline standard for the test task is determined, wherein the similarity between each of the remaining test results and the offline standard is greater than or equal to a predetermined fourth threshold.

[0038] In one embodiment, the performing unsupervised offline standard learning on each of the N test tasks based on the test results of all nodes executing the test set to determine the offline standard of each test task includes:

[0039] Remove erroneous test results from the test results of all nodes executing the same test task;

[0040] determining a median of the remaining test results after removing the erroneous test results;

[0041] The median is determined as the offline standard of the test task.

[0042] In one embodiment, the test task includes at least one of the following:

[0043] A first test task of running a predetermined deep learning model on a node to indirectly test the hardware performance of the node;

[0044] The second test task directly tests the node hardware performance;

[0045] The third test task is to run a predetermined operator on the node to indirectly test the node hardware performance.

[0046] An active testing device for an AI computing cluster, comprising:

[0047] A prediction module, used to predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type when it is determined that the active test time has arrived;

[0048] A first determination module, configured to determine a test node from the AI ​​computing power cluster based on the failure probability of each node;

[0049] A second determination module is used to determine, based on the possible fault type of the test node, a test subset matching the possible fault type from a test set including N test tasks, wherein the test subset includes M test tasks, where M and N are both positive integers, and M is less than or equal to N;

[0050] A test module is used to execute a test on the test node based on the test subset.

[0051] In one embodiment, the prediction module is used to predict the possible fault type of each node and the fault probability corresponding to the possible fault type based on the fault prediction model of each node;

[0052] The generation method of the fault prediction model of each node includes at least one of the following:

[0053] Based on the historical detection data of each node, a fault prediction model of each node is generated in a predetermined statistical manner;

[0054] Generate a fault prediction model for each node based on historical detection data of each node and a predetermined machine learning algorithm;

[0055] Based on the training data set of each node and the predetermined deep learning algorithm, an artificial neural network is trained to obtain a fault prediction model of each node;

[0056] wherein the fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution;

[0057] The fault prediction model based on each node predicts the possible fault type of each node and the fault probability corresponding to the possible fault type, including:

[0058] Real-time detection data of each node is acquired, and the real-time detection data is input into a fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the fault probability corresponding to the possible fault type based on the real-time detection data.

[0059] In one embodiment, the test module is used to execute the M test tasks included in the test subset to obtain M test results corresponding to the M test tasks; for each of the M test results, calculate the similarity with the offline standard of the corresponding test task, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, it is determined that the corresponding test task has been passed, and when the similarity is less than the third threshold, it is determined that the corresponding test task has not been passed.

[0060] An electronic device, comprising:

[0061] Memory;

[0062] processor;

[0063] The memory stores an application program executable by the processor, which is used to enable the processor to execute any of the active testing methods for the AI ​​computing power cluster described above.

[0064] A computer-readable storage medium having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by a processor, the processor executes any of the above-described active testing methods for an AI computing power cluster.

[0065] A program product includes a computer program, which, when executed by a processor, implements any of the above-described active testing methods for an AI computing power cluster.

[0066] It can be seen from the above technical solution that when it is determined that the time for active testing has arrived, the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type are predicted; based on the fault probability of each node, the test node is determined from the AI ​​computing power cluster; based on the possible fault type of the test node, a test subset matching the possible fault type is determined from a test set containing N test tasks, and the test subset contains M test tasks, where M and N are both positive integers, and M is less than or equal to N; based on the test subset, the test node is tested. Therefore, performing active testing on nodes that may fail before a failure occurs can improve the overall stability and efficiency of the AI ​​computing power cluster, such as solving the problem of performance degradation caused by redundancy. Moreover, the fault prediction model of the node and the offline standard of the test task generated based on multiple methods have flexible applicability. In addition, the test subset is selected from the test set based on the prediction results to reduce the amount of test tasks. In addition, multiple types of test tasks enrich the test content. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is an exemplary flowchart of an active testing method for an AI computing power cluster according to an embodiment of the present invention.

[0068] Figure 2 This is an exemplary topological diagram of the active testing framework of the AI ​​computing power cluster according to an embodiment of the present invention.

[0069] Figure 3 FIG. 4 is an exemplary schematic diagram of a process of determining a test subset based on a greedy algorithm according to an embodiment of the present invention.

[0070] Figure 4 This is an exemplary structural diagram of an active testing device for an AI computing power cluster according to an embodiment of the present invention.

[0071] Figure 5 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0073] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0074] The rapid development and widespread application of deep learning technology have driven the expansion of cloud AI infrastructure. With the surge in demand for computing resources, cloud service providers need to invest high costs to build and maintain AI computing clusters. AI cluster computing power usually includes multiple nodes (for example, computing nodes and storage nodes). Frequent communication and data exchange are required between these nodes. If the network is inefficient, communication between nodes becomes slow, which will directly affect the computing power of the AI ​​cluster. Moreover, as AI computing clusters continue to expand, how to maintain high system reliability has become an important challenge. The scheduling and synchronization characteristics of AI workloads mean that once workload failures or performance degradation occur in the AI ​​computing cluster, these problems may quickly spread to all nodes, thereby exacerbating the adverse effects on system reliability.

[0075] In an AI computing cluster, node failure or performance degradation will affect the overall workload. The troubleshooting process is time-consuming and costly, and all related nodes are in a waiting state during problem solving, resulting in a waste of GPU resources.

[0076] After research, it was found that the main causes of failures in AI computing clusters include: (1) Rapid hardware development: AI hardware defects may not be discovered in vendor tests because some defects only appear under specific workloads. (2) Differences in cloud environments: Cloud infrastructure may cause additional failures due to environmental factors such as power and temperature and different test environments. For example, data centers in tropical regions face more problems due to high temperatures, and the defect rate of network links is much higher than that in cold regions, etc. (3) Frequent software iterations: In order to adapt to hardware development and new functions, AI software stacks often need to be iteratively updated, which makes it difficult to ensure the stability and reliability of the software. AI workloads lack resilience to software and hardware failures, and small failures may spread rapidly. To improve the reliability of AI cluster computing power, a variety of hardware redundancy measures have been designed, including redundant GPU computing units, GPU memory row remapping, and over-configured network links. Although these redundancy measures can improve the fault tolerance of the system, they may also accidentally introduce so-called "gray failures" or "slow failures", that is, performance gradually degrades rather than fails immediately, which makes passive fault detection more complicated and difficult.

[0077] At present, the fault detection technologies of AI computing clusters mainly include: (1) Data analysis and pattern recognition: Using AI technology to process massive operation and maintenance data, identifying fault patterns through machine learning algorithms, and then issuing warnings. (2) Real-time monitoring and anomaly detection: The AI ​​system continuously monitors the infrastructure, analyzes performance indicators and log information in real time, and immediately notifies the operation and maintenance personnel once an anomaly is detected. (3) Adaptive learning and optimization: The AI ​​system has the ability to self-learn and adapt, and can continuously optimize its prediction model with the accumulation of time and data. (4) Automated decision support: The AI ​​system can provide decision support based on historical experience and real-time data, helping operation and maintenance personnel quickly locate problems and choose solutions. (5) Intelligent operation and maintenance framework: An integrated intelligent operation and maintenance framework is proposed, which uses machine learning algorithms to analyze historical data, monitor key performance indicators in real time, and trigger automated repair processes. (6) GPU fault detection and self-healing: For the fault detection and self-healing capabilities of GPU hardware in AI scenarios, hardware fault events can be quickly discovered and alarmed, and self-healing recovery operations can be performed.

[0078] However, the above-mentioned existing passive fault detection methods have difficulty in determining the root cause of each fault and restoring system reliability. For example, performance degradation problems caused by redundancy may not be discovered and resolved in a timely manner. In distributed AI systems, due to the large number of nodes and layers, some nodes may not provide appropriate error information when a fault occurs, and need to be investigated by elimination. In addition, workload events may be closely related to the application and not necessarily directly related to the AI ​​computing power cluster. Due to asynchronous execution in GPUs and adaptive routing in networks (for example, InfiniBand networks), faults may be non-deterministic and difficult to reproduce. Even if the fault is successfully located and the current fault is resolved, it is difficult to identify all potential problems. In fact, when a fault occurs, it may indicate that there are already multiple accumulated faults across various components, resulting in a decrease in overall reliability.

[0079] Therefore, passive troubleshooting alone cannot completely solve the above problems, and more advanced methods are needed to improve the reliability and fault detection capabilities of AI computing clusters.

[0080] Although cloud service providers have taken hardware pre-testing and troubleshooting measures, these methods have limitations and cannot improve system reliability. Pre-testing cannot solve reliability problems in cloud environments, while troubleshooting depends on specific workloads, is time-consuming and usually reactive, and cannot proactively solve faults before they affect workloads.

[0081] To improve the reliability of cloud AI infrastructure, the embodiments of the present invention adopt an active detection strategy to prevent failures by verifying nodes before they occur. This can improve the reliability of the system within a reasonable cost by balancing the detection cost and expected failure losses. Through active testing, actions can be taken before failures occur, thereby improving the overall stability and efficiency of the AI ​​computing cluster.

[0082] In order to effectively reduce the mean time between events (MTBI), the active testing strategy of the embodiments of the present invention can meet the following conditions: (1) Comprehensiveness: The testing process is as comprehensive as possible to detect problems that are not discovered in the vendor test and only appear in the workload. This requires verification covering a wide range of AI workload types. (2) Clarity: Given that hardware components may gradually degrade and performance measurements are susceptible to natural changes, clear standards need to be set to distinguish normal performance from defective performance, that is, test results should be consistent and avoid fluctuations. (3) Cost-effectiveness: Although active testing requires additional measurement costs, these costs are much lower than the potential losses associated with failures.

[0083] There are challenges in achieving these conditions: (1) During the verification process, it becomes unrealistic to fully cover all scenarios due to the wide variety of workloads and the exponential growth of node combinations; (2) There is a lack of basic test tasks to determine which nodes are defective, because hardware specifications cannot reliably predict workload performance; (3) The significant variation in AI hardware increases the complexity of distinguishing normal performance from defective performance. Verification time and MTBI may be interdependent, requiring the longest MTBI to be achieved in the least measurement time, which increases the difficulty of strategy design.

[0084] To overcome these challenges, experimental analysis yields the following observations: (1) Although AI workloads are diverse, most workloads can be represented by a small number of typical workloads. (2) By analyzing all intermediate data (or their distribution) in a single test run, rather than relying solely on average metrics, clearer boundaries can be established to maximize the difference between normal and defective values. (3) The failure probability of a node increases with age. If a node has recently been fully validated with a benchmark, a small number of test tasks may be sufficient to cover most potential issues.

[0085] Based on the observations, the active testing of the embodiments of the present invention can follow the following principles: (1) Data-driven approach: Utilizing a large amount of defective nodes and test result data in the cloud AI infrastructure, a data-driven approach is adopted to fit the event probability model and develop statistical standards. (2) Fast and frequent active testing: Active testing can be an ongoing process, and each node should be tested quickly but frequently based on event probability estimates to ensure long-term reliability.

[0086] The above disclosure details the technical defects existing in the related art, the causes of the technical defects, and the thinking and analysis process of overcoming the technical defects. In fact, the cognition of the above technical defects is not common knowledge in the field, but a novel discovery made by the inventor in his research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the inventor in the actual research process, and are not common knowledge in the field.

[0087] Figure 1 FIG. 1 is an exemplary flow chart of an active testing method of an AI computing cluster according to an embodiment of the present invention. For example, the method may be executed by a controller of an AI computing cluster. Figure 1 As shown, the method includes:

[0088] Step 101: When it is determined that the time for active testing has arrived, predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type.

[0089] In one embodiment, determining that an active test opportunity has been reached includes at least one of the following:

[0090] (1) Determine that the AI ​​computing cluster is being started for the first time; (2) Determine that the topology of the AI ​​computing cluster has changed (for example, new nodes have been added, network connections have changed, etc.); (3) Determine that an active test instruction triggered by a user has been received; (4) Determine that a predetermined active detection time point has been reached (for example, the active detection time point is a periodic cycle time point).

[0091] In one embodiment, predicting the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type includes: predicting the possible fault type of each node and the fault probability corresponding to the possible fault type based on the fault prediction model of each node. In other words, for each node in the AI ​​computing power cluster, a respective fault prediction model is generated. Then, based on each fault prediction model corresponding to each node one by one, the possible fault type of the corresponding node and the fault probability corresponding to the possible fault type are predicted.

[0092] Specifically, the fault prediction model of each node is generated in a manner including at least one of the following:

[0093] (1) Based on the historical detection data of each node, a fault prediction model for each node is generated in a predetermined statistical manner.

[0094] Here, a fault prediction model is generated based on the historical detection data and statistical analysis methods of each node. For example, through historical detection data and statistical algorithms, the node operation trend, abnormal points and statistical laws are analyzed to predict the node failure. For example, statistical analysis methods may include: Negative Binomial Regression (NBR) analysis method, Bayesian method, fuzzy logic method or Dempster-Shafer theory, etc.

[0095] (2) Generate a fault prediction model for each node based on the historical detection data of each node and a predetermined machine learning algorithm.

[0096] Machine learning algorithms can learn the normal operation mode of nodes through historical detection data, and predict potential failures based on the learned normal operation mode. For example, commonly used machine learning algorithms may include decision trees, vector machines, and random forest algorithms, etc.

[0097] (3) Based on the training data set of each node and the predetermined deep learning algorithm, the artificial neural network is trained to obtain the fault prediction model of each node.

[0098] Here, the operation mode and failure mode of the node can be learned through a multi-layer neural network to train and obtain a failure prediction model for each node.

[0099] The above exemplary description is a typical example of a method for generating a fault prediction model for determining the timing and node of reaching active testing. Those skilled in the art will appreciate that such description is merely exemplary and is not intended to limit the scope of protection of the embodiments of the present invention.

[0100] In one embodiment, the fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution; based on the fault prediction model of each node, predicting the possible fault type of the node and the fault probability corresponding to the possible fault type includes: obtaining real-time detection data of each node (for example, the real-time detection data may include real-time performance indicators (for example, the rate, bandwidth, throughput, time delay and utilization of the node, etc.) and real-time log information), inputting the real-time detection data into the fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the fault probability corresponding to the possible fault type based on the real-time detection data. In this embodiment, it is assumed that all nodes in the AI ​​computing power cluster follow the same probability distribution of fault events. Therefore, a unified function can be used to predict the probability that each node may experience a failure during the operation of the workload. This method not only simplifies the prediction process, but also improves the generalization ability of the model, so that it can be applied to all nodes in the AI ​​computing power cluster.

[0101] Step 102: Based on the failure probability of each node, determine the test node from the AI ​​computing power cluster.

[0102] In one embodiment, step 102 specifically includes: comparing the failure probability of all possible failure types of each node with a predetermined first threshold value; based on the comparison result, determining a test node from the AI ​​computing power cluster, wherein the failure probability of at least one possible failure type of the test node is greater than or equal to the first threshold value.

[0103] For example: Assume that the AI ​​computing cluster contains nodes 1, 2, 3, and 4.

[0104] Through the fault prediction model of node 1, it is predicted that the possible fault types and probabilities of node 1 include: type A with a probability of 0.3; type B with a probability of 0.2; type C with a probability of 0.5.

[0105] Through the fault prediction model of node 2, it is predicted that the possible fault types and probabilities of node 2 include: type A with a probability of 0.1; type B with a probability of 0.1; type C with a probability of 0.1.

[0106] Through the fault prediction model of node 3, it is predicted that the possible fault types and probabilities of node 3 include: type A with a probability of 0.2; type B with a probability of 0.5; type C with a probability of 0.6; type D with a probability of 0.1.

[0107] Through the fault prediction model of node 4, it is predicted that the possible fault types and probabilities of node 4 include: type A with a probability of 0.2; type B with a probability of 0.2; type C with a probability of 0.1; type D with a probability of 0.1.

[0108] Assume that the predetermined first threshold is 0.4.

[0109] It can be seen that the probability of type C of node 1 (0.5) is greater than the first threshold, so node 1 is determined as a test node. The probabilities of the types of node 2 are all less than the first threshold, so node 2 is not determined as a test node. The probability of type B (0.5) and the probability of type C (0.6) of node 3 are both greater than the first threshold, so node 3 is determined as a test node. The probabilities of the types of node 4 are all less than the first threshold, so node 4 is not determined as a test node.

[0110] Step 103: Based on the possible fault type of the test node, determine a test subset matching the possible fault type from a test set including N test tasks, where the test subset includes M test tasks, where M and N are both positive integers, and M is less than or equal to N.

[0111] In the process of building a comprehensive test suite, the goal is to ensure that the test process can cover all known fault types and can adapt to new unknown fault types. To achieve this goal, the test suite can include end-to-end test tasks, basic test tasks, and module-level micro-basic test tasks, etc.

[0112] In one embodiment, the test task includes at least one of the following:

[0113] (1) Running a predetermined deep learning model (e.g., a machine vision model) on a node to indirectly test the first test task (i.e., the above-mentioned end-to-end test task) of the node hardware performance.

[0114] (2) The second test task (i.e. the basic test task mentioned above) directly tests the node hardware performance.

[0115] (3) The third test task (i.e., the module-level micro-basic test task) is to run the predetermined operators on the node to indirectly test the node hardware performance.

[0116] The above three test tasks are described in detail below.

[0117] (I): End-to-end testing tasks:

[0118] End-to-end test tasks can be built based on a deep understanding of the workload. By analyzing the workload distribution on the AI ​​training platform, typical end-to-end deep learning models can be identified that represent the actual workload, thus helping to prevent potential regression issues. For example, a variety of basic models including CNN, BERT, and LLAMA can be selected as deep learning models to run on the node, and common settings can be selected based on the parameter configurations of these models in actual use (such as batch size and sequence length) to optimize the use of GPU resources and ensure the convergence of training.

[0119] Since workloads are constantly changing, the well-defined end-to-end test tasks may not fully represent future workloads. Therefore, the test set is preferably extensible so that when new workloads or private workloads running in a specific cluster appear, the corresponding models can be added to the test set to maintain the relevance and effectiveness of the verification process.

[0120] (II) Basic test tasks:

[0121] In addition to end-to-end test tasks, the test set can also include basic test tasks to evaluate and simulate rare workload behaviors that may exhibit unique patterns when accessing specific hardware components. Use basic test tasks to evaluate individual hardware components such as CPU, memory, GPU, NIC / HCA, disk, and their interconnections. For example, third-party tools certified by hardware vendors can be used to implement basic test tasks.

[0122] (III) Module-level micro-foundation testing tasks:

[0123] In addition to end-to-end test tasks and basic test tasks, the test set can also include module-level micro-basic test tasks. Module-level micro-basic test tasks simulate common workload patterns or primitives involving multiple nodes. For example, using common shapes of kernel operators and All-Reduce (All-Reduce is a communication operation commonly used in parallel and distributed computing, mainly used to aggregate data between multiple computing nodes (such as GPUs or CPUs). It is often used in the distributed training process of deep learning to integrate gradient information on each node for parameter updates) or common message sizes of many-to-many (All-to-All) collective communication primitives. These patterns and primitives are derived from offline analysis of typical workloads and are continuously enriched and improved through regression cases in diagnostic experience.

[0124] In this way, it is possible to build a test set that covers typical workloads as well as rare and emerging workloads, thus ensuring the comprehensiveness and forward-looking nature of the verification process.

[0125] In one embodiment, based on the possible fault type of the test node, determining a test subset that matches the possible fault type from a test set containing N test tasks includes: determining the test subset from the test set using a greedy algorithm, wherein the coverage of the test subset for the possible fault type satisfies a predetermined second threshold and the total running time of the test subset is minimized.

[0126] Here, the total running time of the test subset is minimized under the premise that the coverage of the test subset for possible fault types meets the predetermined requirements. That is, a set of test tasks is selected from the test set so that the total running time of this set of test tasks is minimized, and this set of test tasks ensures a certain coverage for possible fault types. This problem can be regarded as an optimization problem, in which the best balance needs to be found between coverage and running time. The optimal solution for this optimization problem can be determined based on greedy algorithms and other methods. Among them: coverage is the percentage of defective nodes of possible fault types that have been historically identified by the test subset to all defective nodes of the possible fault type (including unidentified defective nodes).

[0127] Step 104: Based on the test subset, execute the test on the test node.

[0128] In one embodiment, based on the test subset, executing the test on the test node includes:

[0129] (1) Execute the M test tasks included in the test subset and obtain M test results corresponding to the M test tasks;

[0130] (2) For each of the M test results, the similarity with the offline standard of the corresponding test task is calculated respectively, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, it is determined that the corresponding test task has been passed; when the similarity is less than the third threshold, it is determined that the corresponding test task has not been passed.

[0131] In one embodiment:

[0132] (1) When the specific data types of the test result and the offline standard are both numerical, the similarity between the above test result and the offline standard of the corresponding test task can be implemented as: a numerical parameter related to the difference between the test result and the offline standard. For example: the smaller the difference, the larger the numerical parameter (i.e., the greater the similarity); the larger the difference, the smaller the numerical parameter (i.e., the smaller the similarity).

[0133] (2) When the specific data types of the test results and the offline standard are both text, the similarity between the above test results and the offline standard of the corresponding test task can be implemented as: text similarity. For example, the text similarity can be determined by calculating the cosine distance, the longest common subsequence, the Euclidean distance, the Pearson correlation coefficient, etc.

[0134] (3) When the specific data types of the test result and the offline standard are both images, the similarity between the above test result and the offline standard of the corresponding test task can be implemented as: image similarity. For example, the image similarity can be determined by histogram comparison, cosine similarity, hash algorithm, mean square error (MSE), structural similarity (SSIM) and feature matching.

[0135] For test nodes that fail the corresponding test tasks, corresponding alarm prompts can be issued. Based on the alarm prompts corresponding to the test tasks, users can understand the nodes that may fail and the possible failure types of the nodes, so as to intervene in advance to solve the possible failures of the nodes.

[0136] In the first test during cluster construction, the complete test set can be run in the AI ​​computing cluster, and unsupervised offline standard learning can be performed on each test task based on the test results on all nodes to determine the offline standard for each test task.

[0137] In one embodiment, the method also includes: executing a test set including N test tasks in each node in the AI ​​computing power cluster, and based on the test results of executing the test set by all nodes, performing unsupervised offline standard learning on each of the N test tasks to determine the offline standard of each test task.

[0138] In one embodiment, based on the test results of the test set executed by all nodes, unsupervised offline standard learning is performed on each test task in N test tasks to determine the offline standard of each test task, including: removing erroneous test results from all test results of all nodes executing the same test task (for example, removing erroneous test results based on a clustering algorithm); determining the offline standard of the test task based on the remaining test results after removing the erroneous test results, wherein the similarity between each test result in the remaining test results and the offline standard is greater than or equal to a predetermined fourth threshold.

[0139] For example, in the first test during cluster construction, n nodes all execute test task C to obtain the test results S of n nodes. 1 ~S n Based on n test results S 1 ~S n , calculate the offline standard S of the test task CC . Offline Standard S C It is used to distinguish defective result samples from normal result samples. Specifically, the offline standard S of the test task C is calculated C The process includes: (1): From n test results S 1 ~S n Remove the wrong test results and assume that the remaining test results are S 1 ~S k Based on S 1 ~S k , calculate the offline standard S of the test task C C , where for S 1 ~S k Any test result S in i , both satisfy the similarity (S C , S i )>α, where α is an empirical value set according to requirements, i.e., the fourth threshold; similarity (S C ,S i ) is S C With S i The similarity between .

[0140] In one embodiment, based on the test results of the test set executed by all nodes, unsupervised offline standard learning is performed on each test task in N test tasks to determine the offline standard of each test task, including: removing erroneous test results from all test results of all nodes executing the same test task (for example, removing erroneous test results based on a clustering algorithm); determining the median of the remaining test results after removing the erroneous test results; and determining the median as the offline standard of the test task.

[0141] For example, in the first test during cluster construction, n nodes all execute test task C to obtain the test results S of n nodes. 1 ~S n Based on n test results S 1 ~S n , calculate the offline standard S of the test task C C , which will be used to distinguish defective result samples from normal result samples later.

[0142] Specifically, calculate the offline standard S of the test task C C The process includes:

[0143] (1): From n test results S 1 ~S n Remove the wrong test results and assume that the remaining test results are S 1 ~S k .

[0144] (2): In S 1 ~S k Determine the median, assuming it is S 5 , then S 5 is the offline standard for the test task C.

[0145] Figure 2 FIG. 1 is an exemplary topological diagram of an active testing framework of an AI computing cluster according to an embodiment of the present invention. Figure 2 As shown, the active test framework can include a controller, a generator, and an executor.

[0146] The controller can actively or periodically start health check tasks based on the topology changes of the AI ​​computing cluster. The controller can also perform node fault prediction model training tasks and update the offline standards of each test task in the test set.

[0147] The generator determines the test subset in the test set according to the predicted possible fault types of the test nodes, and generates a test execution script for executing each test task in the test subset. The generator can use the node fault prediction model of each node to predict the node fault according to the real-time status of the node, select the test node from all nodes based on the fault probability, and correspondingly select one or more test tasks covering the possible fault types of the test node for each test node, and the one or more test tasks constitute the test subset.

[0148] The executor calls the test execution script to execute each test task in the test subset of each test node separately, and compares it with the offline standard of the test task (the offline standard is obtained based on offline calculation) to screen out defective nodes from the test nodes and feed them back to the platform.

[0149] Specifically, the overall process may include:

[0150] (1): Test set construction: The test set contains different types of test tasks. Each test task can include test tools, test objects, and expected golden indicators.

[0151] Test tasks can include multiple types. For example: (1) One type is a basic test task based on a tool that performs tests on the basic functions and health status of specific hardware. The test content of the basic test task includes CPU, memory, GPU computing unit, GPU memory, network interface card (NIC), disk and their interconnections, such as PCIe and network links. (2) Another type is an end-to-end test task, which selects typical end-to-end artificial intelligence models to perform tests on behalf of customer workloads to ensure the reliability of the infrastructure under specific customer workloads, thereby preventing potential regressions. The end-to-end artificial intelligence models in the end-to-end test task can include different CNN models and Transformer models such as BERT and GPT, and can also include LLM models that adopt multiple parallel strategies (such as the LLaMA2 model using 3D hybrid parallelism).

[0152] The test set is extensible, and basic test tasks and end-to-end test tasks can be added as needed. After the test set is determined, the standard indicator information (i.e., offline standard) of each test task is obtained through test statistics in a healthy hardware environment for comparison during subsequent online testing.

[0153] (2) Training process of node fault prediction model: In order to train the node fault prediction model, it is necessary to collect a certain amount of node detection history data. Specifically, it includes: the time since the last fault event when the node was detected, the historical fault event count, the average fault event interval between different fault event categories, etc. Through these data, the node fault event distribution is modeled as a time-based continuous function. Assuming that all nodes in a large-scale cluster share the same event probability distribution, the same continuous function can be used to predict the probability of each node experiencing an event during the operation of the customer workload. The controller uses the node fault prediction model to determine the detection nodes that need to perform active detection based on pre-set trigger strategies, such as regular or new node access, topology changes of the AI ​​computing cluster, etc.

[0154] Specifically, when designing the fault node prediction function, it is first necessary to ensure that sufficient historical detection data can be collected. The historical detection data can provide insights into the node failure behavior, so that the node failure event distribution can be modeled as a continuous function of time. It can be assumed that all nodes in the AI ​​computing power cluster follow the same event probability distribution. The controller can use the fault prediction model according to the preset trigger strategy. These trigger strategies may include regular inspections, access to new nodes, topology changes of the AI ​​computing power cluster, and active triggering after receiving user instructions, etc. For example, when a new node is added to the cluster, the controller can predict the possible fault type and probability of the new node based on the fault prediction model of the node. If the node is predicted to have a high risk of failure, the controller can trigger active detection of the node to ensure that it does not affect the stability of the entire cluster. Similarly, when the computing power configuration of the cluster changes, the controller can also re-evaluate the failure probability of the relevant nodes based on the fault prediction model of the relevant nodes and arrange detection as needed. To further improve the accuracy of the prediction, the output of the node's fault prediction model can be combined with the real-time monitoring data of the node. Real-time monitoring data provides instant information about the current state of the node, while the prediction model provides long-term trends based on historical data. Through this combination, a more comprehensive understanding of the health status of the nodes can be obtained, and preventive measures can be taken before failures occur. In addition, a feedback mechanism can be designed to feed back actual failure events into the failure prediction model so that the failure prediction model can be continuously optimized and adjusted. This continuous learning and improvement process will make the model more accurate over time, thereby improving the reliability of the entire system. In summary, the design of the fault node prediction function involves data collection, model building, controller design, and the establishment of a feedback mechanism. Through these steps, the failure probability of the node can be predicted, and health detection can be automatically triggered when necessary, thereby ensuring the stable operation of the cluster.

[0155] (3) Selecting a test subset from the test set: When the test node set containing the test node has been determined, the generator selects a test subset corresponding to each detection node from the test set, and generates a test execution script for executing each test task in the corresponding test subset on the detection node. When selecting a test subset in the generator, more abnormal scenarios can be covered with as little time cost as possible, thereby achieving a balance between detection cost and detection effect.

[0156] The core goal of selecting a test subset from a test set is to select a set of test tasks that minimizes the total running time of the set of test tasks while ensuring a certain coverage. This problem can be viewed as an optimization problem, where the best balance needs to be found between coverage and running time.

[0157] Problem definition:

[0158] Test set: Given a set of test tasks B={B 1 ,…,B n}, each test task B i There is a running time t i Defective node set: a set of historically identified defective nodes {M i}. Coverage: The percentage of defective nodes historically identified by the test subset to all defective nodes. Coverage C: The ratio of the number of defective nodes covered by the test subset to the total number of defective nodes. Event probability: The probability of a fault event being found by the test subset, expressed as p×C, where p is the original probability of the fault event.

[0159] Goal: Find a test subset that reduces the probability of a new event: p×(1−C) to p 0 (a preset threshold) while minimizing the total running time. Problem Complexity: This problem is NP-hard because it can be viewed as a variant of the 0-1 knapsack problem, where different items (test tasks) may overlap (cover the same defective nodes). Algorithm Selection: A greedy algorithm is proposed with a time complexity of O(n 2 ), instead of using O(2 n ) complexity to traverse all possible combinations.

[0160] Figure 3 FIG. 2 is an exemplary schematic diagram of a process of determining a test subset based on a greedy algorithm according to an embodiment of the present invention. Figure 3 As shown, the process of determining the test subset includes:

[0161] Step 301: Set an empty test subset S and a total running time T, where T=0.

[0162] Step 302: Test task sorting: sort all test tasks in the test set according to the decreasing suboptimal probability of each test task (ie, the additional coverage brought by each test task).

[0163] The selection process includes steps 303 to 308. Specifically:

[0164] Step 303: traverse the sorted test tasks.

[0165] Step 304: For each base test task B i , calculate the new coverage C′ and total running time T′ after adding S.

[0166] Step 305: Determine whether the condition is met. If so, execute step 307 and its subsequent steps. Otherwise, execute step 306 and its subsequent steps. For example, the condition can be: (p×(1−C′)≤p 0 and T′ is acceptable). Where: p 0 is a predetermined threshold; p is the original probability of a fault event; p×(1−C′) is the event probability at the new coverage C′ after S is added; T′ is the total running time at the new coverage C′ after S is added. For example, when T′ is less than a preset threshold, T′ can be considered acceptable.

[0167] Step 306: Continue to the next test task.

[0168] Step 307: B i Add to S and update T.

[0169] Step 308: Determine whether all test tasks have been processed. If so, execute step 309 and its subsequent steps; otherwise, execute step 303 and its subsequent steps.

[0170] Step 309: The termination condition is met (no more test tasks can be added without violating the event probability threshold p 0 or all test tasks have been considered complete).

[0171] Step 310: Output the benchmark test subset S and the total running time T.

[0172] It can be seen that compared with exhaustively enumerating all possible combinations, the greedy algorithm significantly reduces the computation time. Moreover, by dynamically selecting test tasks, it can adapt to different test requirements and constraints. In addition, the greedy algorithm can be adjusted according to actual conditions, for example, by changing the way of sorting test tasks or adjusting p 0 to optimize performance.

[0173] The test subset selection algorithm described above is an efficient greedy strategy that solves the problem of selecting effective benchmarks in large-scale systems by minimizing the total runtime while ensuring that coverage meets a preset threshold. This approach is particularly suitable for scenarios that require fast decision-making and resource optimization.

[0174] (4) Test node execution test: The executor executes the test execution script generated by the generator to execute each test task in the test subset of the test node on the test node, and compares it with the offline standard of the corresponding test task to analyze the health status of the node. Considering that the end-to-end test task will generate a set of time series indicators, such as the number of tokens per GPU per second (TGS) or loss value of each step, a specific algorithm can be used to determine whether the difference is within the normal range, otherwise it is easy to cause false positives.

[0175] The executor is responsible for executing the test execution script output by the generator, and automatically determines whether the node is defective based on the test results, and provides feedback to the platform. The processing of the executor includes the following points: (a) Test task execution: Execute the selected test subsets on the test node set to generate test results. (b) Offline standard learning: In the first verification during cluster construction, the complete test set is run on all nodes, and based on the test results of all nodes, unsupervised offline standard learning is performed for each test task to obtain the offline standard for each test task. (c) Similarity definition: Define the similarity between two test result samples, including but not limited to single value samples from micro-benchmark tasks, or time series samples from end-to-end benchmark tasks. (d) Empirical cumulative distribution function (CDF): For test samples of two time series, use CDF to define their distance and similarity in the distribution space, where the distance d(S 1 ,S 2 ) represents the absolute integral area between their CDF curves and is normalized to the range [0,1]. (e) Offline standard calculation: Based on the similarity, the offline standard S of the test task is calculated. C , used to distinguish defective result samples and normal result samples of the test task in subsequent detection. For any sample S other than defective samples in the test results of offline standard learning, i , S C Should satisfy the similarity (S C , S i )>α, where α is an empirical value set according to requirements. (f) Application of clustering algorithm: Apply a similarity-based clustering algorithm to exclude defective samples and calculate the median sample of the remaining samples as the offline standard S C (g) Online defect screening: In active testing after offline standard learning, the similarity between the test results and their offline standards is used to determine whether a node should be considered defective. The observed test results are compared with the offline standard S C: The one-way distance between them. (h) Similarity calculation: If the similarity is greater than the threshold, it is considered normal. (i) Repeatability metric: Define the repeatability metric as: the arithmetic mean of the pairwise similarities from N different nodes or runs to determine whether the benchmark setting is acceptable within the empirical similarity threshold α. (j) Adaptive adjustment of test tasks: Adaptively adjust the test tasks to ensure repeatability and minimize the impact of natural variations when running the same workload on different runs or nodes. (k) Enhance the repeatability of test tasks: Enhance the repeatability of test tasks by decoupling different hardware components, adaptively searching test tasks, and re-adjusting and evaluating repeatability after firmware / driver updates. (h) Test nodes: Efficiently test nodes by maximizing the repeatability metric to ensure that the measured samples accurately represent the underlying distribution of hardware performance and variations.

[0176] (5) Test result feedback: The test results of the test tasks in the test subset are recorded in the historical records. If there are abnormal nodes, they are fed back to the operation and maintenance personnel for troubleshooting and repair, and abnormal labels are attached to prevent abnormal nodes from being introduced into the scheduling process and causing training errors. The judgment results of the model are confirmed or corrected through the troubleshooting of the operation and maintenance personnel, and the records are updated. The historical records can be used to update the training of the node fault prediction model.

[0177] (6) Update the node failure prediction model: In order to continuously improve the accuracy of the prediction model of the node health status, it is necessary to train new models based on the increasingly rich test verification and operation and maintenance result data to achieve continuous iteration.

[0178] It can be seen that the embodiment of the present invention proposes an active detection strategy to prevent failures by verifying nodes before failures occur, thereby improving the reliability and fault detection capabilities of the cloud AI computing cluster. The technical solution of the embodiment of the present invention includes: (1) Overall design: a system architecture including a controller, a generator and an executor to realize active startup of health check tasks, generation of benchmark strategies and execution of benchmarks. (2) Test set design: building a comprehensive test set, including end-to-end benchmark tasks and micro-benchmark tasks, etc., to cover a wide range of AI workload types and hardware components. (3) Fault node prediction: using historical detection data, establishing a node fault prediction model to predict the failure type and probability of the node during the operation of the customer workload. (4) Test subset generation: developing an efficient greedy algorithm to select a test subset that minimizes the total running time while ensuring that the coverage meets the preset threshold. (5) Test execution and feedback: the executor executes the test and automatically determines the health status of the node, and feeds back to the platform to optimize the prediction model. The embodiments of the present invention are based on a data-driven approach and a fast and frequent verification strategy. Through experimental analysis and historical data, an active verification system and method are designed to take action before problems arise, thereby improving the overall stability and efficiency of the cloud AI computing cluster.

[0179] The embodiment of the present invention also proposes an active testing device for an AI computing power cluster. Figure 4 FIG. 1 is an exemplary structural diagram of an active testing device for an AI computing cluster according to an embodiment of the present invention. Figure 4 As shown, the active testing device 400 of the AI ​​computing power cluster includes: a prediction module 401, which is used to predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type when it is determined that the time for active testing has arrived; a first determination module 402, which is used to determine the test node from the AI ​​computing power cluster based on the failure probability of each node; a second determination module 403, which is used to determine a test subset matching the possible fault type from a test set containing N test tasks based on the possible fault type of the test node, the test subset containing M test tasks, wherein M and N are both positive integers, and M is less than or equal to N; a test module 404, which is used to perform a test on the test node based on the test subset.

[0180] In one embodiment, determining that the time for active testing has been reached includes at least one of the following: determining that the AI ​​computing power cluster is started for the first time; determining that the topology structure of the AI ​​computing power cluster has changed; determining that an active test instruction triggered by a user is received; and determining that a predetermined active detection time point has been reached.

[0181] In one embodiment, the prediction module 401 is used to predict the possible fault type of each node and the fault probability corresponding to the possible fault type based on the fault prediction model of each node; wherein the generation method of the fault prediction model of each node includes at least one of the following: based on the historical detection data of each node, generating the fault prediction model of each node in a predetermined statistical manner; based on the historical detection data of each node and a predetermined machine learning algorithm, generating the fault prediction model of each node; based on the training data set of each node and a predetermined deep learning algorithm, training the artificial neural network to obtain the fault prediction model of each node.

[0182] In one embodiment, the fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution; the prediction module 401 is used to obtain the real-time detection data of each node, and input the real-time detection data into the fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the failure probability corresponding to the possible fault type based on the real-time detection data.

[0183] In one embodiment, the first determination module 402 is used to compare the failure probability of all possible failure types of each node with a predetermined first threshold value; based on the comparison result, determine a test node from the AI ​​computing power cluster, wherein the failure probability of at least one possible failure type of the test node is greater than or equal to the first threshold value.

[0184] In one embodiment, the second determination module 403 is used to determine a test subset from the test set using a greedy algorithm, wherein the coverage of the test subset for possible fault types meets a predetermined second threshold and the total running time of the test subset is minimized.

[0185] In one embodiment, the test module 404 is used to execute M test tasks included in the test subset to obtain M test results corresponding to the M test tasks; for each test result in the M test results, the similarity with the offline standard of the corresponding test task is calculated, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, it is determined that the corresponding test task has been passed, and when the similarity is less than the third threshold, it is determined that the corresponding test task has not been passed.

[0186] In one embodiment, the testing module 404 is used to execute a test set containing N test tasks in each node in the AI ​​computing power cluster when the AI ​​computing power cluster is in an offline standard learning period, and based on the test results of the test set executed by all nodes, perform unsupervised offline standard learning on each of the N test tasks to determine the offline standard of each test task.

[0187] In one embodiment, the test module 404 is used to remove erroneous test results from the test results of all nodes executing the same test task; based on the remaining test results after removing the erroneous test results, determine the offline standard of the test task, wherein the similarity between each test result in the remaining test results and the offline standard is greater than or equal to a predetermined fourth threshold.

[0188] In one embodiment, the test module 404 is used to remove erroneous test results from the test results of all nodes executing the same test task; determine the median of the remaining test results after removing the erroneous test results; and use the median to determine the offline standard of the test task.

[0189] In one embodiment, the test task includes at least one of the following: a first test task of running a predetermined deep learning model on the node to indirectly test the node hardware performance; a second test task of directly testing the node hardware performance; and a third test task of running a predetermined operator on the node to indirectly test the node hardware performance.

[0190] To sum up, in an embodiment of the present invention, when it is determined that the time for active testing has arrived, the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type are predicted; based on the fault probability of each node, the test node is determined from the AI ​​computing power cluster; based on the possible fault type of the test node, a test subset matching the possible fault type is determined from a test set containing N test tasks, the test subset containing M test tasks, where M and N are both positive integers, and M is less than or equal to N; based on the test subset, the test node is tested. Therefore, performing active testing before a fault occurs can improve the overall stability and efficiency of the AI ​​computing power cluster, for example, it can solve the problem of performance degradation caused by redundancy. Moreover, the fault prediction model of the node and the offline standard of the test task are generated based on multiple methods, which has flexible applicability. In addition, the test subset is selected from the test set based on the prediction results to reduce the amount of test tasks. In addition, multiple types of test tasks realize multi-dimensional performance testing.

[0191] The embodiment of the present invention further provides an electronic device having a processor-memory architecture. Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 5As shown, the electronic device includes a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the computer program is executed by the processor 501, any of the above active testing methods for the AI ​​computing power cluster is implemented. Among them, the memory 502 can be specifically implemented as a variety of storage media such as an electrically erasable programmable read-only memory (EEPROM), a flash memory (Flash memory), and a programmable program read-only memory (PROM). The processor 501 can be implemented to include one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or the central processing unit core can be implemented as a CPU, a GPU, a GPGPU, an MCU or a DSP, and so on.

[0192] It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0193] The hardware modules in each embodiment may be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) to perform a specific operation. For example, specific operations can be performed in various types of chips (for example, artificial intelligence chips). The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software to perform specific operations. As for whether to implement the hardware module mechanically, or using a dedicated permanent circuit, or a temporarily configured circuit (such as configured by software), it can be decided based on cost and time considerations.

[0194] The present invention also provides a machine-readable storage medium, storing instructions for a machine to perform a method as described in the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the embodiments described in the above embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system operated on the computer can also be made to complete part or all of the actual operations by instructions based on the program code. The program code read out from the storage medium can also be written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then the CPU installed on the expansion board or the expansion unit is made to perform part and all of the actual operations based on the instructions of the program code, thereby realizing the functions of any of the embodiments described in the above embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0195] In this article, "schematic" means "serving as an example, instance or explanation", and any diagram or implementation method described as "schematic" in this article should not be interpreted as a more preferred or more advantageous technical solution. In order to make the drawings concise, only the parts related to the present invention are schematically shown in each figure, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some figures, only one of the parts with the same structure or function is schematically drawn, or only one of them is marked. In this article, "one" does not mean that the number of the relevant parts of the present invention is limited to "only one", and "one" does not mean that the number of the relevant parts of the present invention is "more than one". In this article, "upper", "lower", "front", "back", "left", "right", "inside", "outside", etc. are only used to indicate the relative position relationship between the relevant parts, rather than to limit the absolute position of these relevant parts.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An active testing method for an AI computing cluster, characterized in that: include: When it is determined that the time for active testing has arrived, predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type; Based on the failure probability of each node, determining a test node from the AI ​​computing power cluster; Based on the possible fault type of the test node, determine a test subset matching the possible fault type from a test set including N test tasks, the test subset including M test tasks, where M and N are both positive integers, and M is less than or equal to N; Based on the test subset, executing a test on the test node; The performing the test on the test node based on the test subset comprises: Execute the M test tasks included in the test subset to obtain M test results corresponding to the M test tasks; For each of the M test results, calculate the similarity with the offline standard of the corresponding test task, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, determine that the corresponding test task is passed, and when the similarity is less than the third threshold, determine that the corresponding test task is not passed; When the AI ​​computing power cluster is in an offline standard learning period, the test set including the N test tasks is executed in each node in the AI ​​computing power cluster, and based on the test results of all nodes executing the test set, unsupervised offline standard learning is performed on each of the N test tasks to determine the offline standard of each test task; The offline criteria for determining each test task include: Remove erroneous test results from the test results of all nodes executing the same test task; Based on the remaining test results after removing the erroneous test results, determine the offline standard of the test task, wherein the similarity between each test result in the remaining test results and the offline standard is greater than or equal to a predetermined fourth threshold; or, determine the median of the remaining test results after removing the erroneous test results; and determine the median as the offline standard of the test task.

2. The method according to claim 1, characterized in that The determining that the active test opportunity has arrived includes at least one of the following: Determining that the AI ​​computing cluster is started for the first time; Determining that the topology of the AI ​​computing power cluster has changed; Determining that an active test instruction triggered by a user is received; Determine that a predetermined active detection time point has been reached.

3. The method according to claim 1, characterized in that The predicted possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type include: Based on the fault prediction model of each node, predict the possible fault type of each node and the fault probability corresponding to the possible fault type; The generation method of the fault prediction model of each node includes at least one of the following: Based on the historical detection data of each node, a fault prediction model of each node is generated in a predetermined statistical manner; Generate a fault prediction model for each node based on historical detection data of each node and a predetermined machine learning algorithm; Based on the training data set of each node and a predetermined deep learning algorithm, an artificial neural network is trained to obtain a fault prediction model for each node.

4. The method according to claim 3, characterized in that The fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution; The fault prediction model based on each node predicts the possible fault type of each node and the fault probability corresponding to the possible fault type, including: Real-time detection data of each node is acquired, and the real-time detection data is input into a fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the fault probability corresponding to the possible fault type based on the real-time detection data.

5. The method according to claim 1, characterized in that The determining of the test node from the AI ​​computing power cluster based on the failure probability of each node includes: Compare the failure probabilities of all possible failure types of each node with a predetermined first threshold value respectively; Based on the comparison result, a test node is determined from the AI ​​computing power cluster, wherein the failure probability of at least one possible failure type of the test node is greater than or equal to the first threshold.

6. The method according to claim 1, characterized in that The determining, based on the possible fault type of the test node, a test subset matching the possible fault type from a test set including N test tasks comprises: The test subset is determined from the test set by a greedy algorithm, wherein the coverage of the test subset for the possible fault type meets a predetermined second threshold and the total running time of the test subset is minimized.

7. The method according to any one of claims 1 to 6, characterized in that The test task includes at least one of the following: A first test task of running a predetermined deep learning model on a node to indirectly test the hardware performance of the node; The second test task directly tests the node hardware performance; The third test task is to run a predetermined operator on the node to indirectly test the node hardware performance.

8. An active testing device for an AI computing cluster, characterized in that: include: A prediction module, used to predict the possible fault type of each node in the AI ​​computing power cluster and the fault probability corresponding to the possible fault type when it is determined that the active test time has arrived; A first determination module, configured to determine a test node from the AI ​​computing power cluster based on the failure probability of each node; A second determination module is used to determine, based on the possible fault type of the test node, a test subset matching the possible fault type from a test set including N test tasks, wherein the test subset includes M test tasks, where M and N are both positive integers, and M is less than or equal to N; A test module, configured to perform a test on the test node based on the test subset; The test module is used to execute the M test tasks included in the test subset to obtain M test results corresponding to the M test tasks; for each of the M test results, calculate the similarity with the offline standard of the corresponding test task, wherein when the similarity is greater than or equal to a third threshold associated with the corresponding test task, it is determined that the corresponding test task has been passed, and when the similarity is less than the third threshold, it is determined that the corresponding test task has not been passed; wherein when the AI ​​computing power cluster is in an offline standard learning period, the test set including the N test tasks is executed in each node in the AI ​​computing power cluster, and based on the test results of executing the test set on all nodes, unsupervised offline standard learning is performed on each of the N test tasks to determine the offline standard of each test task; The method for determining the offline standard for each test task includes: removing erroneous test results from the test results of all nodes executing the same test task; determining the offline standard for the test task based on the remaining test results after removing the erroneous test results, wherein the similarity between each of the remaining test results and the offline standard is greater than or equal to a predetermined fourth threshold; or determining the median of the remaining test results after removing the erroneous test results; and determining the median as the offline standard for the test task.

9. The device according to claim 8, characterized in that The prediction module is used to predict the possible fault type of each node and the fault probability corresponding to the possible fault type based on the fault prediction model of each node; The generation method of the fault prediction model of each node includes at least one of the following: Based on the historical detection data of each node, a fault prediction model of each node is generated in a predetermined statistical manner; Generate a fault prediction model for each node based on historical detection data of each node and a predetermined machine learning algorithm; Based on the training data set of each node and the predetermined deep learning algorithm, an artificial neural network is trained to obtain a fault prediction model of each node; wherein the fault prediction model of each node generated in a predetermined statistical manner has the same event probability distribution; The fault prediction model based on each node predicts the possible fault type of each node and the fault probability corresponding to the possible fault type, including: Real-time detection data of each node is acquired, and the real-time detection data is input into a fault prediction model of each node, so that the fault prediction model of each node predicts the possible fault type and the fault probability corresponding to the possible fault type based on the real-time detection data.

10. An electronic device, characterized in that: include: Memory; processor; The memory stores an application program executable by the processor, which is used to enable the processor to execute the active testing method for the AI ​​computing power cluster as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, cause the processor to execute the active testing method for an AI computing power cluster as described in any one of claims 1 to 7.

12. A program product comprising a computer program, characterized in that When the computer program is executed by a processor, the active testing method for the AI ​​computing power cluster as described in any one of claims 1 to 7 is implemented.

Images