Performance detection method and device for cluster system, electronic equipment and medium

Abstract

The application provides a performance detection method and device for a cluster system, electronic equipment and a medium. The method comprises the following steps: submitting M first test tasks to a computing cluster in a first round of testing, each first test task occupying P target nodes; determining whether there is a candidate abnormal node in each target node according to the task completion state of each first test task; submitting K second test tasks to the computing cluster in a second round of testing, each second test task occupying a node pair, each node pair comprising a candidate node and a normal node in the first round of testing; the candidate node comprises a candidate abnormal node and / or a remaining node in the first round of testing, the remaining node being a node other than the target node occupied by each first test task in N nodes; and determining whether there is an abnormal node in the computing cluster according to the task completion state of each second test task. In this way, the efficiency and accuracy of computing cluster detection can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a performance testing method, apparatus, electronic device and storage medium for cluster systems. Background Technology

[0002] With the development of artificial intelligence (AI) technology, computing clusters built on AI chips have been widely used in various scenarios, such as finance, supercomputing, and energy. As the number of computing clusters increases, their maintenance has become a key concern.

[0003] Currently, static script detection or manual binary search is commonly used to check the status of each node in a computing cluster. However, the static script method for node status detection cannot adapt to the dynamic and complex interaction scenarios in the actual operation of the computing framework, and is prone to missed detections and false positives. When using the manual binary search method, the detection time increases exponentially with the cluster size, and the operational efficiency is difficult to meet the real-time requirements of large-scale clusters. Summary of the Invention

[0004] In view of this, this application provides a performance testing method, apparatus, electronic device and storage medium for cluster systems, in order to at least solve the technical problems existing in the related art.

[0005] According to a first aspect of this application, a performance testing method for a cluster system is provided, the cluster system including a computing cluster, the computing cluster including N nodes, where N is a positive integer greater than 1; the method includes:

[0006] In the first round of testing, M first test tasks are submitted to the computing cluster, each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1;

[0007] Based on the task completion status of each of the first test tasks, determine whether there are any candidate abnormal nodes among the target nodes;

[0008] In the second round of testing, K second test tasks are submitted to the computing cluster. Each second test task occupies a node pair. Each node pair includes a candidate node and a normal node from the first round of testing. The candidate node includes candidate abnormal nodes from the first round of testing and / or remaining nodes from the first round of testing. The remaining nodes refer to the nodes other than the target nodes occupied by each of the N nodes.

[0009] Based on the task completion status of each of the second test tasks, determine whether there are any abnormal nodes in the computing cluster.

[0010] In one alternative implementation, M is obtained by rounding down the ratio of N to P; and / or, K is determined based on the sum of the number of candidate abnormal nodes and the number of remaining nodes.

[0011] In one optional implementation, determining whether there are candidate abnormal nodes among the target nodes based on the task completion status of each of the first test tasks includes:

[0012] For each of the first test tasks, if the task status of the first test task is not "completed", the target node corresponding to the first test task is determined as the candidate abnormal node.

[0013] In one optional implementation, determining whether there are abnormal nodes in the computing cluster based on the task completion status of each of the second test tasks includes:

[0014] For each second test task, if the task status of the second test task is not "completed", the candidate nodes in the node pair corresponding to the second test task are identified as the abnormal nodes.

[0015] In an optional implementation, the cluster system further includes a computing framework running on top of the computing cluster, the computing framework being used to invoke at least one of the normal nodes to perform computing tasks according to the model training task of the target model; the method further includes:

[0016] During the execution of the model training task, the model floating-point operation utilization rate of the computing framework is determined;

[0017] Determine the operational status information of the computing framework for the target model under different combinations of parallel parameters;

[0018] Determine the operational stability of the computing framework within a preset time period;

[0019] Based on the floating-point operation utilization, the running status information, and the running stability, it is determined whether the performance, compatibility, and stability of the computing framework meet the preset requirements.

[0020] In one alternative implementation, when determining the model floating-point operation utilization of the computing framework, iterative data from the model training warm-up phase and iterative data from the breakpoint saving phase are excluded.

[0021] In an optional implementation, the method further includes:

[0022] Obtain task failure logs, which are generated by any of the nodes after a task fails to execute;

[0023] The task failure logs are matched against a preset rule base based on preset matching rules to obtain attribution matching results; the rule base includes the mapping relationship between failure logs and attribution domains.

[0024] In an optional implementation, the method further includes:

[0025] If the attribution matching result indicates that no corresponding attribution domain is matched, the retrieval enhancement generation model is invoked to analyze and process the task failure log to obtain the attribution result.

[0026] According to a second aspect of this application, a performance testing device for a cluster system is provided, the device comprising:

[0027] The first task submission module is used to submit M first test tasks to the computing cluster in the first round of testing. Each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1.

[0028] The first node determination module is used to determine whether there are candidate abnormal nodes among the target nodes based on the task completion status of each of the first test tasks.

[0029] The second task submission module is used to submit K second test tasks to the computing cluster in the second round of testing. Each second test task occupies a node pair, and each node pair includes a candidate node and a normal node in the first round of testing. The candidate node includes candidate abnormal nodes in the first round of testing and / or remaining nodes in the first round of testing. The remaining nodes refer to the nodes other than the target nodes occupied by each of the N nodes.

[0030] The second node determination module is used to determine whether there are abnormal nodes in the computing cluster based on the task completion status of each second test task.

[0031] In one alternative implementation, M is obtained by rounding down the ratio of N to P; and / or, K is determined based on the sum of the number of candidate abnormal nodes and the number of remaining nodes.

[0032] In one optional implementation, the first node determination module is specifically used for:

[0033] For each of the first test tasks, if the task status of the first test task is not "completed", the target node corresponding to the first test task is determined as the candidate abnormal node.

[0034] In one optional implementation, the second node determining module is specifically used for:

[0035] For each second test task, if the task status of the second test task is not "completed", the candidate nodes in the node pair corresponding to the second test task are identified as the abnormal nodes.

[0036] In an optional implementation, the cluster system further includes a computing framework running on top of the computing cluster, the computing framework being used to invoke at least one of the normal nodes to execute computing tasks according to the model training task of the target model; the device further includes a framework performance detection module, the framework performance detection module being used to:

[0037] During the execution of the model training task, the model floating-point operation utilization rate of the computing framework is determined;

[0038] Determine the operational status information of the computing framework for the target model under different combinations of parallel parameters;

[0039] Determine the operational stability of the computing framework within a preset time period;

[0040] Based on the floating-point operation utilization, the running status information, and the running stability, it is determined whether the performance, compatibility, and stability of the computing framework meet the preset requirements.

[0041] In one optional implementation, the framework performance detection module is specifically used for:

[0042] When determining the model floating-point operation utilization of the computing framework, iterative data from the model training warm-up phase and the breakpoint saving phase are excluded.

[0043] In an optional embodiment, the apparatus further includes a fault attribution determination module, the fault attribution determination module being configured to:

[0044] Obtain task failure logs, which are generated by any of the nodes after a task fails to execute;

[0045] The task failure logs are matched against a preset rule base based on preset matching rules to obtain attribution matching results; the rule base includes the mapping relationship between failure logs and attribution domains.

[0046] In an optional implementation, the fault attribution determination module is further configured to:

[0047] If the attribution matching result indicates that no corresponding attribution domain is matched, the retrieval enhancement generation model is invoked to analyze and process the task failure log to obtain the attribution result.

[0048] According to a third aspect of this application, an electronic device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the performance detection method for cluster systems described in the first aspect are performed.

[0049] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the performance testing method for cluster systems described in the first aspect.

[0050] The performance testing method, apparatus, electronic device, and storage medium for cluster systems provided in this application submit M test tasks in the first round of testing, with each task occupying P target nodes. Based on the task status of the target nodes, normal nodes and candidate abnormal nodes are initially screened. Then, in the second round of testing, the remaining nodes from the first round and each candidate abnormal node are paired with a normal node, and two-node tasks are submitted for cross-validation. Finally, based on the task status of each node pair, the existence of an abnormal node is determined. This improves the accuracy of detection compared to static script detection methods and improves the efficiency compared to manual binary search methods. In other words, the performance testing method for cluster systems in this application not only improves detection accuracy but also detection efficiency.

[0051] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0053] Figure 1This is a flowchart illustrating a performance testing method for cluster systems according to an exemplary embodiment of this application;

[0054] Figure 2 This is a schematic diagram illustrating a node anomaly detection process for a cluster system, as shown in an exemplary embodiment of this application.

[0055] Figure 3 This is a schematic diagram illustrating the relationship between floating-point operation utilization and iteration count in a model according to an exemplary embodiment of this application;

[0056] Figure 4 This is a functional block diagram of a performance testing device for a cluster system, as illustrated in an exemplary embodiment of this application.

[0057] Figure 5 This is a functional block diagram of another performance testing device for cluster systems illustrated in an exemplary embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0061] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0062] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0063] With the development of artificial intelligence technology, computing clusters built on AI chips have been widely used. Research has found that current methods typically employ static script detection or manual binary search to check the status of each node within the computing cluster. However, the static script method for node status detection cannot adapt to the dynamic and complex interaction scenarios in actual operation of the computing framework, and is prone to missed detections and false positives. When using the manual binary search method, the detection time increases exponentially with the cluster size, making it difficult to meet the real-time requirements of large-scale clusters in terms of operational efficiency.

[0064] Based on the above research, this application provides a performance testing method for a cluster system, wherein the cluster system includes a computing cluster with N nodes, where N is a positive integer greater than 1; the method includes: in a first round of testing, submitting M first test tasks to the computing cluster, each first test task occupying P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1; determining whether there are candidate abnormal nodes among the target nodes based on the task completion status of each first test task; in a second round of testing, submitting K second test tasks to the computing cluster, each second test task occupying a node pair, each node pair including a candidate node and a normal node from the first round of testing; the candidate node includes the candidate abnormal node from the first round of testing and / or the remaining node from the first round of testing, the remaining node referring to the nodes other than the target nodes occupied by each of the first test tasks among the N nodes; determining whether there are abnormal nodes in the computing cluster based on the task completion status of each second test task.

[0065] In this embodiment, M test tasks are submitted in the first round of testing, each task occupying P target nodes. Normal nodes and candidate abnormal nodes are initially screened based on the task status of the target nodes. Then, in the second round of testing, the remaining nodes from the first round and each candidate abnormal node are paired with a normal node, and dual-node tasks are submitted for cross-validation. Finally, the existence of an abnormal node is determined based on the task status of each node pair. This improves the accuracy of detection compared to static script detection methods and improves the efficiency compared to manual binary search methods. In other words, the performance testing method for cluster systems in this application not only improves detection accuracy but also detection efficiency.

[0066] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0067] To facilitate understanding of this embodiment, the executing entity of the performance testing method for cluster systems will first be introduced. The executing entity of this performance testing method for cluster systems can be an electronic device. The electronic device can include a terminal device or a server. The terminal device can include mobile devices, user terminals, terminals, in-vehicle devices, computing devices, and wearable devices, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms.

[0068] In some possible implementations, this performance testing method for cluster systems can be implemented by the processor calling computer-readable instructions stored in memory.

[0069] See Figure 1 The diagram shown is a flowchart of a performance testing method for a cluster system provided in an embodiment of this application. The cluster system includes a computing cluster, which includes N nodes, where N is a positive integer greater than 1. The method includes the following steps S101 to S104:

[0070] S101, In the first round of testing, M first test tasks are submitted to the computing cluster, each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1.

[0071] For example, please refer to the following: Figure 2The computing cluster can be built based on AI chips. The cluster can include multiple nodes, each of which can be a single computing server within the cluster, and each node can include eight GPUs (Graphics Processing Units). The nodes can be interconnected using the RoCE protocol, an RDMA (Remote Direct Memory Access) network protocol that allows direct access to remote memory over Ethernet, enabling high bandwidth, low latency, and low CPU (Central Processing Unit) overhead. In other embodiments, each node can include multiple dedicated processors (Neural Processing Units, NPUs) designed for the computing needs of artificial intelligence.

[0072] For example, the first test task can be a model training task. For instance, in this embodiment, a large language model training task (the first test task) can be run, with each first test task having 4 pods (target nodes). In other embodiments, the number of target nodes for each first test task can be 8 or 16, etc., and there is no specific limitation thereto.

[0073] Here, M can be obtained by rounding down the ratio of N to P. For example, if the cluster includes 131 nodes N, then M can be N / P, or 131 / 4, and then rounded down to 32. This allows for the submission of 32 first test tasks. Determining the value of M by rounding down the ratio of N to P maximizes the number of target nodes in the first round of testing, thereby improving testing efficiency.

[0074] S102, based on the task completion status of each of the first test tasks, determine whether there are candidate abnormal nodes among the target nodes.

[0075] The process of determining whether there are candidate abnormal nodes among the target nodes includes: determining which of the target nodes are normal nodes and which are candidate abnormal nodes.

[0076] After submitting the first test task to its corresponding target node, the task status can be checked after a preset time. For example, this preset time can include 5 minutes, 8 minutes, 10 minutes, etc. That is, the status query time points can be set based on the 50th percentile (5 minutes), 80th percentile (8 minutes), and twice the 50th percentile (10 minutes) of the task completion time, and the task status of each node can be checked at these time points. If all task statuses are in a final state, the process can be exited early. The final state can include "completed" and "failed" states. For each test task, if the task status is "completed," the target node occupied by that first test task is determined to be a normal node; if the task status is still not "completed" at the last time point, the target node occupied by that first test task is determined to be a candidate abnormal node. The non-"completed" states include "creating," "running," and "failed" states.

[0077] In some embodiments, when determining whether there are candidate abnormal nodes among the target nodes based on the task completion status of each first test task, it may include: for each first test task, if the task status of the first test task is not "completed", determining the target node corresponding to the first test task as the candidate abnormal node.

[0078] For example, regarding the aforementioned 32 first test tasks, if 29 of them successfully run, then the 116 target nodes associated with these 29 first test tasks are considered normal nodes. The remaining 12 target nodes corresponding to the other 3 first test tasks are considered candidate abnormal nodes, totaling 3 × 4 = 12. Since there are 3 remaining nodes from the first round of testing, these 3 remaining nodes and the 12 candidate abnormal nodes can be used as candidate nodes for the next round of testing to further verify their performance.

[0079] It is understood that in some embodiments, after the first round of testing, it is necessary to determine whether the number of candidate nodes is less than the number of normal nodes, and if the number of candidate nodes is less than the number of normal nodes, proceed to the next round of task submission and judgment. For example, since the number of candidate nodes 15 is less than the number of normal nodes 116, the second round of testing can proceed. In some embodiments, if there are no candidate nodes in the first round of testing (no remaining nodes and no candidate abnormal nodes), then the second round of testing is not required.

[0080] S103, in the second round of testing, K second test tasks are submitted to the computing cluster. Each second test task occupies a node pair. Each node pair includes a candidate node and a normal node from the first round of testing. The candidate node includes the candidate abnormal node from the first round of testing and / or the remaining node from the first round of testing. The remaining node refers to the other nodes among the N nodes besides the target node occupied by each of the first test tasks.

[0081] For example, K can be determined based on the sum of the number of candidate abnormal nodes and the number of remaining nodes. For instance, in conjunction with the foregoing embodiments, if the number of candidate abnormal nodes is 12 and the number of remaining nodes is 3, then K is 15; if there are no candidate abnormal nodes in the first round of testing, then K is determined by the number of remaining nodes, that is, K is 3; if the total number of nodes N is 128, and all nodes participated in the first round of testing, with no remaining nodes, then K is determined by the number of candidate abnormal nodes, 12, that is, K is 12.

[0082] It should be noted that since a candidate node can form a node pair with a normal node, the number of node pairs is the same as the number of candidate nodes. For example, if there are 15 candidate nodes, the number of node pairs is 15.

[0083] S104, Based on the task completion status of each of the second test tasks, determine whether there are any abnormal nodes in the computing cluster.

[0084] Determining whether there are abnormal nodes in the computing cluster includes: determining which nodes in the computing cluster are abnormal.

[0085] Here, since a node pair contains a normal node and a candidate node, and the normal node has already passed the verification in the first round of verification, in the second round of testing, if the task status of the second test task is completed, it means that the node pair executing the second test task are all normal nodes. If the second test task fails, the candidate node in the node pair is determined to be an abnormal node.

[0086] In some embodiments, determining whether there are abnormal nodes in the computing cluster based on the task completion status of each of the second test tasks may include:

[0087] For each second test task, if the task status of the second test task is not "completed", the candidate nodes in the node pair corresponding to the second test task are identified as the abnormal nodes.

[0088] For example, if each candidate node is paired with a normal node and 15 second test tasks (dual-node tasks) are submitted, 11 tasks will succeed and 4 tasks will fail (not completed). Then, the candidate nodes in the node pairs occupied by the 4 failed tasks are abnormal nodes.

[0089] In this embodiment, a two-round dynamic task triggering mechanism is adopted. The first round submits a multi-node task, setting query time windows based on runtime to initially screen node status. The second round pairs candidate abnormal nodes and remaining nodes with normal nodes, submitting dual-node tasks for cross-validation, which improves the accuracy of abnormal node confirmation and monitoring efficiency. In actual verification, using the method of this application, node detection efficiency can be reduced from 117 minutes to 19 minutes.

[0090] In some embodiments, the cluster system further includes a computing framework running on top of the computing cluster, the computing framework being used to invoke at least one of the normal nodes to perform computing tasks according to the model training task of the target model; the method further includes the following (a) to (d):

[0091] (a) During the execution of the model training task, determine the model floating-point operation utilization of the computing framework;

[0092] (b) Determine the operational status information of the computational framework for the target model under different combinations of parallel parameters;

[0093] (c) Determine the operational stability of the computing framework within a preset time period;

[0094] (d) Based on the floating-point operation utilization, the running status information, and the running stability, determine whether the performance, compatibility, and stability of the computing framework meet the preset requirements.

[0095] Specifically, the performance of the computing framework can be determined to meet the preset performance requirements if the utilization rate of each type of floating-point operation, the running status information, and the running stability all meet the preset performance requirements.

[0096] Understandably, requirements for performance, compatibility, and stability can be set separately, depending on actual needs; no specific limitations will be made here.

[0097] See Figure 3 As shown, Figure 3The horizontal axis represents the number of model iterations, and the vertical axis represents the model's floating-point utilization rate. When determining the model's floating-point utilization rate of the computational framework, iteration data from the model training warm-up phase and the checkpoint saving phase can be excluded. That is, the performance metric of the computational framework is based on model floating-point utilization (MFU), excluding iteration data from the training warm-up phase and the checkpoint saving phase. This improves the accuracy of MFU determination.

[0098] For example, the target model may include Dense models and MoE (Mixture of Experts) models, etc.

[0099] The parallel parameter combinations can include various combinations of tensor parallelism (TP) and pipeline parallelism (PP). For example, for the DeepSeek-MoE-16B model, tests and verifications were performed with (TP, PP) combinations of (1,1), (1,2), (2,1), and (2,2).

[0100] The training warm-up phase refers to the stage in which the learning rate is gradually increased in the early stages of training to improve the model's convergence stability. The breakpoint saving phase refers to the stage of periodically saving the model and training state to support resumption of training after interruption. For example, the training warm-up phase refers to the process of gradually increasing the learning rate from a small value (usually close to 0) linearly (or according to a specific strategy) to a preset initial learning rate in the initial few steps (or epochs) of model training. The breakpoint saving phase refers to the periodic saving of the model state (weights, optimizer state, current epoch / step, etc.) to disk during training so that training can be resumed from the point of interruption after training interruption (such as power failure, OutOfMemoryError, manual pause) without having to start from the beginning.

[0101] In addition, for operational stability testing, taking a large language model as an example, the number of iterations can be increased (e.g., 10,000 steps) to simulate long-term load, such as running for 72 hours, and the results can be detected (whether the operation crashes).

[0102] In this embodiment, taking the large language model as an example, it has been verified that the MFU of version v2.3 (70.5%) is greater than that of version v2.2 (70.3%), and the performance meets the requirements. Furthermore, through compatibility verification of distributed strategy combinations such as TP and PP, performance regressions or hidden crashes in the computing framework during version iterations can be identified. In addition, regarding stability testing, the large language model can run continuously for 72 hours without crashing, and its stability meets the preset requirements.

[0103] The performance testing method for cluster systems provided in this application can not only detect the status of each node in the computing cluster and the stability of the computing framework, but also analyze and process fault attribution. Therefore, in some embodiments, the method further includes the following (I) to (II):

[0104] (I) Obtain the task failure log, which is generated by any of the nodes after the task execution fails;

[0105] (II) The task failure logs are matched against a preset rule base based on preset matching rules to obtain attribution matching results; the rule base includes the mapping relationship between failure logs and attribution domains.

[0106] For example, historical failed task logs can be obtained first, then keywords can be extracted from these logs and attributions assigned, forming a mapping relationship between logs and attribution domains. For instance, failed task log A corresponds to attribution domain X, failed task log B to attribution domain Y, failed task log C to attribution domain Z, and so on. Based on these mapping relationships, a pre-defined rule base can be obtained. After obtaining the current failed task logs, the AC automaton algorithm can be used to match the rules and obtain the matching results.

[0107] For example, if the current failed task log matches a historical failed task log A, then the attribution domain for the current failed task log is the attribution domain X corresponding to failed task log A. If no match is found, the Retrieval Augmented Generation (RAG) model can be invoked to analyze and process the log. This RAG model can include a rule base imported as a knowledge base and a large language model to handle task failure logs where no rules were matched.

[0108] Therefore, in some embodiments, the method further includes:

[0109] If the attribution matching result indicates that no corresponding attribution domain is matched, the retrieval enhancement generation model is invoked to analyze and process the task failure log to obtain the attribution result.

[0110] In addition to obtaining the attribution results, the corresponding confidence level will also be obtained. For example, in the implementation, the request body is in JSON format and contains the field "error_log (failure log)", and the response body contains the fields "domain (domain)" and "confidence (confidence level)".

[0111] For example, inputting the log: "RuntimeError: MCCL error in" → rule base hit → returns {"domain": "communication", "confidence": 1}; Another example, inputting the log: "CUDAerror :the launch timed out and was terminated" → calling the API → returns {"domain": "operator", "confidence": 0.92}.

[0112] In this embodiment of the application, after obtaining the task failure log, the preset rule base is first called to perform attribution domain matching. If no matching result is found, the RAG model is then called to perform attribution domain analysis. This can improve the attribution accuracy. For example, the attribution accuracy can be increased from less than 60% to 87.3%, and the average response time is 14 seconds, which improves the processing efficiency.

[0113] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0114] Based on the same technical concept, this disclosure also provides a performance testing device for cluster systems corresponding to the performance testing method for cluster systems. Since the principle of the device in this disclosure for solving the problem is similar to the performance testing method for cluster systems described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0115] Reference Figure 4 The diagram shown is a schematic of a performance testing device for a cluster system provided in an embodiment of this disclosure. The device 400 includes:

[0116] The first task submission module 401 is used to submit M first test tasks to the computing cluster in the first round of testing. Each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1.

[0117] The first node determination module 402 is used to determine whether there are candidate abnormal nodes among the target nodes based on the task completion status of each first test task.

[0118] The second task submission module 403 is used to submit K second test tasks to the computing cluster in the second round of testing. Each second test task occupies a node pair, and each node pair includes a candidate node and a normal node in the first round of testing. The candidate node includes candidate abnormal nodes in the first round of testing and / or remaining nodes in the first round of testing. The remaining nodes refer to the nodes other than the target nodes occupied by each of the N nodes.

[0119] The second node determination module 404 is used to determine whether there are abnormal nodes in the computing cluster based on the task completion status of each second test task.

[0120] In one alternative implementation, M is obtained by rounding down the ratio of N to P; and / or, K is determined based on the sum of the number of candidate abnormal nodes and the number of remaining nodes.

[0121] In one optional implementation, the first node determination module 402 is specifically used for:

[0122] For each of the first test tasks, if the task status of the first test task is not "completed", the target node corresponding to the first test task is determined as the candidate abnormal node.

[0123] In one optional implementation, the second node determination module 404 is specifically used for:

[0124] For each second test task, if the task status of the second test task is not "completed", the candidate nodes in the node pair corresponding to the second test task are identified as the abnormal nodes.

[0125] In one alternative implementation, see [link to implementation details]. Figure 5 As shown, the cluster system further includes a computing framework running on top of the computing cluster. This computing framework is used to invoke at least one of the normal nodes to execute computing tasks according to the model training task of the target model. The device also includes a framework performance detection module 405, which is used to:

[0126] During the execution of the model training task, the model floating-point operation utilization rate of the computing framework is determined;

[0127] Determine the operational status information of the computing framework for the target model under different combinations of parallel parameters;

[0128] Determine the operational stability of the computing framework within a preset time period;

[0129] Based on the floating-point operation utilization, the running status information, and the running stability, it is determined whether the performance, compatibility, and stability of the computing framework meet the preset requirements.

[0130] In one optional implementation, the framework performance detection module 405 is specifically used for:

[0131] When determining the model floating-point operation utilization of the computing framework, iterative data from the model training warm-up phase and the breakpoint saving phase are excluded.

[0132] In an optional embodiment, the apparatus further includes a fault attribution determination module 406, the fault attribution determination module 406 being configured to:

[0133] Obtain task failure logs, which are generated by any of the nodes after a task fails to execute;

[0134] The task failure logs are matched against a preset rule base based on preset matching rules to obtain attribution matching results; the rule base includes the mapping relationship between failure logs and attribution domains.

[0135] In an optional implementation, the fault attribution determination module 406 is further configured to:

[0136] If the attribution matching result indicates that no corresponding attribution domain is matched, the retrieval enhancement generation model is invoked to analyze and process the task failure log to obtain the attribution result.

[0137] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0138] Based on the same technical concept, this disclosure also provides an electronic device. (See also...) Figure 6 The diagram shows the structure of an electronic device 600 provided in this embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 stores execution instructions and includes a main memory 6021 and an external memory 6022. The main memory 6021, also called internal memory, is used to temporarily store computational data in the processor 601, as well as data exchanged with external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the main memory 6021.

[0139] In this embodiment, the memory 602 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 601. That is, when the electronic device 600 is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the application code stored in the memory 602, and then executes the method described in any of the foregoing embodiments.

[0140] The memory 602 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0141] Processor 601 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0142] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 600. In other embodiments of this application, the electronic device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0143] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the performance testing method for cluster systems described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0144] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the performance testing method for cluster systems in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0145] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0146] Furthermore, embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0147] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0148] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0149] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0150] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0151] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0152] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0153] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A performance detection method for a cluster system, characterized by, The cluster system includes a computing cluster, which comprises N nodes, where N is a positive integer greater than 1; the method includes: In the first round of testing, M first test tasks are submitted to the computing cluster. Each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1. The first test task is a model training task, and M is obtained by rounding down the ratio of N to P. The nodes include multiple graphics processors. Based on the task completion status of each of the first test tasks, determine whether there are any candidate abnormal nodes among the target nodes; In the second round of testing, K second test tasks are submitted to the computing cluster. Each second test task occupies a node pair. Each node pair includes a candidate node and a normal node from the first round of testing. The candidate node includes candidate abnormal nodes from the first round of testing and / or remaining nodes from the first round of testing. The remaining nodes refer to the nodes other than the target nodes occupied by each of the N nodes. Based on the task completion status of each of the second test tasks, determine whether there are any abnormal nodes in the computing cluster.

2. The method of claim 1, wherein, The value of K is determined based on the sum of the number of candidate abnormal nodes and the number of remaining nodes.

3. The method of claim 1, wherein, The step of determining whether there are candidate abnormal nodes among the target nodes based on the task completion status of each of the first test tasks includes: For each of the first test tasks, if the task status of the first test task is not "completed", the target node corresponding to the first test task is determined as the candidate abnormal node.

4. The method of claim 1, wherein, The step of determining whether there are abnormal nodes in the computing cluster based on the task completion status of each second test task includes: For each second test task, if the task status of the second test task is not "completed", the candidate nodes in the node pair corresponding to the second test task are identified as the abnormal nodes.

5. The method of claim 1, wherein, The cluster system further includes a computing framework running on top of the computing cluster, the computing framework being used to invoke at least one of the normal nodes to execute computing tasks according to the model training task of the target model; the method further includes: During the execution of the model training task, the model floating-point operation utilization rate of the computing framework is determined; Determine the operational status information of the computing framework for the target model under different combinations of parallel parameters; Determine the operational stability of the computing framework within a preset time period; Based on the floating-point operation utilization, the running status information, and the running stability, it is determined whether the performance, compatibility, and stability of the computing framework meet the preset requirements.

6. The method of claim 5, wherein, When determining the model floating-point operation utilization of the computing framework, iterative data from the model training warm-up phase and the breakpoint saving phase are excluded.

7. The method of claim 1, wherein, The method further includes: Obtain task failure logs, which are generated by any of the nodes after a task fails to execute; The task failure logs are matched against a preset rule base based on preset matching rules to obtain attribution matching results; the rule base includes the mapping relationship between failure logs and attribution domains.

8. The method of claim 7, wherein, The method further includes: If the attribution matching result indicates that no corresponding attribution domain is matched, the retrieval enhancement generation model is invoked to analyze and process the task failure log to obtain the attribution result.

9. A performance detection apparatus for a cluster system, characterized by comprising: The cluster system includes a computing cluster, which comprises N nodes, where N is a positive integer greater than 1; the device includes: The first task submission module is used to submit M first test tasks to the computing cluster in the first round of testing. Each first test task occupies P target nodes, where P is a power of 2 and n is a positive integer greater than or equal to 1. The first test task is a model training task, and M is obtained by rounding down the ratio of N to P. The nodes include multiple graphics processors. The first node determination module is used to determine whether there are candidate abnormal nodes among the target nodes based on the task completion status of each of the first test tasks. The second task submission module is used to submit K second test tasks to the computing cluster in the second round of testing. Each second test task occupies a node pair, and each node pair includes a candidate node and a normal node in the first round of testing. The candidate node includes candidate abnormal nodes in the first round of testing and / or remaining nodes in the first round of testing. The remaining nodes refer to the nodes other than the target nodes occupied by each of the N nodes. The second node determination module is used to determine whether there are abnormal nodes in the computing cluster based on the task completion status of each second test task.

10. An electronic device, comprising: include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the performance testing method for cluster systems as described in any one of claims 1-8 are performed.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the performance testing method for cluster systems as described in any one of claims 1-8.

Images