Fault detection method applied to heterogeneous training cluster

Abstract

The invention relates to a fault detection method applied to a heterogeneous training cluster, and the method comprises the steps: starting a detection process based on a defined unified interface; and executing a packet communication detection process, if a fault node is detected, outputting a detection result and ending the current detection process, otherwise, executing a packet training detection process, outputting a detection result and ending the current detection process. Compared with the prior art, the fault node can be accurately and efficiently detected, the problems that single communication detection is low in detection rate, and single training detection is long in time consumption and high in cost are solved, meanwhile, fault detection of a large-scale heterogeneous training cluster is supported, any chip can be supported, and the fault detection complexity applied to the heterogeneous training cluster is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of model training, and in particular to a fault detection method applied to a heterogeneous training cluster. Background Art

[0002] During the training process of large models, hardware failures are inevitable. When a hardware failure causes an abnormal training task, it is necessary to quickly detect the abnormal node and remove it, and then replace it with a spare node to resume training. As the number of model parameters increases, the computing power required for large model training increases, and the cluster scale also becomes larger. In a large-scale heterogeneous training cluster composed of multiple types of chips, when a training task is interrupted due to a hardware failure, how to quickly and accurately detect the faulty node (the longer the training task is interrupted, the higher the capital cost) is an important task.

[0003] Currently, the commonly used solution is to use the binary method for communication detection and training detection, that is, to divide all nodes into two groups and detect them separately. The problematic group is further divided into two. However, communication detection only detects some communication functions between training cards (such as allreduce, allgather, etc.). The test takes a short time but cannot fully reproduce the environment during the training process (the computing volume is large and the hardware temperature is high during the training process); training detection requires preparing a model with the same structure as the training task but with a smaller number of parameters in advance (it can be trained with a small number of nodes), and then using a group of nodes to train this small-parameter model to see if it can be trained normally.

[0004] In summary, the detection tasks of the binary method cannot be run in parallel, and there are problems of long time consumption and high cost. Two detection tasks need to be submitted in one round, and the problematic group is further divided into two. For example, for a ten-thousand-card training cluster (8 cards per node, a total of 1250 nodes), 8 rounds of detection need to be performed (625 -> 312 -> 156 -> 78 -> 39 -> 20 -> 10 -> 5 rounds).

[0005] In addition, the fault detection rate of the communication detection method is low. During communication detection, the computing volume of the chip is relatively low, and it cannot fully reproduce the environment during the training process (the computing volume is large and the hardware temperature is high during the training process), and it may not be able to detect the faulty node. In addition, since it only supports fault detection of homogeneous chips, the cost of migrating to support heterogeneous training clusters is too high in the case of heterogeneous chips. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a fault detection method applied to a heterogeneous training cluster, which can accurately and efficiently detect faulty nodes and at the same time support fault detection of large-scale heterogeneous training clusters.

[0007] The object of the present invention can be achieved by the following technical solutions: A fault detection method applied to a heterogeneous training cluster, comprising the following steps:

[0008] S1. Based on the defined unified interface, start the detection process;

[0009] S2. Execute the group communication detection process. If a faulty node is detected, output the detection result and end the current detection process; otherwise, execute step S3;

[0010] S3. Execute the group training detection process, output the detection result, and end the current detection process.

[0011] Further, in step S1, the unified interface is specifically defined in the form of a function declaration.

[0012] Further, the unified interface in step S1 includes a data structure, a public interface, a group communication detection unified interface, and a group training detection unified interface.

[0013] Further, the data structure includes: a group communication detection task data structure; a communication detection function script; a detection result data structure; a group training detection task data structure; a training detection function script; a node detection task data structure; a node detection script node information data structure.

[0014] Further, the public interface includes: a group start task; a stop task; collecting detection results;

[0015] The group communication detection unified interface includes: a chip type identifier; a communication detection function interface;

[0016] The group training detection unified interface includes: a chip type identifier; a training detection function interface.

[0017] Further, step S1 is specifically a method of adapting the group start task, the communication detection function, the training detection function, the stop task method, and the method of collecting detection results for different chips by using the unified interface.

[0018] Further, the group communication detection process in step S2 is specifically as follows:

[0019] The first round: Group by N nodes as a unit, execute a custom script in groups. If the calculation fails, directly locate the faulty node; if the communication fails, locate the group containing the faulty node;

[0020] Second round: Select one node from one normal group and one abnormal group from the results of the first round to form a two-node Group. Then, for each two-node Group, execute a custom communication script separately. If the result is normal, it indicates that the node in the original abnormal group is normal; if the result is abnormal, it indicates that the node in the original abnormal group is confirmed as a faulty node.

[0021] Further, the custom script is specifically defined by functions such as matmul, allreduce, allgather, or alltoall.

[0022] Further, the grouping training detection process in step S3 is specifically as follows:

[0023] First round: Group by M nodes as a unit, and group to execute the training task with small parameter quantities. The group with normal training is the normal node, and if the training is abnormal, it is marked as an abnormal group, indicating that there are faulty nodes in the abnormal group.

[0024] Second round: Select one node from (M - 1) normal groups and one abnormal group from the results of the first round, and re-perform the training task with small parameter quantities with M nodes as a group. If the training is abnormal, it indicates that the node provided by the original abnormal group is a faulty node; otherwise, it indicates that the node provided by the original abnormal group is normal.

[0025] Further, the abnormal training specifically corresponds to the situation where tgs (training speed) is lower than the preset threshold and loss (loss function value) is abnormal.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] The present invention proposes a fault detection method applied to a heterogeneous training cluster. First, group for communication detection. If a faulty node is detected, the detection process ends. If no faulty node can be detected, then group for training detection. Thus, by combining communication detection and training detection, and designing to perform detection simultaneously in groups, it can improve the accuracy and efficiency of fault detection, and solve the problems of low detection rate of single communication detection, long time consumption and high cost of single training detection.

[0028] The present invention defines a unified interface. When the overall training task includes multiple heterogeneous chips, use the unified interface to adapt the methods of starting tasks in groups, communication detection functions, training detection functions, stopping task methods, and collecting detection results for each chip. Thus, in the entire process of scheduling the fault detection process, there is no need to perceive the chip type. After chip adaptation, the differences of heterogeneous chips can be shielded from affecting the fault detection process, enabling the unified fault detection method to support any chip, reducing the complexity of fault detection applied to a heterogeneous training cluster, and reducing costs. Description of the Drawings

[0029] Figure 1 Schematic diagram of the method flow of the present invention;

[0030] Figure 2 Schematic diagram of the application process of the embodiment;

[0031] Figure 3 Schematic diagram of packet communication detection in the present invention;

[0032] Figure 4 Schematic diagram of the packet training detection process in the present invention;

[0033] Figure 5 Schematic diagram of packet training detection in the present invention. Detailed implementation manners

[0034] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Embodiment

[0036] As Figure 1 shown, a fault detection method applied to a heterogeneous training cluster includes the following steps:

[0037] S1. Based on the defined unified interface, start the detection process;

[0038] S2. Execute the packet communication detection process. If a faulty node is detected, output the detection result and end the current detection process; otherwise, execute step S3;

[0039] S3. Execute the packet training detection process, output the detection result, and end the current detection process.

[0040] This embodiment applies the above solution. As Figure 2 shown, first, perform communication detection in groups. If a faulty node cannot be detected, then perform small parameter training detection in groups, and define a standard interface for fault detection and a process for multiple chip adaptation access, so as to support fault detection of a large-scale heterogeneous training cluster.

[0041] The main contents are as follows:

[0042] I. Define a unified interface

[0043] 1.1. Data structure

[0044] It includes: packet communication detection task data structure; communication detection function script; detection result data structure; packet training detection task data structure; training detection function script; node detection task data structure; node detection script node information data structure.

[0045] 1.2. Public interface:

[0046] Including: ① Grouped start task; ② Stop task; ③ Collect detection results.

[0047] Grouped communication detection unified interface:

[0048] Including: ① Chip type identifier; ② Communication detection function interface.

[0049] Grouped training detection unified interface:

[0050] Including: ① Chip type identifier; ② Training detection function interface.

[0051] 1.3. Multi-chip adaptation process:

[0052] Since the overall training task involves multiple types of chips, for each type of chip, methods for adapting grouped start tasks, communication detection functions, training detection functions, stop task methods, and methods for collecting detection results are required. During the entire fault detection process scheduling, the chip types do not need to be sensed.

[0053] II. Design of the detection process

[0054] 2.1. Grouped communication detection process:

[0055] First round: Group by N nodes as a unit, and execute custom scripts (functions such as matmul, allreduce, allgather, and alltoall) for each group. If the calculation fails, the faulty node can be directly located; if the communication fails, only the group containing the faulty node can be located.

[0056] Second round: Take one node from a normal group and one from an abnormal group in the results of the first round to form multiple two-node Groups, and execute custom communication scripts respectively. If the result is normal, it indicates that the node in the original abnormal group is normal; if the result is abnormal, it indicates that the node in the original abnormal group is indeed abnormal and is the faulty node.

[0057] As Figure 3 shown, in this embodiment, grouping is performed with 8 nodes as a unit. The results of the first round show that all nodes in Group0 are normal, and there are communication faulty nodes in Group1; in the second round, Node0 and Node8 are grouped together, Node1 and Node9 are grouped together, and so on, and communication detection is performed respectively. The result shows that the communication between Node0 and Node8 is abnormal, indicating that Node8 is the faulty node.

[0058] 2.2. Grouped training detection process

[0059] As Figure 4As shown, in the first round: this embodiment groups nodes into units of 4, and performs training and testing tasks (small parameter training tasks) in groups. The group with normal training is the normal node, and the training abnormality (low tgs, abnormal loss, etc.) is marked as the abnormal group, indicating that there is a faulty node in the group.

[0060] In the second round, four nodes are still grouped together, but one node is taken from each of the three normal groups and one abnormal group, and the small parameter training task is re-performed in groups of four nodes. Training abnormalities (low tgs, abnormal loss, etc.) indicate that the nodes provided by the original abnormal group are faulty nodes, and normal training indicates that the nodes provided by the original abnormal group are normal nodes.

[0061] It should be noted that tgs is the training speed. The larger the tgs value, the better the performance. Loss is the value output by the loss function. As the training progresses, the loss value usually becomes smaller.

[0062] like Figure 5 As shown, in this embodiment, the first round of training is performed with four nodes as a group. If Group1 is found to be abnormal in training, it is marked as an abnormal group, and Group0, Group2, and Group3 are marked as normal groups if they are trained normally. In the second round, one node is taken from each of the abnormal group Group1 and the three normal groups to form a new group. If the training result of the new group G0123_0 is abnormal, it means that Node4 provided by the abnormal group marked in the first round is a faulty node.

[0063] In summary, this solution combines the two methods of communication detection and training detection, and performs grouping at the same time, which can detect faulty nodes more accurately and efficiently, solving the problems of low detection rate of single communication detection, long time consumption and high cost of single training detection; on the other hand, it defines the standard interface of fault detection and the access process of multiple chips. For large-scale heterogeneous training clusters, the fault detection process does not perceive the chip hardware structure in the cluster, and realizes the decoupling of the detection process and chip structure. The inspection process is more convenient to expand multiple heterogeneous chips, and the chip access cost is lower. This solution was applied to practice and ran for several months in an actual training task using multiple domestic chips with a total of 10,000 cards. It has been verified that this solution can achieve the above technical effects.

Claims

1. A fault detection method applied to a heterogeneous training cluster, characterized in that, It includes the following steps: S1. Based on the defined unified interface, start the detection process; S2. Execute the group communication detection process. If a faulty node is detected, output the detection result and end the current detection process. Otherwise, execute step S3; S3. Execute the group training detection process, output the detection result, and end the current detection process.

2. The fault detection method for heterogeneous training clusters according to claim 1, wherein, In step S1, the unified interface is specifically defined in the form of a function declaration.

3. A fault detection method applied to a heterogeneous training cluster according to claim 1, characterized in that, The unified interface in step S1 includes a data structure, a public interface, a unified interface for group communication detection, and a unified interface for group training detection.

4. The fault detection method for a heterogeneous training cluster according to claim 3, wherein, The data structure includes: a data structure for group communication detection tasks; a communication detection function script; a data structure for detection results; a data structure for group training detection tasks; a training detection function script; a data structure for node detection tasks; a node detection script node information data structure.

5. The fault detection method for heterogeneous training clusters according to claim 4, wherein The public interface includes: a group start task; a stop task; collecting detection results; The unified interface for group communication detection includes: a chip type identifier; a communication detection function interface; The unified interface for group training detection includes: a chip type identifier; a training detection function interface.

6. The fault detection method for heterogeneous training clusters according to claim 5, characterized in that, Step S1 is specifically a method of adapting the group start task, communication detection function, training detection function, stop task method, and collecting detection result method for different chips using the unified interface.

7. A fault detection method applied to a heterogeneous training cluster according to claim 1, characterized in that, The group communication detection process in step S2 is specifically as follows: The first round: Group by N nodes as a unit, execute a custom script for each group. If the calculation fails, directly locate the faulty node. If the communication fails, locate the group containing the faulty node; The second round: Take one node from a normal group and an abnormal group in the results of the first round to form a two-node Group. Then, for each two-node Group, execute a custom communication script respectively. If the result is normal, it indicates that the node in the original abnormal group is normal. If the result is abnormal, it indicates that the node in the original abnormal group is confirmed as a faulty node.

8. A fault detection method applied to a heterogeneous training cluster according to claim 7, characterized in that The custom script is specifically defined through functions such as matmul, allreduce, allgather, or alltoall.

9. A fault detection method applied to a heterogeneous training cluster according to claim 1, wherein The group training detection process in step S3 is specifically as follows: The first round: Group by M nodes as a unit, execute a small-parameter training task for each group. The groups with normal training are normal nodes, and the groups with abnormal training are marked as abnormal groups, indicating that there are faulty nodes in the abnormal groups; The second round: Take one node from (M - 1) normal groups and an abnormal group in the results of the first round, and re-execute the small-parameter training task with M nodes as a group. If the training is abnormal, it indicates that the node provided by the original abnormal group is a faulty node. Otherwise, it indicates that the node provided by the original abnormal group is normal.

10. A fault detection method applied to a heterogeneous training cluster according to claim 9, characterized in that The abnormal training specifically corresponds to the situation where tgs is lower than the preset threshold and the loss is abnormal.

Images