Fault handling method for large-model cluster training and related apparatus

Abstract

Embodiments of the present application disclose a fault handling method for large-model cluster training and a related apparatus. The method comprises: periodically collecting communication traffic of each node in a large-model cluster, and computing the total communication traffic corresponding to each traffic collection moment; when it is detected that a training interruption occurs to the large-model cluster, acquiring an interruption time period; on the basis of the total communication traffic corresponding to each traffic collection moment within the interruption time period, selecting one traffic collection moment from among the traffic collection moments within the interruption time period as a communication interruption moment; and on the basis of the communication traffic of each node at the communication interruption moment, determining a faulty node.

Description

Fault processing method for large model cluster training and related device

[0001] The present application claims priority to the Chinese patent application No. 202411263017.6, filed on September 10, 2024, and entitled "Fault processing method for large model cluster training and related device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of network communication technology, in particular to a fault processing method for large model cluster training and related device. BACKGROUND

[0003] In the field of artificial intelligence (AI) technology, a large model refers to a deep learning model with a large number of parameters, typically reaching millions or tens of millions, or even hundreds of millions. Such models have high expressive power and prediction performance, and can handle more complex tasks.

[0004] Large models have extremely high demands for computing resources and storage space, and a single computing node often cannot meet the requirements, so cluster training becomes an effective solution, by distributing training tasks to multiple computing nodes, thereby significantly improving training efficiency.

[0005] TECHNICAL CONTENT

[0006] Embodiments of the present application provide a fault processing method and device for large model cluster training, an electronic device, a computer readable storage medium, and a computer program product. Embodiments of the present application can quickly locate the fault node in the large model cluster after the training interruption of the large model cluster.

[0007] The embodiments of the present application provide a fault processing method for large model cluster training, the method comprising: periodically collecting communication traffic of each node in the large model cluster, and calculating total communication traffic corresponding to each traffic collection time; when detecting that the large model cluster has a training interruption, obtaining an interruption period; selecting one traffic collection time as a communication interruption time among each traffic collection time in the interruption period according to the total communication traffic corresponding to each traffic collection time in the interruption period; and determining a fault node according to the communication traffic of each node at the communication interruption time.

[0008] The embodiment of the application further provides a fault processing device for large model cluster training, the device comprising: a traffic collection module configured to periodically collect communication traffic of each node in the large model cluster and calculate total communication traffic corresponding to each traffic collection time; a traffic calculation module configured to, when detecting that the large model cluster has a training interruption, acquire an interruption period; a time determination module configured to select one traffic collection time as a communication interruption time among each traffic collection time in the interruption period according to the total communication traffic corresponding to each traffic collection time in the interruption period; and a node positioning module configured to determine a fault node according to communication traffic of each node at the communication interruption time.

[0009] The embodiment of the application further provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the fault processing method for large model cluster training as described above.

[0010] The embodiment of the application further provides a computer readable storage medium having a computer program stored thereon, when the computer program is executed by a processor of an electronic device, the electronic device executes the fault positioning method for large model cluster training as described above.

[0011] The embodiment of the application further provides a computer program product comprising a computer program, when the computer program is executed by a processor of an electronic device, the computer program product implements the fault processing method for large model cluster training as described above.

[0012] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the application.

[0013] BRIEF DESCRIPTION OF DRAWINGS

[0014] Fig. 1 is a schematic diagram of an exemplary implementation environment related to the application.

[0015] Fig. 2 shows an architecture diagram for network state detection of the large model cluster 110 in the implementation environment shown in Fig. 1.

[0016] Fig. 3 is a flowchart of a fault processing method for large model cluster training according to an exemplary embodiment of the application.

[0017] Fig. 4 shows a traffic view diagram of an exemplary large model training task.

[0018] Fig. 5 is a flowchart of a fault processing method for large model cluster training further proposed on the basis of the embodiment shown in Fig. 3.

[0019] FIG. 6 is a flowchart of a method for handling a failure in large model cluster training according to another example embodiment of the present application.

[0020] FIG. 7 is a flowchart of another method for handling a failure in large model cluster training according to another example embodiment of the present application.

[0021] FIG. 8 is a flowchart of yet another method for handling a failure in large model cluster training according to another example embodiment of the present application.

[0022] FIG. 9 is a block diagram of an apparatus for handling a failure in large model cluster training according to an example embodiment of the present application.

[0023] FIG. 10 illustrates a structural diagram of a computer system suitable for implementing an electronic device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0024] The example embodiments will be described in detail with reference to the accompanying drawings. In the following description, the same numbers are used to denote the same elements, unless otherwise specified. The embodiments described in the following example embodiments do not represent all the embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities, and do not necessarily correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely exemplary illustrations, and do not necessarily include all the contents and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed depending on the actual situation.

[0027] In the present application, "a plurality of" means two or more. The association relationship of "and / or" describes the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally means that the associated objects before and after are in an "or" relationship.

[0028] The terms "first", "second", "third", and "fourth" and the like in the description and in the claims of the present application and the accompanying drawings are used to distinguish between similar objects, not to describe a particular sequential order. The terms "comprises", "comprising", "includes", "including" and the like are meant to be interpreted open-ended. For example, a process, a method, a system, a product or a device that comprises a list of steps or units is not limited to the listed steps or units but can optionally further comprise additional steps or units not listed. Alternatively, a process, a method, a system, a product or a device that comprises a list of steps or units can optionally further comprise additional steps or units not listed.

[0029] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0030] First of all, it should be pointed out that the large model is also called pre-training model (PTM), cornerstone model, which refers to a deep neural network (DNN) with large parameters, which is trained on a large amount of unlabeled data, and uses the function approximation capability of the large parameter DNN to extract common features from the data. Through fine tuning, parameter efficient fine tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the large model can achieve ideal results in the few-shot or zero-shot scene.

[0031] The large model can be divided into language models (such as ELMO, BERT, GPT), visual models (such as swin-transformer, ViT, V-MOE), speech models (such as VALL-E), multi-modal models (such as ViBERT, CLIP, Flamingo, Gato) and the like according to the data modality processed, wherein the multi-modal model refers to a model that establishes the feature representation of two or more data modalities. The large model is an important tool for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface connecting multiple specific task models.

[0032] The large model usually has the following characteristics:

[0033] First, distributed training;

[0034] Distributed training refers to splitting and sharing the workload of training a model to multiple microprocessors. The parameters of a large model are large, and the training data is large, exceeding the capacity of a single machine, so distributed parallel speedup is needed. Parallel mechanisms can include data parallelism (DP), model parallelism (MP), pipeline parallelism (PP), and hybrid parallelism (HP). Structural design includes parameter server-based, reduce-based, MPI-based, and other structures.

[0035] Second, model compression and quantization;

[0036] Model compression and quantization refer to techniques that help reduce model size and speed up model inference, thereby reducing the cost of model storage and computation. Model compression typically includes pruning, low-rank decomposition, and knowledge distillation, while model quantization refers to converting floating-point parameters in the model to fixed-point or integer parameters, thereby reducing model size and speeding up model inference.

[0037] Third, adaptive computing;

[0038] Adaptive computing refers to automatically adjusting the computational load and accuracy of a model based on different input data to achieve the goal of improving model computational efficiency while maintaining model accuracy. Adaptive computing can flexibly adjust the computational load and accuracy of a model on different input data, thereby better balancing the computational efficiency and accuracy of the model.

[0039] Fourth, model parallel computing;

[0040] Model parallel computing refers to distributing the computational tasks of a model to multiple computing devices (such as CPUs, GPUs, TPUs, etc.) for simultaneous computation, thereby speeding up the training and inference of the model. Model parallel computing can effectively utilize computing resources, improving the computational efficiency and training speed of the model.

[0041] Large models have extremely high demands for computing resources and storage space, and a single computing node often cannot meet the requirements, so cluster training becomes an effective solution. By distributing training tasks to multiple computing nodes, training efficiency is significantly improved. The network cluster used for large model training often has a large scale, with a large number of devices and communication links, resulting in a high frequency of faults, which in turn causes the large model training task to be unable to run stably for a long time.

[0042] In some implementation methods, when a large model training task is interrupted, the following processing flow is typically performed:

[0043] 1. Collecting node logs: The operation and maintenance personnel need to log in to each node device to collect relevant log files. Since the number of nodes contained in the large model cluster is large, this process is very time-consuming.

[0044] 2. Analyzing alarm information: Alarm information usually contains key information about the running state of the device. However, since the alarm information may come from different systems and devices, manual comparison and analysis are needed to determine certain devices or systems that cause failures.

[0045] 3. Comparing node by node: In order to further locate the network or device problem, the operation and maintenance personnel may need to compare the logs and performance indicators of multiple node devices. This process requires writing scripts for careful comparison and analysis to find the root cause of the failure device.

[0046] 4. Fault location: After determining the possible failure device, the operation and maintenance personnel need to further analyze to determine the specific cause of the problem. This process may involve code review, hardware detection, and other aspects.

[0047] Therefore, the current fault location technology implementation has at least the disadvantages of long time consumption, high labor cost, and high error risk. Based on the above disadvantages, the time consumption of a normal training interruption recovery is usually more than one day. During this recovery period, the computing resources of the nodes are in a waiting state, resulting in a large amount of computing resource loss, which not only affects the current large model training task, but also may affect other businesses that rely on these resources.

[0048] To solve the above technical problems, the embodiments of the present application propose a large model cluster training fault processing method, a large model cluster training fault processing device, an electronic device, a computer readable storage medium, and a computer program product. The following will be described in detail.

[0049] First, please refer to FIG. 1, which is a schematic diagram of an exemplary implementation environment related to the present application. The implementation environment is a large model training system, which includes a large model cluster 110, an alarm device 120, and a user terminal 130.

[0050] The large model cluster 110 is composed of multiple nodes, each of which can be a device with a GPU (Graphics Processing Unit, graphics processing unit), such as a terminal, a server, etc., which is not limited here. By distributing the large model training task to multiple nodes, efficient large model training is performed.

[0051] The nodes in the large model cluster 110 can be divided into master nodes and slave nodes. The master nodes are mainly responsible for coordinating the training tasks in the entire cluster, which can include, for example, assigning training tasks to individual slave nodes, managing the distribution and synchronization of training data, detecting the status of individual slave nodes, collecting and summarizing local gradients calculated by individual slave nodes and performing global parameter updates, distributing updated parameters to individual slave nodes, etc. The slave nodes are mainly responsible for executing the training tasks assigned by the master nodes, such as calculating local gradients, sending local gradients to the master node for aggregation, receiving updated parameters from the master node, etc.

[0052] Due to the high demand for computing resources and storage space of large models, the large model cluster 110 often has a large network size, a large number of nodes and communication links, resulting in a high frequency of faults, which in turn causes the large model training task to be unable to run stably for a long time. Therefore, the alarm device 120 is configured to periodically collect the communication traffic of each node in the large model cluster during the process of training the large model in the large model cluster 110, and perform traffic statistics to obtain the total communication traffic corresponding to each traffic collection time. When a disruption in the training of the large model is detected, the corresponding disruption period is obtained, and then according to the total communication traffic corresponding to each traffic collection time in the disruption period, a traffic collection time is selected as the communication disruption time among the traffic collection times in the disruption period, and the fault nodes are determined according to the communication traffic of each node at the communication disruption time, thereby quickly and accurately locating the fault nodes of the large model cluster 110.

[0053] After determining the fault nodes, the alarm device 120 can send relevant information of the fault nodes to the user terminal 130, so that the operation and maintenance personnel can obtain the relevant information of the fault nodes through the user terminal 130, thereby triggering the corresponding fault handling of the fault nodes. For example, the fault nodes can be removed from the large model cluster 110, and another node device is added to the large model cluster 110, thereby recovering the large model training.

[0054] The alarm device 120 can also have an alarm prompt function, for example, the alarm device 120 can be equipped with a display panel or screen, a voice module, etc. alarm prompt components, so that after determining the fault nodes, the alarm device 120 can prompt the operation and maintenance personnel to perform corresponding fault handling on the specific fault nodes through the alarm prompt components configured by itself.

[0055] It should be noted that the alarm device 120 can be a terminal such as a computer, a smart phone, a tablet computer, a notebook computer, etc., can also be a stand-alone physical server, and can also be a server cluster or a distributed system composed of multiple physical servers, and the present application does not limit this. The user terminal 130 can be a terminal such as a computer, a smart phone, a tablet computer, a notebook computer, etc., and the present application also does not limit this.

[0056] Please continue to refer to FIG. 2, which shows an architecture diagram for network state detection of the large model cluster 110 in the implementation environment shown in FIG. 1. As shown in FIG. 2, it specifically shows that each node in the large model cluster 110 can be detected for network state by INT (Inband Network Telemetry) probe packets and SFLOW (Solution for Network Monitoring) traffic collection methods.

[0057] INT is a technology for collecting network status by means of data plane traffic. Telemetry information is carried and transmitted by data plane traffic, and does not depend on the control plane. By embedding INT instructions in data packets, these instructions are interpreted by forwarding devices and relevant information is filled in, and then reported level by level, thereby realizing real-time monitoring of network status and remote reporting of data. As shown in FIG. 2, by INT probe packets, PFC (Priority Flow Control) packets, CNP (Congestion Notification Packet) packets and other network state data of each node in the large model cluster 110 can be obtained.

[0058] SFLOW is a network traffic detection technology based on packet sampling, mainly used for statistical analysis of network traffic. SFLOW agents are embedded in the ASIC (Application Specific Integrated Circuit) chips of the network devices to be detected, supporting flow-based data packet statistical sampling and time-based network interface statistical sampling, and the sampled data packets are forwarded to the designated SFLOW traffic sampler for analysis. For example, for the implementation environment shown in FIG. 1, SFLOW agent programs can be deployed in each node to sample the data packets of each node, and SFLOW traffic sampler programs are deployed in the alarm device 120, thereby realizing the collection of communication traffic of each node in the large model cluster 110 by the alarm device 120.

[0059] The SFLOW technology can support collection and reporting of quintuple flow information, which refers to five key elements in network communication, including source IP (Internet Protocol) address, source port number, destination IP address, destination port number, and transport layer protocol. These five elements collectively define a specific communication flow in the network. Thus, by collecting service flows using SFLOW, the communication traffic of each node in the large model cluster 110 can be obtained.

[0060] In actual application scenarios, appropriate detection technologies can be selected according to actual application requirements to detect the network status of the large model cluster 110, or multiple technologies can be combined to detect the network status of the large model cluster 110 to achieve more comprehensive network detection. The present application does not limit this.

[0061] Referring to FIG. 3, FIG. 3 is a flowchart of a fault processing method for large model cluster training according to an example embodiment of the present application. The method can be applied to the example implementation environment shown in FIG. 1, for example, can be specifically executed by the alarm device 120, or jointly executed by the alarm device 120 and the large model cluster 110. Of course, the method can also be applied to other implementation environments and executed by terminals or servers in other implementation environments, or jointly executed by terminals and servers in other implementation environments. The present application does not limit this.

[0062] As shown in FIG. 3, the fault processing method for large model cluster training includes S310-S340, which are described in detail as follows.

[0063] S310, periodically collect the communication traffic of each node in the large model cluster, and calculate the total communication traffic corresponding to each traffic collection time.

[0064] In the present application, the communication traffic is remote direct memory access (RDMA) traffic between nodes in the large model cluster. During the large model cluster training process, the nodes in the large model cluster use RDMA communication technology for communication, so that the memory spaces of different nodes can directly transmit data, thereby generating RDMA traffic. The RDMA traffic of a node in the large model cluster includes: the traffic generated by the node directly accessing the memory of other nodes in the large model cluster, and the traffic generated by other nodes directly accessing the memory of the node.

[0065] The traffic collection time refers to the time when the communication traffic of each node is actually collected. The total communication traffic corresponding to a certain traffic collection time is the sum of the communication traffic of all nodes in the large model cluster at that traffic collection time.

[0066] The periodic collection of communication traffic of each node in the large model cluster can be achieved by the way of collecting traffic flow shown in SFLOW in FIG. 2, or other ways to periodically collect the communication traffic of each node, which is not limited in the present application.

[0067] For example, the data transmission message can be reported by the switch with a preset sampling ratio, such as a sampling ratio of 64K:1 or other sampling ratios, so that the communication traffic can be calculated according to the reported sampling message and the sampling ratio. It can be understood that the sampling ratio of 64K:1 is to sample one message for reporting in every 64000 data transmission messages. The communication traffic of each node can also be reported once every preset time interval, which is not limited in the present application.

[0068] S320, when detecting that the large model cluster appears training interruption, obtaining the interruption period.

[0069] The traffic view of the large model training task can be detected to determine whether the large model cluster appears training interruption.

[0070] The traffic view is used to show and analyze the communication traffic change of the large model training task.

[0071] The large model training process will cause the communication traffic to decrease every certain time interval due to the execution of checkpoint operation. This kind of communication traffic decrease is considered as normal condition. If the communication traffic decrease other than this normal condition is detected, it means that the training interruption is detected. Or it can be understood that the training interruption will cause the communication traffic of the large model training task to abnormally decrease. If the abnormal decrease of the communication traffic is detected, it means that the training interruption is detected. Since the decrease of the communication traffic caused by the execution of checkpoint operation in the large model training process is relatively slow, if the communication traffic of the large model training task at a certain time is detected to decrease to below a preset proportion of the standard traffic, such as below 80% of the standard traffic, it can be determined that this decrease of the communication traffic is not caused by the execution of checkpoint operation, so as to determine that the abnormal decrease of the communication traffic occurs, thereby detecting that the large model training appears interruption, wherein the standard traffic is a pre-set traffic threshold, for example, 200000Gbps.

[0072] FIG. 4 shows a schematic diagram of a traffic view of an exemplary large model training task, as shown in FIG. 4, the communication traffic of the large model training task changes over time, wherein the communication traffic of the large model training task drops below 80% of the standard traffic 200000 Gbps within 08:46-09:00 and 10:15-10:40, which is considered as an abnormal drop in communication traffic, and thus the training interruption of the large model training cluster is detected at the moment of the abnormal drop in communication traffic. It should be understood that the communication traffic of the large model training task at each moment is represented by the sum of the communication traffic of all nodes in the large model cluster at that moment.

[0073] The interruption period represents the time period corresponding to the training interruption of the large model cluster, for example, after detecting the training interruption of the large model cluster, a time period with a preset time length before and after the moment is obtained as the interruption period. For example, if the moment of the training interruption of the large model cluster is represented as T, and the preset time length is 10 minutes (min), then the time period T-10 min ~ T+10 min can be taken as the interruption period. The moment T of the training interruption of the large model cluster can also be referred to as the moment of the abnormal drop in communication traffic.

[0074] S330, according to the communication total traffic corresponding to each traffic collection moment in the interruption period, selecting one traffic collection moment as the communication interruption moment among the traffic collection moments in the interruption period.

[0075] Considering the communication propagation chain of the large model cluster training, all nodes participating in the training essentially form a communication graph, and if there is a faulty node, it will cause communication abnormalities in other nodes communicating with the faulty node, which is manifested as an abnormal drop in the overall traffic of the large model training task. Therefore, according to the communication total traffic corresponding to each traffic collection moment in the interruption period, the moment of the overall traffic abnormal drop of the large model training task can be determined, and this moment can be taken as the communication interruption moment.

[0076] It should be noted that in the present application, the time when the large model cluster is detected to have a training interruption is not directly taken as the communication interruption time, but according to the total communication traffic corresponding to each traffic collection time in the interruption period, one traffic collection time in the interruption period is selected as the communication interruption time, considering that due to real factors such as network delay or certain business function settings, the time when the large model cluster is detected to have a training interruption may not be the real communication interruption time. Therefore, after detecting that the large model cluster has a training interruption, by obtaining the interruption period, and then according to the total communication traffic corresponding to each traffic collection time in the interruption period, one traffic collection time in the interruption period is selected as the communication interruption time, the accuracy of the communication interruption time can be improved, and thus the positioning accuracy of the fault node can be improved.

[0077] S340, according to the communication traffic of each node at the communication interruption time, determine the fault node.

[0078] Considering that the fault node is the root cause node that causes the large model training interruption, at the communication interruption time, the fault node should be the first node that is blocked in communication, and the fault node has a relatively obvious phenomenon of traffic drop, so it can be determined that the node with the lowest communication traffic at the communication interruption time is the fault node.

[0079] In the technical scheme provided in the present application, when the large model training interruption is detected, the fault node in the large model cluster is automatically analyzed based on the collected communication traffic, which can make the positioning of the fault node in the large model cluster more efficient and reduce the waste of computing resources caused by the large model training interruption. Moreover, when the large model training interruption is detected, according to the total communication traffic corresponding to each traffic collection time in the interruption period, one traffic collection time in the interruption period is selected as the communication interruption time, and then according to the communication traffic of each node at the communication interruption time, the fault node is determined, which can accurately locate the root cause node that causes the large model training interruption, which also shortens the time required for the large model training recovery to a certain extent, so as to further reduce the waste of computing resources caused by the large model training interruption.

[0080] After the fault node is determined, the fault cause of the fault node can be further determined as network failure or device failure according to the network state information corresponding to the fault node. If it is network failure, the large model training can be continued after the network is recovered. If it is device failure, a new node device can be replaced. For example, the network state information of the fault node can be obtained in the manner of the INT probe packet shown in FIG. 2, or other manners, which are not limited herein. The network state information of the fault node can include GPU utilization, PFC packet, ECN packet, etc., which are not limited herein.

[0081] Please further refer to FIG. 5, which is a flowchart of the fault processing method of the large model cluster training further proposed on the basis of the embodiment shown in FIG. 3.

[0082] As shown in FIG. 5, the process of selecting one traffic collection time as the communication on-off time among the traffic collection times in the interruption period according to the total communication traffic corresponding to each traffic collection time in the interruption period includes S510-S520, which are described in detail as follows:

[0083] S510, arrange the total communication traffic corresponding to each traffic collection time in the interruption period in chronological order to obtain a total communication traffic sequence.

[0084] As described before, in the large model cluster, all the nodes participating in the training essentially constitute a communication graph, and if a fault node appears, other nodes communicating with the fault node will all produce communication abnormalities. On this basis, in order to exclude the influence of other traffic decline nodes, the differential value is used to smooth the larger traffic decline amplitude of other nodes to accurately locate the fault node.

[0085] In order to facilitate understanding of the principle of using the differential value to smooth the larger traffic decline amplitude of other nodes, for example, it is assumed that the communication traffic of the fault node (i.e. root cause node) decreases from 100Gbps at t-1 time to 20Gbps at t time, the traffic decline amplitude is 80%, the differential value is (20-100) / 1s=-80, and the communication traffic of another node decreases from 60Gbps at t-1 time to 5Gbps at t time, the traffic decline amplitude is 91.7%, the differential value is (5-60) / 1s=-55. The traffic decline amplitude of another node is obviously larger than that of the fault node, but the differential value of another node is smaller than that of the fault node. Therefore, it can be obtained that the differential value can smooth the larger traffic decline amplitude of other nodes, thereby accurately locating the fault node.

[0086] It can be understood that the difference refers to a calculation manner of the difference between the values of a sequence at different points, and the result of the difference calculation reflects the change between discrete quantities. Therefore, the total communication traffic corresponding to each traffic collection time in the interruption period needs to be arranged in chronological order to obtain a total communication traffic sequence, and then the total communication traffic sequence is subjected to a difference operation, so as to select one traffic collection time as the communication interruption time based on the result of the difference operation.

[0087] S520, calculating the difference value corresponding to each total communication traffic in the total communication traffic sequence, and selecting one traffic collection time as the communication interruption time from the traffic collection times in the interruption period according to the difference value corresponding to each total communication traffic in the total communication traffic sequence.

[0088] The difference operation can be divided into three types: forward difference, backward difference and central difference. The forward difference refers to subtracting the value of a point from the value after the point, and the obtained result is the forward difference value of the point. The backward difference refers to subtracting the value of a point from the value before the point, and the obtained result is the backward difference value of the point. The central difference takes the difference between the average value of the two points before and after a point and the value of the point as the central difference value of the point.

[0089] Since the fault location is concerned about the case of abnormal decrease in traffic, it is more suitable to use the forward difference value to determine the communication interruption time. For example, the forward difference value is calculated for each total communication traffic in the total communication traffic sequence, a forward difference sequence corresponding to the total communication traffic sequence is constructed based on the forward difference value corresponding to each total communication traffic, and then the smallest forward difference value in the forward difference sequence is traversed to determine the traffic collection time corresponding to the smallest forward difference value as the communication interruption time.

[0090] For example, the total communication traffic sequence is represented as S1: {(t1, traffic1), (t2, traffic2), …, (tn, trafficn)}, and the forward difference value is represented as n n wherein 1≤i≤n, the forward difference sequence S2 can be obtained by calculating the forward difference value of each time in the total communication traffic sequence S1: Thus, the smallest forward difference value in the forward difference sequence S2 can be traversed. The traffic collection time corresponding to the smallest forward difference value is determined as the communication interruption time.

[0091] If the communication interruption time is represented as t i ​and the traffic sequence of each node in the large model cluster at t i is expressed as S3:{(t i , N1, traffic1), (t i , N2, traffic2)…, (t n , N n , traffic n )}, where N i represents the training node i, then the node corresponding to the smallest traffic value in the traffic sequence S3 can be traversed, and this node is the fault node.

[0092] In this application, the communication interruption time can be accurately determined based on the forward difference value. Specifically, the traffic collection time corresponding to the minimum difference value is determined as the communication interruption time, and then the node with the lowest traffic at the communication interruption time is determined as the fault node. The fault node is equivalent to the node with the lowest traffic at the communication interruption time with the smallest difference value. Since the communication interruption time is determined based on the forward difference value, under this premise, the node with the lowest traffic at the communication interruption time is also the node with the largest traffic drop, so the determined fault node is also determined based on the forward difference value, which can ensure accuracy.

[0093] In this application, backward difference or central difference can also be used to determine the communication interruption time, but some improvements need to be made. For example, in the case of using backward difference value to determine the communication interruption time, for the constructed backward difference sequence, the maximum backward difference value should be traversed, and the next traffic collection time of the traffic collection time corresponding to the maximum backward difference value traversed is determined as the communication interruption time. For example, in the case of using central difference value to determine the communication interruption time, for the constructed central difference sequence, the maximum central difference value should also be traversed, and the next traffic collection time of the traffic collection time corresponding to the maximum central difference value traversed is determined as the communication interruption time. Or an improved central difference sequence can be constructed, in which the difference value corresponding to each traffic collection time is the difference between the average value of the traffic collection time after and the traffic collection time before in the communication total traffic sequence and the current traffic collection time value. The minimum difference value in the improved central difference sequence is traversed, and the next traffic collection time of the traffic collection time corresponding to the maximum central difference value traversed is determined as the communication interruption time.

[0094] Regardless of which difference value scheme is used, since the difference value can smooth the larger traffic drop of other nodes, the fault node can be accurately located. However, it should be noted that the difference value schemes in the above examples are all first-order difference values, and second-order difference or other multi-order difference schemes can also be used to locate the fault node, which is not limited by the present application.

[0095] Please continue to refer to FIG. 6, which is a flowchart of a method for fault handling of large model cluster training according to another exemplary embodiment of the present application. As shown in FIG. 6, the method further includes S610-S620 based on the embodiment shown in FIG. 3, which are described in detail as follows:

[0096] S610, after the faulty node is removed from the large model cluster, when a newly added node is detected, establishing a communication connection between the remaining nodes in the large model cluster and the newly added node.

[0097] After the faulty node is determined, the rapid recovery of the large model cluster training is further carried out in combination with the business side. Exemplarily, the faulty node can be removed from the large model cluster, and a new node is added to the large model cluster, and the large model training is recovered by replacing the faulty node with the newly added node. For example, after receiving the prompt information about the faulty node, the operation and maintenance personnel manually remove the faulty node from the large model cluster, and then add another new device to the large model cluster.

[0098] When the newly added node is detected, a communication connection can be established between the remaining nodes in the large model cluster after the faulty node is removed and the newly added node, and the large model training is recovered based on the large model cluster with the established communication connection.

[0099] Exemplarily, as shown in FIG. 7, the process of establishing a communication connection between the remaining nodes in the large model cluster and the newly added node can include the following steps:

[0100] S710, restarting the networking sub-process respectively deployed in the remaining nodes and the newly added node;

[0101] S720, networking between nodes based on the networking sub-process deployed in each node, establishing a communication connection between the remaining nodes and the newly added node.

[0102] It can be understood that the node devices that can be used to build a large model cluster all have a networking sub-process deployed therein, and the automatic networking between nodes is realized through the networking sub-process, thereby establishing the cluster communication of the large model cluster.

[0103] Therefore, by restarting the networking sub-process respectively deployed in the remaining nodes and the newly added node, the networking between nodes is automatically carried out based on the restarted networking sub-process respectively deployed in the remaining nodes and the newly added node, thereby establishing a communication connection between the remaining nodes and the newly added node.

[0104] Exemplarily, the process of networking between nodes by the networking sub-process can include the following steps:

[0105] S721, obtain host group information from the shared storage server, the host group information being used to record host information of each node currently contained by the large model cluster;

[0106] S722, save the host information of the local node to the shared storage server to update the host group information in the shared storage server;

[0107] S723, continuously query the host group information in the shared storage server, when the number of host information contained in the queried host group information reaches a preset number value, and the number of host information contained in the queried host group information does not change within a preset time length, perform networking configuration on the local node, and establish communication connection between nodes based on the networking configuration.

[0108] It can be understood that the shared storage server is used to provide a shared storage space, and the shared storage space is used to store host group information (Host Group), which is used to record host information of all nodes currently contained by the large model cluster. The networking sub-process obtains the currently recorded host group information from the shared storage server, and then saves the host information of the local node in the shared storage server. This can be understood as that the networking sub-process adds its own node to the host group, and synchronizes the host information to the host group information recorded in the shared storage server. The networking sub-process continuously queries the host group information in the shared storage server, when the number of host information contained in the queried host group information reaches a preset number value, and the number of host information contained in the queried host group information does not change within a preset time length, it is indicated that all nodes are ready for networking, and therefore the local node is configured for networking, so as to establish communication connection between nodes based on the networking configuration.

[0109] The networking sub-process performs networking configuration on the local node, which may include, for example, configuring local rank, global rank, world size and the like, so as to automatically perform cluster communication networking based on the configured information, to complete subsequent training of the large model cluster.

[0110] The local rank can be referred to as local rank, which is the ranking or identifier between multiple computing units (such as GPUs) on a single node, and is used for parallel computing and communication operation within the node. For example, within each computing unit, the role and task within the node and communication and data exchange with other computing units can be determined by the local rank value.

[0111] The global rank can be referred to as global rank, which is the ranking or identifier between nodes in the large model cluster.

[0112] The world size can be referred to as a world size, which refers to the number of nodes in the large model cluster, that is, the number of nodes participating in the large model training. The world size can be used to ensure that all nodes in the large model cluster can participate in the large model training process, and also helps to reasonably divide the large model training task to different nodes to realize data distribution, parameter synchronization and gradient aggregation and other operations.

[0113] Thus, the networking sub-process can obtain the ranking information corresponding to the local node, such as local rank, global rank, world size and the like, based on the host group information in the shared storage server, and then establish the communication connection between the nodes based on the determined ranking information, thereby realizing the establishment of the communication connection between the remaining nodes and the newly added node.

[0114] As shown in FIG. 8, the process of establishing the communication connection between the remaining nodes and the newly added node in the large model cluster can further include the following steps:

[0115] S810, determining the node role corresponding to the fault node;

[0116] S820, establishing the communication connection between the remaining nodes and the newly added node according to the determined node role.

[0117] It can be understood that the node role includes a master node or a slave node. As described previously, in the large model cluster, the master node is mainly responsible for coordinating the training task in the entire cluster, and the slave node is mainly responsible for executing the training task allocated by the master node.

[0118] The communication connection strategies corresponding to different node roles are different, and therefore, the node role corresponding to the fault node is first determined, and then the communication connection strategy matched with the node role is used to establish the communication connection between the remaining nodes and the newly added node.

[0119] In the case where the node role of the fault node is a master node, since the previous master node has been removed from the large model cluster, when the newly added node is detected, the master node and the slave node need to be re-determined from the remaining nodes and the newly added node, so as to establish the communication connection between the re-determined master node and the slave node.

[0120] For example, the master node can be elected from the remaining nodes and the newly added node through an election mechanism, and the other nodes except the re-determined master node are all slave nodes. For example, the remaining nodes and the newly added node can elect a new master node through a heartbeat mechanism or other election algorithm, and the specific content of the election mechanism is not limited herein.

[0121] After the removal of the faulty node, the remaining nodes detect that the master node is unavailable, and thus initiate an election mechanism of their own deployment to re-elect a master node from the remaining nodes and the newly joined nodes.

[0122] The management responsibility in the large model cluster is taken over by the re-determined master node, for example, the re-determined master node broadcasts a master node update message containing the node address of the re-determined master node, so as to update the communication graph according to the node address of the re-determined master node, and the communication connection between nodes is based on the updated communication graph.

[0123] It can be understood that in the large model cluster, a communication graph needs to be established between the master node and the slave node according to certain logic and data dependency relationship to ensure that the training task can be efficiently and correctly executed. For example, a checkpoint file is used to save the state data of the large model cluster, such as the state of the training task execution, data partition information, dependency relationship, etc., to restore the computing state of the node when the training task fails or needs to be restarted. Thus, according to the state data in the checkpoint file, a directed graph can be constructed to represent the communication relationship between nodes. The graph node can be a master node or a slave node, and the edge represents the data or control information flow between nodes. This directed graph is called a communication graph.

[0124] In the case where the node role of the faulty node is a slave node, the communication connection between the remaining nodes is maintained, and the communication graph of the removed node is restored in the newly joined nodes, so that the newly joined nodes perform communication connection between nodes based on the restored communication graph, thereby establishing communication connection between the remaining nodes and the newly joined nodes.

[0125] S620, resuming the large model training based on the large model cluster with established communication connection.

[0126] As described above, since the checkpoint file saves the state data of the large model cluster, the training state is restored from the checkpoint file to continue the large model training based on the large model cluster with established communication connection.

[0127] Through experiments, for an exemplary large model cluster training scenario, the method provided by the embodiments of the present application can shorten the time from 4 hours to within 3 minutes from the discovery of the faulty node to the recovery of the cluster training, greatly improving the efficiency of the large model training and reducing the waste of computing resources and labor costs caused by the interruption of the large model training task.

[0128] Please refer to FIG. 9, which is a block diagram of a fault handling device for large model cluster training according to an exemplary embodiment of the present application. As shown in FIG. 9, the device comprises:

[0129] The traffic collection module 910 is configured to periodically collect the communication traffic of each node in the large model cluster, and calculate the total communication traffic corresponding to each traffic collection time point respectively.

[0130] The training interruption module 920 is configured to, when detecting that the large model cluster has a training interruption, acquire an interruption period.

[0131] The time point determination module 930 is configured to, according to the total communication traffic corresponding to each traffic collection time point in the interruption period, select one traffic collection time point from each traffic collection time point in the interruption period as a communication interruption time point.

[0132] The node positioning module 940 is configured to determine the faulty node according to the communication traffic of each node at the communication interruption time point.

[0133] It should be noted that the apparatus provided in the above embodiments and the method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. The fault processing apparatus for large model cluster training provided in the above embodiments can be used in actual applications, and the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0134] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the fault processing method for large model cluster training provided in the above embodiments.

[0135] FIG. 10 shows a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. It should be noted that the electronic device can be the alarm apparatus 120 in the implementation environment shown in FIG. 1, or a terminal or a server in other implementation environments, which is not limited herein. It should also be noted that the computer system 1000 of the electronic device shown in FIG. 10 is only an example, and should not limit the functions and use range of the embodiments of the present application.

[0136] As shown in FIG. 10, the computer system 1000 includes a central processing unit (CPU) 1001 which can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from the storage section 1008 into a random access memory (RAM) 1003, such as performing the methods described in the above embodiments. In the RAM 1003, various computer programs and data required for system operation are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0137] Connected to the I / O interface 1005 are an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable recording medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read therefrom is installed in the storage section 1008 as necessary.

[0138] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable recording medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the system of the present application are performed.

[0139] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer program contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination of the above.

[0140] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order from that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the function involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0141] The units involved in the embodiments of the present application can be implemented in software or in hardware, and the described units can also be arranged in a processor. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0142] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which is executed by a processor of an electronic device to implement the fault processing method for large model cluster training as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0143] The embodiment of the application further provides a computer program product, which comprises a computer program stored in a computer readable storage medium. A processor of an electronic device reads the computer program from the computer readable storage medium, and the processor executes the computer program, so that the electronic device executes the fault processing method of the large model cluster training provided in each of the above embodiments.

[0144] The above is only a preferred exemplary embodiment of the application, and is not intended to limit the implementation of the application. Those skilled in the art can easily make corresponding modifications or changes according to the main concept and spirit of the application, and the protection scope of the application should be subject to the protection scope required by the claims.

[0145] It can be understood that in the specific embodiments of the application, data related to communication traffic, host group information, ranking information, etc. is involved. When the above embodiments of the application are applied to specific products or technologies, the user's permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A fault handling method for large model cluster training, comprising: periodically collecting communication traffic of each node in the large model cluster, and calculating total communication traffic corresponding to each traffic collection time; when detecting that the large model cluster has a training interruption, obtaining an interruption period; selecting one traffic collection time as a communication interruption time from the traffic collection times in the interruption period according to the total communication traffic corresponding to each traffic collection time in the interruption period; determining a faulty node according to the communication traffic of each node at the communication interruption time.

2. The method of claim 1, wherein, selecting one traffic collection time as a communication interruption time from the traffic collection times in the interruption period according to the total communication traffic corresponding to each traffic collection time in the interruption period, comprising: arranging the total communication traffic corresponding to each traffic collection time in the interruption period in chronological order to obtain a total communication traffic sequence; calculating a difference value corresponding to each total communication traffic in the total communication traffic sequence, and selecting one traffic collection time as the communication interruption time from the traffic collection times in the interruption period according to the difference value corresponding to each total communication traffic in the total communication traffic sequence.

3. The method of claim 2, wherein, calculating a difference value corresponding to each total communication traffic in the total communication traffic sequence, and selecting one traffic collection time as the communication interruption time from the traffic collection times in the interruption period according to the difference value corresponding to each total communication traffic in the total communication traffic sequence, comprising: calculating a forward difference value of each total communication traffic in the total communication traffic sequence, and constructing a forward difference sequence corresponding to the total communication traffic sequence based on the forward difference value of each total communication traffic; determining the smallest forward difference value in the forward difference sequence, and selecting the traffic collection time corresponding to the smallest forward difference value as the communication interruption time.

4. The method according to any one of claims 1 to 3, wherein, determining a faulty node according to the communication traffic of each node at the communication interruption time, comprising: determining a node with the lowest communication traffic at the communication interruption time according to the communication traffic of each node at the communication interruption time; determining the node with the lowest communication traffic as the faulty node.

5. The method according to any one of claims 1 to 4, wherein, The communication traffic is remote direct memory access (RDMA) traffic.

6. The method according to any one of claims 1 to 5, wherein, Further comprising: detecting a traffic view of a large model training task in the large model cluster to determine whether the large model cluster has a training interruption, wherein the traffic view is used to display and analyze the communication traffic change of the large model training task.

7. The method according to any one of claims 1 to 6, wherein, The method further comprises: after the faulty node is removed from the large model cluster, when a new node is detected, establishing a communication connection between the remaining nodes in the large model cluster and the new node; resuming large model training based on the large model cluster with the established communication connection.

8. The method of claim 7, wherein, When a new node is detected, establishing a communication connection between the remaining nodes in the large model cluster and the new node, comprising: restarting a networking sub-process respectively deployed in the remaining nodes and the new node; Based on the networking sub-process, inter-node networking is performed, and a communication connection between the remaining nodes and the newly added node is established.

9. The method of claim 8, wherein, Based on the networking sub-process, inter-node networking is performed, and a communication connection between the remaining nodes and the newly added node is established, including: Obtain host group information from a shared storage server, the host group information being used to record host information of each node currently contained by the large model cluster; Save the host information of the local node to the shared storage server to update the host group information in the shared storage server; Continuously query the host group information in the shared storage server, and when the number of host information contained in the queried host group information reaches a preset number value and the number does not change within a preset time length, perform networking configuration on the local node, and establish a communication connection between nodes based on the networking configuration.

10. The method of claim 9, wherein, Perform networking configuration on the local node, and establish a communication connection between nodes based on the networking configuration, including: Determine the ranking information corresponding to the local node through the host group information; Establish a communication connection between nodes based on the determined ranking information of the local node.

11. The method according to any one of claims 8-10, wherein, When a newly added node is detected, establish a communication connection between the remaining nodes in the large model cluster and the newly added node, including: Determine the node role corresponding to the fault node, the node role including a master node or a slave node; Based on the node role, establish a communication connection between the remaining nodes and the newly added node.

12. The method of claim 11, wherein, Based on the node role, re-establish a communication connection between the remaining nodes and the newly added node, including: When the node role is a master node, re-determine a master node from the remaining nodes and the newly added node, and take other nodes except the re-determined master node as slave nodes; Establish a communication connection between the re-determined master node and the slave nodes.

13. The method of claim 12, wherein, Establish a communication connection between the re-determined master node and the slave nodes, including: Broadcast a master node update message by the re-determined master node, the master node update message containing a node address of the re-determined master node; Based on the node address of the re-determined master node, re-establish a communication graph, and establish a communication connection between nodes based on the communication graph.

14. The method of claim 11, wherein, Based on the node role, re-establish a communication connection between the remaining nodes and the newly added node, including: When the node role is a slave node, maintain the existing communication connection between the remaining nodes, and restore a communication graph of a removed node in the newly added node, so that the newly added node establishes a communication connection between nodes based on the restored communication graph.

15. A fault handling apparatus for large model cluster training, comprising: a traffic collection module configured to periodically collect communication traffic of each node in the large model cluster and calculate total communication traffic corresponding to each traffic collection time; a training interruption module configured to obtain an interruption period when detecting that the large model cluster has a training interruption; a time determination module, configured to select one traffic collection time from each traffic collection time in the interruption period as the communication interruption time according to the total communication traffic corresponding to each traffic collection time in the interruption period; a node positioning module, configured to determine the fault node according to the communication traffic of each node at the communication interruption time. 16.An electronic device, comprising: one or more processors; a memory for storing one or more computer programs that, when executed by the one or more processors, cause the electronic device to implement the method of any one of claims 1-14. 17.A computer-readable storage medium having stored thereon a computer program that, when executed by a processor of an electronic device, causes the electronic device to perform the method of any one of claims 1-14. 18.A computer program product comprising a computer program that, when executed by a processor of an electronic device, implements the method of any one of claims 1-14.

Images