Fault handling method and related device based on large model cluster training

Abstract

The embodiments of the present application disclose a fault handling method and related devices based on large model cluster training. The method includes: periodically collecting the communication traffic of each node in the large model training cluster; when a training interruption is detected in the large model training cluster, obtaining the interruption period, and calculating the total communication traffic corresponding to each traffic collection moment contained in the interruption period according to the communication traffic of each node at each traffic collection moment; determining the communication interruption moment according to the total communication traffic corresponding to each traffic collection moment in the interruption period; determining the faulty node according to the communication traffic of each node at the communication interruption moment. The embodiments of the present application can realize the rapid positioning of faulty nodes in the large model training cluster.

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a fault handling method and related devices based on large model cluster training. Background Art

[0002] In the field of artificial intelligence (AI), large models refer to deep learning models with large-scale parameters, which can usually reach millions, tens of millions, or even hundreds of millions. Such models have high expressive power and predictive performance and can handle more complex tasks.

[0003] Large models have extremely high requirements for computing resources and storage space, and a single computing node often cannot meet the requirements. Therefore, cluster training becomes an effective solution, which significantly improves training efficiency by distributing training tasks to multiple computing nodes. The network clusters used for large model training are often huge in scale, with a large number of devices and communication links, resulting in a high frequency of failures, which in turn makes it impossible for large model training tasks to run stably for a long time.

[0004] At present, when large model training is interrupted, it is necessary to manually collect the logs of the computing nodes and compare and analyze them node by node, which takes a long time to locate the faulty node. During this period, the computing resources of the nodes in the large model training cluster are in a waiting state, resulting in a large amount of computing resources being lost. Therefore, a more efficient faulty node location solution is urgently needed. Summary of the invention

[0005] To solve the above technical problems, the embodiments of the present application provide a fault handling method and device based on large model cluster training, an electronic device, a computer-readable storage medium, and a computer program product. The embodiments of the present application can quickly locate the faulty node in the large model training cluster after the large model cluster training is interrupted.

[0006] One aspect of an embodiment of the present application provides a fault handling method based on large model cluster training, the method comprising: periodically collecting communication traffic of each node in the large model training cluster; when a training interruption is detected in the large model training cluster, obtaining an interruption period, and calculating the total communication traffic corresponding to each traffic collection moment included in the interruption period according to the communication traffic of each node at each traffic collection moment; determining the communication interruption moment according to the total communication traffic corresponding to each traffic collection moment in the interruption period; determining the faulty node according to the communication traffic of each node at the communication interruption moment.

[0007] In another exemplary embodiment, the communication interruption time is determined according to the total communication flow corresponding to each flow collection moment within the interruption period, including: arranging the total communication flow corresponding to each flow collection moment within the interruption period in chronological order to obtain a total communication flow sequence; calculating the differential values ​​corresponding to each total communication flow in the total communication flow sequence, and determining the flow collection moment to which the total communication flow corresponding to the minimum differential value belongs as the communication interruption time.

[0008] In another exemplary embodiment, the differential values ​​corresponding to each total communication flow in the total communication flow sequence are calculated, and the flow collection moment to which the total communication flow corresponding to the minimum differential value belongs is determined as the communication interruption moment, including: calculating the forward differential value for each total communication flow in the total communication flow sequence, and constructing a forward differential sequence corresponding to the total communication flow sequence based on the forward differential values ​​corresponding to each total communication flow; traversing the minimum forward differential value in the forward differential sequence, and determining the flow collection moment corresponding to the minimum forward differential value as the communication interruption moment.

[0009] In another exemplary embodiment, the faulty node is determined based on the communication traffic of each node at the time of the communication interruption, including: determining the node with the lowest communication traffic at the time of the communication interruption according to the communication traffic of each node at the time of the communication interruption; and using the determined node with the lowest communication traffic as the faulty node.

[0010] In another exemplary embodiment, the method further includes: after the faulty node is removed from the large model training cluster, when a newly joined node is detected, re-establishing the communication connection between the remaining nodes in the large model training cluster and the newly joined node; and resuming the large model training cluster based on the re-established communication connection to perform large model training.

[0011] In another exemplary embodiment, when a newly joined node is detected, the communication connection between the remaining nodes in the large model training cluster and the newly joined node is re-established, including: restarting the networking sub-processes deployed respectively in the remaining nodes and the newly joined node; performing inter-node networking based on the networking sub-process to re-establish the communication connection between the remaining nodes and the newly joined node.

[0012] In another exemplary embodiment, inter-node networking is performed based on the networking sub-process to re-establish the communication connection between the remaining nodes and the newly joined node, including: obtaining host group information from a shared storage service client, the host group information being used to record the host information corresponding to the nodes currently contained in the large model training cluster; saving the host information of the local node to the shared storage service client to update the host group information in the shared storage service client; continuously querying the host group information in the shared storage service client, and when the number of host information contained in the queried host group information reaches a preset number value, and the number does not change within a preset time period, the local node is networked to establish a communication connection between the nodes based on the networking configuration.

[0013] In another exemplary embodiment, a networking configuration is performed on local nodes, and a communication connection is established between nodes based on the networking configuration, including: determining ranking information corresponding to the local nodes through the host group information; and establishing a communication connection between nodes based on the determined ranking information of the local nodes.

[0014] In another exemplary embodiment, when a newly joined node is detected, the communication connection between the remaining nodes in the large model training cluster and the newly joined node is re-established, including: determining the node role corresponding to the faulty node, the node role including a master node or a slave node; and re-establishing the communication connection between the remaining nodes and the newly joined node based on the node role.

[0015] In another exemplary embodiment, the node role is a master node; based on the node role, the communication connection between the remaining nodes and the newly joined node is re-established, including: redetermining the master node from the remaining nodes and the newly joined nodes, and using other nodes except the redetermined master node as slave nodes; establishing a communication connection between the redetermined master node and the slave node.

[0016] In another exemplary embodiment, establishing a communication connection between a re-determined master node and a slave node includes: broadcasting a master node update message by the re-determined master node, wherein the master node update message contains a node address of the re-determined master node; and re-establishing a communication graph based on the node address of the re-determined master node to establish a communication connection between nodes based on the communication graph.

[0017] In another exemplary embodiment, the node role is a slave node; based on the node role, the communication connection between the remaining nodes and the newly joined node is reestablished, including: maintaining the communication connection between the remaining nodes, and restoring the communication graph of the removed nodes in the newly joined node, so that the newly joined node can establish a communication connection between nodes based on the restored communication graph.

[0018] On the other hand, an embodiment of the present application provides a fault handling device based on large model cluster training, which includes: a traffic collection module, configured to periodically collect the communication traffic of each node in the large model training cluster; a traffic calculation module, configured to obtain the interruption period when a training interruption in the large model training cluster is detected, and calculate the total communication traffic corresponding to each traffic collection moment included in the interruption period according to the communication traffic of each node at each traffic collection moment; a time determination module, configured to determine the communication interruption time according to the total communication traffic corresponding to each traffic collection moment in the interruption period; a node positioning module, configured to determine the faulty node according to the communication traffic of each node at the communication interruption moment.

[0019] Another aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the fault handling method based on large model cluster training as described above.

[0020] Another aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the fault handling method based on large model cluster training as described above.

[0021] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor of an electronic device, implements the fault handling method based on large model cluster training as described above.

[0022] In the technical solution provided in the embodiments of the present application, the communication traffic of each node in the large model training cluster is periodically collected, and when the large model training is interrupted, the communication interruption time is determined according to the total communication traffic corresponding to each traffic collection time during the interruption period, and then the faulty node is determined according to the communication traffic of each node at the time of communication interruption. Therefore, when the large model training is interrupted, the faulty node in the large model training cluster is automatically analyzed based on the collected communication traffic, which can make the faulty node location in the large model training cluster more efficient.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of an exemplary implementation environment involved in this application.

[0025] Figure 2 Indicates the Figure 1 A schematic diagram of the architecture of a large model training cluster 110 for performing network status detection in an example implementation environment.

[0026] Figure 3 It is a flowchart of a fault handling method based on large model cluster training shown in an exemplary embodiment of the present application.

[0027] Figure 4 A schematic diagram of a traffic view for an exemplary large model training task is shown.

[0028] Figure 5 is Figure 3 A flowchart of a fault handling method based on large model cluster training further proposed on the basis of the illustrated embodiment.

[0029] Figure 6 It is a flowchart of a fault handling method based on large model cluster training shown in another exemplary embodiment of the present application.

[0030] Figure 7 is Figure 6 A flowchart of another fault handling method based on large model cluster training is proposed on the basis of the illustrated embodiment.

[0031] Figure 8 is Figure 6 A flowchart of another fault handling method based on large model cluster training is proposed on the basis of the illustrated embodiment.

[0032] Fig. 9 It is a block diagram of a fault handling device based on large model cluster training shown in an exemplary embodiment of the present application.

[0033] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0038] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] First of all, it should be noted that the big model is also called the pre-training model (PTM) or the cornerstone model. It refers to a deep neural network (DNN) with large parameters. It is trained on a large amount of unlabeled data. The function approximation ability of the large-parameter DNN is used to enable the big model to extract common features from the data. After fine tuning, parameter efficient fine tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the big model can achieve ideal results in few-shot or zero-shot scenarios.

[0041] According to the data modality processed, large models can be divided into language models (such as ELMO, BERT, GPT), visual models (such as swin-transformer, ViT, V-MOE), speech models (such as VALL-E), multimodal models (such as ViBERT, CLIP, Flamingo, Gato), etc., among which multimodal models refer to models that establish feature representations of two or more data modalities. Large models are important tools for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface to connect multiple specific task models.

[0042] Large models usually have the following characteristics:

[0043] First, distributed training;

[0044] Distributed training refers to splitting the workload of the training model and sharing it among multiple microprocessors. Large models have large parameters and large training data, which exceed the capacity of a single machine, so distributed parallelism is needed to speed up. Parallel mechanisms can include data parallelism (DP), model parallelism (MP), pipeline parallelism (PP), and hybrid parallelism (HP). Structural designs include those based on parameter servers, reduce, and MPI (Message Passing Interface).

[0045] Second, model compression and quantization;

[0046] Model compression and quantization refers to reducing the model size and accelerating model reasoning through compression and quantization techniques, thereby reducing the model's storage and computing costs. Model compression usually includes pruning, low-rank decomposition, knowledge distillation, etc. Model quantization refers to converting floating-point parameters in the model into fixed-point or integer parameters, thereby reducing the model size and accelerating model reasoning.

[0047] Third, adaptive computing;

[0048] Adaptive computing refers to automatically adjusting the amount of computation and accuracy of the model based on different input data, so as to achieve the purpose of improving the model's computational efficiency while maintaining the model's accuracy. Adaptive computing can flexibly adjust the amount of computation and accuracy of the model based on different input data, thereby better balancing the model's computational efficiency and accuracy.

[0049] Fourth, model parallel computing;

[0050] Model parallel computing refers to assigning the model's computing tasks to multiple computing devices (such as CPU, GPU, TPU, etc.) to perform computing simultaneously, thereby accelerating the training and reasoning of the model. Model parallel computing can effectively utilize computing resources and improve the computing efficiency and training speed of the model.

[0051] Large models have extremely high requirements for computing resources and storage space, and a single computing node often cannot meet the requirements. Therefore, cluster training becomes an effective solution, which significantly improves training efficiency by distributing training tasks to multiple computing nodes. The network clusters used for large model training are often huge in scale, with a large number of devices and communication links, resulting in a high frequency of failures, which in turn makes it impossible for large model training tasks to run stably for a long time.

[0052] In related technologies, when a large model training task is interrupted, the following processing flow is usually performed:

[0053] 1. Collect node logs: Operation and maintenance personnel need to log in to each node device and collect relevant log files. Since the number of nodes in the large model training cluster is large, this process is very time-consuming.

[0054] 2. Analyze alarm information: Alarm information usually contains key information about the equipment's operating status. However, since alarm information may come from different systems and devices, manual comparison and analysis is required to identify certain devices or systems that have caused the failure.

[0055] 3. Compare nodes one by one: In order to further locate network or device problems, operation and maintenance personnel may need to compare the logs and performance indicators of multiple node devices. This process requires writing scripts for careful comparison and analysis in order to find the root cause of the faulty device.

[0056] 4. Fault location: After identifying the possible faulty device, the operation and maintenance personnel need to conduct further analysis to determine the specific cause of the problem. This process may involve code review, hardware testing and other aspects.

[0057] It can be concluded that the current fault location technology has at least disadvantages such as long time consumption, high labor costs, and high error risks. Due to the above disadvantages, it usually takes more than one day to recover from a normal training interruption. During this recovery period, the computing resources of the node are in a waiting state, resulting in a large amount of computing resource loss. This loss of computing power not only affects the current large model training tasks, but may also affect other businesses that rely on these resources.

[0058] In order to solve the above technical problems, the embodiments of the present application propose a fault handling method based on large model cluster training, a fault handling device based on large model cluster training, an electronic device, a computer readable storage medium and a computer program product. These embodiments will be described in detail below.

[0059] First see Figure 1 , Figure 1 1 is a schematic diagram of an exemplary implementation environment involved in the present application. The implementation environment is a large model training system, including a large model training cluster 110, an alarm device 120 and a user terminal 130.

[0060] The large model training cluster 110 is composed of multiple nodes, each of which can be a device with a GPU (Graphics Processing Unit), such as a terminal, a server, etc., which is not limited here. By distributing the large model training tasks to multiple nodes, efficient large model training can be performed.

[0061] The nodes in the large model training cluster 110 can be divided into master nodes and slave nodes. The master node is mainly responsible for coordinating the training tasks in the entire cluster, for example, it may include assigning training tasks to each slave node, managing the distribution and synchronization of training data, detecting the status of each slave node, collecting and summarizing the local gradients calculated by each slave node and performing global parameter updates, distributing the updated parameters to each slave node, etc. The slave node is mainly responsible for executing the training tasks assigned by the master node, such as calculating the local gradient, sending the local gradient to the master node for aggregation, and receiving the updated parameters from the master node.

[0062] Since large models have extremely high requirements for computing resources and storage space, the large model training cluster 110 often has a huge network scale, with a large number of nodes and communication links, resulting in a high frequency of failures, which in turn causes the large model training task to be unable to run stably for a long time. Therefore, an alarm device 120 is set to detect the flow of the large model training process carried out in the large model training cluster 110. When an interruption in the large model training is detected, the corresponding interruption period is obtained, and the total communication flow corresponding to each flow collection moment included in the interruption period is calculated. Then, the communication interruption time is determined according to the total communication flow corresponding to each flow collection moment in the interruption period, and then the faulty node is determined according to the communication flow of each node at the communication interruption time, thereby quickly and accurately locating the faulty node of the large model training cluster 110.

[0063] After determining the faulty node, the alarm device 120 may send relevant information of the faulty node to the user terminal 130, so that the operation and maintenance personnel can obtain relevant information of the faulty node through the user terminal 130, thereby triggering corresponding fault processing for the faulty node. For example, the faulty node may be removed from the large model training cluster 110, and another node device may be added to the large model training cluster 110, thereby resuming the large model training.

[0064] The alarm device 120 may also have an alarm prompt function. For example, the alarm device 120 may be equipped with alarm prompt components such as a display panel or screen, a voice module, etc., so that after determining the faulty node, the alarm device 120 can prompt the operation and maintenance personnel to perform corresponding fault processing on the specific faulty node through the alarm prompt components configured by itself.

[0065] It should be noted that the alarm device 120 can be a terminal such as a computer, a smart phone, a tablet computer, a laptop computer, or an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, which is not limited in this embodiment. The user terminal 130 can be a terminal such as a computer, a smart phone, a tablet computer, a laptop computer, or the like, which is not limited in this embodiment.

[0066] Please continue reading Figure 2 , Figure 2 Indicates the Figure 1 A schematic diagram of the architecture of the large model training cluster 110 for network status detection in the exemplary implementation environment. Figure 2 As shown, it specifically illustrates that the network status of each node in the large model training cluster 110 can be detected by INT (Inband Network Telemetry) detection message and SFLOW (Solution for Network Monitoring) collection business flow.

[0067] INT is a technology that uses data plane services to collect network status information. It carries and transmits telemetry information through data plane service traffic and does not rely on the control plane. By embedding INT instructions in data messages, these instructions are interpreted and filled in by the forwarding device, and then reported step by step, thus achieving real-time monitoring of network status and remote reporting of data. Figure 2 As shown, through the INT detection message, network status data such as PFC (Priority Flow Control) messages and CNP (Congestion Notification Packet) messages of each node in the large model training cluster 110 can be obtained.

[0068] SFLOW is a network traffic detection technology based on packet sampling, which is mainly used for statistical analysis of network traffic. The SFLOW agent is embedded in the ASIC (Application Specific Integrated Circuit) chip of the network device being detected, which supports flow-based data packet statistical sampling and time-based network interface statistical sampling, and forwards the sampled data packets to the specified SFLOW traffic sampler for analysis. For example, for Figure 1 In the example implementation environment, a SFLOW agent program can be deployed in each node to sample the data packets of each node, and a SFLOW traffic sampler program can be deployed in the alarm device 120, so that the alarm device 120 can collect the communication traffic of each node in the large model training cluster 110.

[0069] Exemplarily, SFLOW technology can support the collection and reporting of five-tuple traffic information, which refers to five key elements in network communication, including source IP (Internet Protocol) address, source port number, destination IP address, destination port number and transport layer protocol, through which a specific communication flow in the network is defined. Therefore, by collecting business flows through SFLOW, the communication flow of each node in the large model training cluster 110 can be obtained.

[0070] In actual application scenarios, appropriate detection technology can be selected according to actual application requirements to detect the network status of the large model training cluster 110, or a combination of multiple technologies can be used to detect the network status of the large model training cluster 110 to achieve more comprehensive network detection. This embodiment does not limit this.

[0071] See also Figure 3 , Figure 3is a flowchart of a fault handling method based on large model cluster training shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment illustrated in the example may be specifically executed by the alarm device 120, or jointly executed by the alarm device 120 and the large model training cluster 110. Of course, the method may also be applied to other implementation environments, and executed by a terminal or server in other implementation environments, or jointly executed by a terminal and a server in other implementation environments, and this embodiment does not limit this.

[0072] like Figure 3 As shown, in an exemplary embodiment, the fault handling method based on large model cluster training includes S310-S340, which are described in detail as follows:

[0073] S310, periodically collect the communication traffic of each node in the large model training cluster.

[0074] The first thing you need to understand is that communication traffic refers to the amount of communication services between two switching centers in the network, that is, the amount of information data that needs to be transmitted between the two switching centers, usually the sum of the traffic in both directions between the two switching centers.

[0075] Can be Figure 2 The SFLOW business flow collection method shown in the figure is used to periodically collect the communication traffic of each node in the large model training cluster, or other methods are used to periodically collect the communication traffic of each node. This embodiment is not limited to this.

[0076] Exemplarily, the data transmission message can be reported by the switch at a preset sampling ratio, such as a sampling ratio of 64K:1 or other sampling ratios, so that the communication flow can be calculated based on the reported sampled messages and the sampling ratio. It is understandable that the sampling ratio of 64K:1 means that one message is sampled and reported out of every 64,000 data transmission messages. The switch can also report the communication flow of each node once every preset time period, and this embodiment does not limit this.

[0077] S320, when a training interruption is detected in the large model training cluster, the interruption period is obtained, and based on the communication traffic of each node at each traffic collection time, the total communication traffic corresponding to each traffic collection time included in the interruption period is calculated.

[0078] In some exemplary embodiments, the traffic view of the large model training task can be detected to determine whether the large model training cluster has a training interruption. The traffic view is used to display and analyze the communication traffic changes of the large model training task.

[0079] During the large model training process, the communication traffic will decrease at regular intervals due to the checkpoint operation (checkpoint is used to save the current state of the model, such as model parameters, optimizer state, etc.). This traffic decrease is considered normal. If a traffic decrease other than this normal situation is detected, it means that a training interruption has been detected. Alternatively, it can be understood that a training interruption will cause an abnormal decrease in the traffic of the large model training task. If an abnormal decrease in traffic is detected, it means that a training interruption has been detected. Since the decrease in traffic caused by the checkpoint operation during the large model training process is relatively slow, if the traffic at a certain moment is detected to be below the preset proportion of the standard traffic, such as below 80% of the standard traffic, it can be determined that the traffic decrease is not caused by the checkpoint operation, and it is determined that an abnormal decrease in traffic has occurred, thereby detecting that the large model training has been interrupted.

[0080] Figure 4 A schematic diagram of a flow view of an exemplary large model training task is shown, such as Figure 4 As shown in the figure, during 08:46-09:00 and 10:15-10:40, the traffic of the large model training task dropped to less than 80% of the standard traffic of 200,000 Gbps, which was regarded as an abnormal traffic drop. Therefore, the training interruption of the large model training cluster was detected at the time of the abnormal traffic drop.

[0081] The interruption period indicates the time period corresponding to the training interruption of the large model training cluster. For example, after the moment when the training interruption of the large model training cluster occurs is detected, the time period of the preset duration before and after this moment is obtained as the interruption period. For example, if the moment when the training interruption of the large model training cluster occurs is represented as T, the time period of T-10min~T+10min represents the interruption period. The moment T when the training interruption of the large model training cluster occurs can also be called the moment when the traffic drops abnormally.

[0082] The traffic collection moment is the moment when the communication traffic of each node is actually collected. After obtaining the interruption period, the communication traffic of each node in the large model training cluster at each traffic collection moment is calculated for each traffic collection moment, thereby obtaining the total communication traffic corresponding to each traffic collection moment included in the interruption period.

[0083] S330, determining a communication interruption time according to the total communication traffic corresponding to each traffic collection time within the interruption period.

[0084] This embodiment takes into account the communication propagation chain of large model cluster training. In essence, all nodes participating in the training constitute a communication graph. If a faulty node appears, other nodes communicating with the faulty node will cause communication anomalies. This anomaly is usually manifested as an abnormal decrease in the overall traffic of the large model training task. Therefore, this embodiment can determine the moment when the overall traffic of the large model training task decreases abnormally based on the total communication traffic corresponding to each traffic collection moment during the interruption period, and use this moment as the communication interruption moment.

[0085] It should be noted that this embodiment does not directly use the moment when the training interruption of the large model training cluster is detected as the communication interruption moment, but determines the communication interruption moment according to the total communication flow corresponding to each flow collection moment in the interruption period. This is because due to realistic factors, such as network delays or certain business function settings, the moment when the training interruption of the large model training cluster is detected may not necessarily be the actual communication interruption moment. Therefore, after detecting that the training interruption of the large model training cluster occurs, this embodiment obtains the interruption period, and then determines the communication interruption moment according to the total communication flow corresponding to each flow collection moment in the interruption period, which can improve the accuracy of the communication interruption moment, and further promote the accuracy of locating the faulty node.

[0086] S340, determining the faulty node according to the communication traffic of each node at the time of communication interruption.

[0087] This embodiment also takes into account that the faulty node is the root cause node that causes the interruption of large model training. At the moment of communication interruption, the faulty node should be the first node whose communication is blocked, and the phenomenon of traffic drop at the faulty node is more obvious. Therefore, it is possible to determine the node with the lowest communication traffic at the moment of communication interruption, and determine the node with the lowest communication traffic as the faulty node.

[0088] It can be concluded that the technical solution provided by this embodiment automatically analyzes the faulty nodes in the large model training cluster based on the collected communication traffic when the large model training is interrupted, which can make the faulty node location in the large model training cluster more efficient and reduce the waste of computing resources caused by the interruption of large model training. In addition, when the large model training is interrupted, the present embodiment first determines the communication interruption time according to the total communication traffic corresponding to each traffic collection time in the interruption period, and then determines the faulty node according to the communication traffic of each node at the time of communication interruption, which can accurately locate the root cause node causing the interruption of large model training, which also shortens the time required for large model training recovery to a certain extent, so as to further reduce the waste of computing resources caused by the interruption of large model training.

[0089] In another exemplary embodiment, after the faulty node is determined, the faulty node can be further determined to be a network fault or a device fault according to the network status information corresponding to the faulty node. If it is a network fault, the large model training can be continued after the network is restored. If it is a device fault, a new node device needs to be replaced. For example, Figure 2 The network status information of the faulty node may be obtained by using the INT detection message method shown in the figure, or other methods may be used to obtain the network status information of the faulty node, which is not limited here. The network status information of the faulty node may include GPU utilization, PFC message, ECN message, etc., which is not limited here.

[0090] Please read further Figure 5 , Figure 5 is Figure 3 A flowchart of a fault handling method based on large model cluster training further proposed on the basis of the illustrated embodiment.

[0091] like Figure 5 As shown, in an exemplary embodiment, the process of determining the communication on-off time according to the total communication flow corresponding to each flow collection time in the interruption period in S330 includes S510-S520, which are described in detail as follows:

[0092] S510, arranging the total communication flows corresponding to each flow collection moment in the interruption period in chronological order to obtain a total communication flow sequence.

[0093] As mentioned above, in a large model training cluster, all nodes participating in the training essentially form a communication graph, in which if a faulty node appears, other nodes communicating with the faulty node will have communication anomalies. On this basis, in order to eliminate the influence of other nodes with reduced traffic, this embodiment uses differential values ​​to smooth out the larger traffic reduction of other nodes to accurately determine the faulty node.

[0094] In order to facilitate understanding of the principle of using differential values ​​in this embodiment to smooth out the larger traffic drop of other nodes, for example, assuming that the communication traffic of the faulty node (i.e., the root cause node) drops from 100 Gbps at time t-1 to 20 Gbps at time t, the traffic drop is 80%, and the differential value is (20-100) / 1s=-80, while the communication traffic of another node drops from 60 Gbps at time t-1 to 5 Gbps at time t, the traffic drop is 91.7%, and the differential value is (5-60) / 1s=-55. The traffic drop of the other node is obviously larger than the traffic drop of the faulty node, but the differential value of the other node is smaller than the differential value of the faulty node. It can be obtained that the use of differential values ​​can smooth out the larger traffic drop of other nodes, thereby accurately locating the faulty node.

[0095] This embodiment first accurately determines the communication interruption time based on the differential value, specifically determines the flow collection time corresponding to the minimum differential value as the communication interruption time, and then determines the node with the lowest communication flow at the communication interruption time as the faulty node, which is equivalent to the faulty node being the node with the lowest flow at the communication interruption time with the smallest differential value. Since the communication interruption time is determined based on the differential value, under this premise, the node with the lowest flow at the communication interruption time is also the node with the largest flow drop, so the faulty node determined by this embodiment is also equivalent to the one determined based on the differential value, which can ensure accuracy.

[0096] It can be understood that differential refers to a way of calculating the difference in the values ​​of a sequence at different points, and the result of differential calculation reflects the changes between discrete quantities. Therefore, it is necessary to arrange the total communication traffic corresponding to each traffic collection moment in the interruption period in chronological order to obtain the total communication traffic sequence, and then perform differential operation on the total communication traffic sequence to determine the communication interruption time based on the result of differential operation.

[0097] S520, calculating the difference values ​​corresponding to the respective total communication flows in the total communication flow sequence, and determining the flow collection time to which the total communication flow corresponding to the minimum difference value belongs as the communication interruption time.

[0098] Difference operations can be divided into three types: forward difference, backward difference, and central difference. Forward difference refers to subtracting the value of a point from the value of the sequence after that point, and the result is the forward difference value of the point. Backward difference refers to subtracting the value of a point from the value before that point, and the result is the backward difference value of the point. Central difference takes the difference between the average value of the two points before and after a point and the value of the point as the central difference value of the point.

[0099] Since the present embodiment performs fault location and focuses on the abnormal decrease in traffic, it is more suitable to use the forward differential value to determine the communication interruption time. For example, the forward differential value is calculated for each communication total flow in the communication total flow sequence, and the forward differential sequence corresponding to the communication total flow sequence is constructed based on the forward differential value corresponding to each communication total flow, and then the smallest forward differential value in the forward differential sequence is traversed to determine the traffic collection time corresponding to the smallest forward differential value as the communication interruption time.

[0100] For example, the total communication flow sequence is represented as S1: {( , ), ( ),…,( )}, and the forward difference value is expressed as ,in, , by calculating the forward difference value at each moment in the total communication flow sequence S1, we can get the forward difference sequence S2: {( ),( ),…,( )}, so that the minimum forward difference value min( , the minimum forward difference value min( The corresponding time is taken as the communication interruption time.

[0101] If the communication interruption time is expressed as , and each node in the large model training cluster is The traffic sequence at each moment is represented as S3: {( , , ), ( )…, ( )},in Represents a training node , then the node corresponding to the smallest flow value in the flow sequence S3 can be traversed, and this node is the fault node.

[0102] In other exemplary embodiments, the communication interruption time can also be determined by using backward difference or central difference, but some improvements are still needed. For example, in the case of using backward difference value to determine the communication interruption time, the maximum backward difference value should be traversed for the constructed backward difference sequence, and the next moment of the flow collection time corresponding to the maximum backward difference value obtained by traversal is determined as the communication interruption time. For another example, in the case of using central difference value to determine the communication interruption time, the maximum central difference value should also be traversed for the constructed central difference sequence, and the next moment of the flow collection time corresponding to the maximum central difference value obtained by traversal is determined as the communication interruption time. Alternatively, an improved central difference sequence can also be constructed, in which the differential value corresponding to each flow collection time is the difference between the average value of the later time and the previous time in the total communication flow sequence and the current time value, and the smallest differential value in the improved central difference sequence is traversed, and the next moment of the flow collection time corresponding to the maximum central difference value obtained by traversal can be determined as the communication interruption time.

[0103] Regardless of which differential value scheme is used, since the differential value can smooth out the larger traffic drop of other nodes, the faulty node can be accurately located. However, it should be noted that the differential value schemes in the above examples are all first-order differential values. In other embodiments, multi-order differential schemes such as second-order differentials can also be used to locate the faulty node, and this embodiment does not limit this.

[0104] Please continue reading Figure 6 , Figure 6 FIG. 1 is a flowchart of a fault handling method based on large model cluster training, shown in another exemplary embodiment of the present application. Figure 6 As shown, in an exemplary embodiment, the method Figure 3 The embodiment shown further includes S610-S620, which are described in detail as follows:

[0105] S610, after the faulty node is removed from the large model training cluster, when a newly added node is detected, the communication connection between the remaining nodes in the large model training cluster and the newly added node is re-established.

[0106] After the faulty node is determined, this embodiment further cooperates with the business side to quickly recover the large model cluster training. Exemplarily, the faulty node needs to be removed from the large model training cluster, and a new node needs to be added to the large model training cluster, so that the new node can replace the faulty node to recover the large model training. For example, after receiving the prompt information about the faulty node, the operation and maintenance personnel manually move the faulty node from the large model training cluster, and then add another new device to the large model training cluster.

[0107] When a newly added node is detected, it is necessary to re-establish the communication connection between the remaining nodes in the large model training cluster after the failed node is removed and the newly added node, so as to resume the large model training based on the large model training cluster with the re-established communication connection.

[0108] For example, Figure 7 As shown, the process of re-establishing the communication connection between the remaining nodes in the large model training cluster and the newly added node may include the following steps:

[0109] S710, restart the networking subprocesses deployed in the remaining nodes and the newly added node respectively;

[0110] S720, performing inter-node networking based on the networking sub-process deployed in each node to re-establish the communication connection between the remaining nodes and the newly added node.

[0111] It can be understood that the node devices that can be used to build a large model training cluster are all deployed with networking sub-processes, and the networking sub-processes are used to realize automatic networking between nodes, thereby establishing cluster communication of the large model training cluster.

[0112] Therefore, this embodiment restarts the networking sub-processes respectively deployed in the remaining nodes and the newly joined nodes, thereby automatically performing inter-node networking based on restarting the networking sub-processes respectively deployed in the remaining nodes and the newly joined nodes, thereby re-establishing the communication connection between the remaining nodes and the newly joined nodes.

[0113] Exemplarily, the process of the networking subprocess performing inter-node networking may include the following steps:

[0114] S721, obtaining host group information from the shared storage service end, where the host group information is used to record host information corresponding to the nodes currently contained in the large model training cluster;

[0115] S722, saving the host information of the local node to the shared storage service client to update the host group information in the shared storage service client;

[0116] S723, continuously query the host group information in the shared storage service client. When the number of host information contained in the queried host group information reaches a preset number value, and the number of host information contained in the queried host group information does not change within a preset time period, perform networking configuration on the local node and establish a communication connection between the nodes based on the networking configuration.

[0117] It can be understood that the shared storage service end is used to provide a shared storage space, which is used to store host group information (Host Group), and the host group information is used to record the host information corresponding to all nodes currently contained in the large model training cluster. The networking subprocess obtains the host group information recorded in the front of the room from the shared storage service end, and then saves the host information of the local node in the shared storage service end. This can be understood as the networking subprocess adding its own node to the host group and synchronizing the host information to the host group information recorded by the shared storage service end. The networking subprocess continuously queries the host group information in the shared storage service end. When the number of host information contained in the queried host group information reaches a preset number value, and the number of host information contained in the queried host group information is within the preset time length, it means that all nodes are ready for networking, so the local node is configured for networking to establish a communication connection between nodes based on the networking configuration.

[0118] The networking subprocess configures the networking of the local node, which may include configuring local rank, global rank, world size and other information, so as to automatically perform cluster communication networking based on the configured information to complete the training.

[0119] Local rank can be called local ranking, which refers to the ranking or identifier between multiple computing units (such as GPUs) on a single node, which is used for parallel computing and communication operations within the node. For example, in each computing unit, the local rank value can be used to determine the role and task within the node, as well as communication and data exchange with other computing units.

[0120] Global rank can be called global ranking, which refers to the ranking or identifier between each node in a large model training cluster.

[0121] World size refers to the number of nodes in the large model training cluster, that is, the number of nodes participating in the large model training. World size can ensure that all nodes in the large model training cluster can participate in the large model training process. It also helps to reasonably divide the large model training tasks to different nodes, and realize data distribution, parameter synchronization, gradient aggregation and other operations.

[0122] Therefore, the networking sub-process can obtain the ranking information corresponding to the local node based on the host group information in the shared storage service client, such as local rank, global rank, world size, etc., and then establish the communication connection between the nodes based on the determined ranking information, thereby realizing the re-establishment of the communication connection between the remaining nodes and the newly joined nodes.

[0123] In another exemplary embodiment, Figure 8 As shown, the process of re-establishing the communication connection between the remaining nodes in the large model training cluster and the newly added node may also include the following steps:

[0124] S810, determining a node role corresponding to the faulty node;

[0125] S820: Re-establish the communication connection between the remaining nodes and the newly added node according to the determined node role.

[0126] It is understandable that the node roles include master nodes or slave nodes. As mentioned above, in a large model training cluster, the master node is mainly responsible for coordinating the training tasks in the entire cluster, and the slave node is mainly responsible for executing the training tasks assigned by the master node.

[0127] Different node roles correspond to different communication connection strategies. Therefore, this embodiment first determines the node role corresponding to the failed node, and then adopts the communication connection strategy matching the node role to re-establish the communication connection between the remaining nodes and the newly added node.

[0128] In the case where the node role of the failed node is the master node, since the previous master node has been removed from the large model training cluster, when a newly added node is detected, it is necessary to redetermine the master node and slave node from the remaining nodes and the newly added nodes, so as to establish a communication connection between the redetermined master node and slave node.

[0129] Exemplarily, a master node may be elected from the remaining nodes and the newly added nodes through an election mechanism, and all nodes except the newly determined master node are used as slave nodes. For example, a new master node may be elected between the remaining nodes and the newly added nodes through a heartbeat mechanism or other election algorithms, and this embodiment does not limit the specific content of the election mechanism.

[0130] In some exemplary embodiments, after the failed node is removed, the remaining nodes will detect that the master node is unavailable, and thus initiate their own deployed election mechanism to re-elect the master node from the remaining nodes and the newly joined nodes.

[0131] The re-determined master node takes over the management responsibilities in the large model training cluster. For example, the re-determined master node broadcasts a master node update message, and the master node update message contains the node address of the re-determined master node, so as to update the communication graph according to the node address of the re-determined master node, thereby establishing communication connections between nodes based on the updated communication graph.

[0132] It is understandable that in a large model training cluster, a communication graph needs to be established between the master node and the slave node based on certain logic and data dependencies to ensure that the training task can be executed efficiently and correctly. Exemplarily, a checkpoint file is used to save the state data of the large model training cluster, such as the state of the training task execution, data partition information, dependencies, etc., to restore the computing state of the node when the training task fails or needs to be restarted. Therefore, a directed graph can be constructed based on the state data in the checkpoint file to represent the communication relationship between nodes. The graph nodes can be master nodes or slave nodes, and the edges represent the flow of data or control information between nodes. This directed graph is called a communication graph.

[0133] When the node role of the failed node is a slave node, the communication connection between the remaining nodes is maintained, and the communication graph of the removed node is restored in the newly joined node, so that the newly joined node can establish communication connection between nodes based on the restored communication graph, thereby re-establishing the communication connection between the remaining nodes and the newly joined node.

[0134] S620, the large model training cluster is restored to perform large model training based on the reestablished communication connection.

[0135] As described above, since the checkpoint file stores the status data of the large model training cluster, the large model training cluster that has reestablished the communication connection restores the training status from the checkpoint file to continue the large model training.

[0136] After experiments, for an exemplary large-model cluster training scenario, the method provided in this embodiment can shorten the time from the discovery of a faulty node to the recovery of cluster training from 4 hours to less than 3 minutes, greatly improving the efficiency of large-model training and reducing the waste of computing resources and labor costs caused by the mid-stage of large-model training tasks.

[0137] See also Fig. 9 , Fig. 9 FIG. 1 is a block diagram of a fault handling device based on large model cluster training according to an exemplary embodiment of the present application. Fig. 9 As shown, the device comprises:

[0138] The traffic collection module 910 is configured to periodically collect the communication traffic of each node in the large model training cluster;

[0139] The traffic calculation module 920 is configured to obtain the interruption period when a training interruption is detected in the large model training cluster, and calculate the total communication traffic corresponding to each traffic collection moment included in the interruption period according to the communication traffic of each node at each traffic collection moment;

[0140] The time determination module 930 is configured to determine the communication interruption time according to the total communication flow corresponding to each flow collection time in the interruption period;

[0141] The node location module 940 is configured to determine the faulty node according to the communication traffic of each node at the time of communication interruption.

[0142] In another exemplary embodiment, the time determination module 930 includes:

[0143] The sequence generation unit is configured to arrange the total communication flows corresponding to the respective flow collection moments in the interruption period in chronological order to obtain a total communication flow sequence;

[0144] The difference calculation unit is configured to calculate the difference values ​​corresponding to each communication total flow in the communication total flow sequence, and determine the flow collection time to which the communication total flow corresponding to the minimum difference value belongs as the communication interruption time.

[0145] In another exemplary embodiment, the difference calculation unit is further configured to:

[0146] Calculate the forward difference value for each communication total flow in the communication total flow sequence respectively, and construct the forward difference sequence corresponding to the communication total flow sequence based on the forward difference value corresponding to each communication total flow;

[0147] The smallest forward difference value in the forward difference sequence is traversed to determine the traffic collection time corresponding to the smallest forward difference value as the communication interruption time.

[0148] In another exemplary embodiment, the node location module 940 is further configured to:

[0149] According to the communication flow of each node at the time of communication interruption, determine the node with the lowest communication flow at the time of communication interruption;

[0150] The node with the lowest communication flow is determined as the faulty node.

[0151] In another exemplary embodiment, the apparatus further comprises:

[0152] A communication reconstruction module is configured to reestablish the communication connection between the remaining nodes in the large model training cluster and the newly added node when a newly added node is detected after the failed node is removed from the large model training cluster;

[0153] The training recovery module is configured to resume large model training based on the large model training cluster that has reestablished communication connection.

[0154] In another exemplary embodiment, the communication reconstruction module includes:

[0155] A process restart unit, configured to restart the networking subprocesses respectively deployed in the remaining nodes and the newly added nodes;

[0156] The networking unit is configured to perform inter-node networking based on the networking sub-process to re-establish the communication connection between the remaining nodes and the newly added node.

[0157] In another exemplary embodiment, the networking unit is further configured to:

[0158] Obtain host group information from the shared storage service. The host group information is used to record the host information corresponding to the nodes currently contained in the large model training cluster;

[0159] The host information of the local node is saved to the shared storage service client to update the host group information in the shared storage service client;

[0160] The host group information in the shared storage service client is continuously queried. When the number of host information contained in the queried host group information reaches a preset value and the number does not change within a preset time, the local node is network configured to establish a communication connection between the nodes based on the network configuration.

[0161] In another exemplary embodiment, the networking unit is further configured to:

[0162] Determine the ranking information corresponding to the local node through the host group information;

[0163] The communication connection between the nodes is performed based on the determined ranking information of the local nodes.

[0164] In another exemplary embodiment, the communication reconstruction module further includes:

[0165] A node role determination unit, configured to determine a node role corresponding to a failed node, the node role including a master node or a slave node;

[0166] The connection establishing unit is configured to re-establish the communication connection between the remaining nodes and the newly added node based on the node roles.

[0167] In another exemplary embodiment, the node role is a master node; the connection establishing unit is further configured to:

[0168] Re-determine the master node from the remaining nodes and the newly added nodes, and use the other nodes except the re-determined master node as slave nodes;

[0169] A communication connection is established between the re-determined master node and slave node.

[0170] In another exemplary embodiment, the connection establishing unit is further configured to:

[0171] The newly determined master node broadcasts a master node update message, wherein the master node update message contains the node address of the newly determined master node;

[0172] The communication graph is re-established based on the node address of the re-determined master node, so as to establish communication connections between the nodes based on the communication graph.

[0173] In another exemplary embodiment, the node role is a slave node; the connection establishing unit is further configured to:

[0174] The communication connections between the remaining nodes are maintained, and the communication graph of the removed nodes is restored in the newly added node, so that the newly added node can perform communication connections between nodes based on the restored communication graph.

[0175] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the fault handling device based on large model cluster training provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0176] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs. When the one or more computer programs are executed by the one or more processors, the electronic device implements the fault handling method based on large model cluster training provided in the above-mentioned embodiments.

[0177] Fig.10 The structure diagram of the computer system of the electronic device suitable for implementing the embodiment of the present application is shown. It should be noted that the electronic device can be Figure 1 The alarm device 120 in the implementation environment shown may also be a terminal or server in other implementation environments, which is not limited here. Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0178] like Fig.10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage part 1008 to a random access memory (RAM) 1003, such as executing the method described in the above embodiment. Various computer programs and data required for system operation are also stored in RAM 1003. CPU 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0179] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0180] In particular, according to an embodiment of the present application, the process described above with reference to the flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0181] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A more specific example of a computer-readable storage medium may include, but is not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer program contained in the computer-readable medium may be transmitted with any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0182] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0184] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, implements the fault handling method based on large model cluster training as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0185] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the fault handling method based on large model cluster training provided in each of the above embodiments.

[0186] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

[0187] It can be understood that in the specific implementation of the present application, data related to communication traffic, host group information, ranking information, etc. are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A fault handling method based on large model cluster training, characterized in that: include: Periodically collect the communication traffic of each node in the large model training cluster; When a training interruption is detected in the large model training cluster, the interruption period is obtained, and according to the communication traffic of each node at each traffic collection time, the total communication traffic corresponding to each traffic collection time included in the interruption period is calculated; Arrange the total communication flows corresponding to the respective flow collection moments in the interruption period in chronological order to obtain a total communication flow sequence, calculate the differential values ​​corresponding to the respective total communication flows in the total communication flow sequence, and determine the flow collection moment to which the total communication flow corresponding to the minimum differential value belongs as the communication interruption moment; According to the communication flow of each node at the communication interruption time, the node with the lowest communication flow at the communication interruption time is determined, and the determined node with the lowest communication flow is used as the fault node.

2. The method according to claim 1, characterized in that Calculating the difference values ​​corresponding to the respective total communication flows in the total communication flow sequence, and determining the flow collection time to which the total communication flow corresponding to the minimum difference value belongs as the communication interruption time, including: Calculating forward differential values ​​for each communication total flow in the communication total flow sequence respectively, and constructing a forward differential sequence corresponding to the communication total flow sequence based on the forward differential values ​​corresponding to each communication total flow; The smallest forward difference value in the forward difference sequence is traversed to determine the flow collection time corresponding to the smallest forward difference value as the communication interruption time.

3. The method according to claim 1, characterized in that: The method further comprises: After the faulty node is removed from the large model training cluster, when a newly added node is detected, reestablishing a communication connection between the remaining nodes in the large model training cluster and the newly added node; The large model training cluster is restored based on the reestablished communication connection to perform large model training.

4. The method according to claim 3, characterized in that When a newly added node is detected, re-establishing a communication connection between the remaining nodes in the large model training cluster and the newly added node includes: Restarting the networking subprocesses respectively deployed in the remaining nodes and the newly added node; Inter-node networking is performed based on the networking sub-process to re-establish the communication connection between the remaining nodes and the newly added node.

5. The method according to claim 4, characterized in that Performing inter-node networking based on the networking sub-process to re-establish a communication connection between the remaining nodes and the newly added node includes: Obtain host group information from the shared storage service end, where the host group information is used to record host information corresponding to the nodes currently contained in the large model training cluster; The host information of the local node is saved to the shared storage service client to update the host group information in the shared storage service client; The host group information in the shared storage service client is continuously queried. When the number of host information contained in the queried host group information reaches a preset number value and the number does not change within a preset time period, a networking configuration is performed on the local node to establish a communication connection between the nodes based on the networking configuration.

6. The method according to claim 5, characterized in that Performing networking configuration on local nodes and establishing communication connections between nodes based on the networking configuration includes: Determine the ranking information corresponding to the local node through the host group information; The communication connection between the nodes is performed based on the determined ranking information of the local nodes.

7. The method according to claim 4, characterized in that When a newly added node is detected, re-establishing a communication connection between the remaining nodes in the large model training cluster and the newly added node includes: Determine a node role corresponding to the faulty node, where the node role includes a master node or a slave node; The communication connection between the remaining nodes and the newly added node is re-established based on the node roles.

8. The method according to claim 7, characterized in that The node role is a master node; Re-establishing a communication connection between the remaining nodes and the newly added node based on the node roles includes: Re-determine the master node from the remaining nodes and the newly added nodes, and use the other nodes except the re-determined master node as slave nodes; A communication connection is established between the re-determined master node and slave node.

9. The method according to claim 8, characterized in that Establishing a communication connection between the newly determined master node and slave node, including: The newly determined master node broadcasts a master node update message, wherein the master node update message contains a node address of the newly determined master node; A communication graph is re-established based on the re-determined node address of the master node, so as to establish communication connections between nodes based on the communication graph.

10. The method according to claim 7, characterized in that The node role is a slave node; Re-establishing a communication connection between the remaining nodes and the newly added node based on the node roles includes: The communication connections existing between the remaining nodes are maintained, and the communication graph of the removed nodes is restored in the newly added node, so that the newly added node performs communication connections between nodes based on the restored communication graph.

11. A fault handling device based on large model cluster training, characterized in that: include: A traffic collection module, configured to periodically collect the communication traffic of each node in the large model training cluster; The traffic calculation module is configured to obtain the interruption period when a training interruption is detected in the large model training cluster, and calculate the total communication traffic corresponding to each traffic collection moment included in the interruption period according to the communication traffic of each node at each traffic collection moment; The time determination module is configured to arrange the total communication flows corresponding to the respective flow collection moments in the interruption period in chronological order to obtain a total communication flow sequence, calculate the difference values ​​corresponding to the respective total communication flows in the total communication flow sequence, and determine the flow collection moment to which the total communication flow corresponding to the minimum difference value belongs as the communication interruption moment; The node positioning module is configured to determine the node with the lowest communication flow at the time of communication interruption according to the communication flow of each node at the time of communication interruption, and use the determined node with the lowest communication flow as the fault node.

12. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of an electronic device, the method according to any one of claims 1 to 10 is implemented.

Images