A fault processing method and device, program product, and storage medium

Abstract

The application provides a fault processing method and device, a program product and a storage medium. The method comprises the following steps: receiving fault information sent by a fault server node corresponding to a first task, and obtaining a plurality of logs corresponding to the first task from a centralized storage area; judging a fault type of the fault server node according to the fault information and the plurality of logs, wherein the fault type comprises one of a hardware fault type, a network exception type and a program exception type; determining a fault processing strategy corresponding to the fault type, and executing the fault processing strategy, wherein the fault processing strategy is used for locating a fault and solving the fault.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a fault handling method and device, program product, and storage medium. Background Technology

[0002] In current large-scale model training, the training clusters for large models typically employ thousands or even tens of thousands of CPUs. When using multiple nodes to complete large-scale model training, individual chip failures, one or more network link failures, or network degradation issues such as reduced bandwidth and increased latency may occur even if the network link is normal. Other nodes can only wait and eventually time out. Currently, it is not possible to quickly locate the fault and restart the training task. Furthermore, the recovery from large-scale model training job faults requires steps such as re-establishing cluster connections, recompiling operators, and loading checkpoints, which waste time. Therefore, the occurrence of faults in large-scale model training will greatly waste the computing resources of the entire cluster. In other existing technologies, the detection process requires manual execution and complex operations, which also adds great difficulty to large-scale model training. Therefore, how to quickly and accurately locate network faults has become an urgent problem to be solved. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide fault handling methods and devices, program products, and storage media, which can quickly locate and resolve faults.

[0004] The fault handling method provided in this application includes: Receive fault information sent by the faulty server node corresponding to the first task, and retrieve multiple logs corresponding to the first task from the centralized storage area; The fault type of the faulty server node is determined based on the fault information and multiple logs. The fault type includes one of the following: hardware fault type, network abnormal type, and program abnormal type. Determine the fault handling strategy corresponding to the fault type and execute the fault handling strategy. The fault handling strategy is used to locate and resolve the fault.

[0005] The fault handling device provided in this application includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the above-described fault handling method.

[0006] This application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the above-mentioned fault handling method.

[0007] The computer-readable storage medium provided in this application is used to store a computer program that causes a computer to perform the above-described fault handling method.

[0008] In the technical solution of this application, fault information sent by a faulty server node corresponding to a first task is received, and multiple logs corresponding to the first task are obtained from a centralized storage area. Based on the fault information and the multiple logs, the fault type of the faulty server node is determined, including one of hardware fault type, network anomaly type, and program anomaly type. A fault handling strategy corresponding to the fault type is determined and executed. The fault handling strategy is used to locate and resolve the fault. Thus, by determining the fault type through received fault information and actively obtained corresponding logs, and determining a fault resolution strategy based on the determined fault type, automated and accurate location and handling of network faults are achieved, improving the efficiency of completing the first task. Attached Figure Description

[0009] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0010] Figure 1 This is a schematic diagram of a single-track network provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of the mechanism for creating training tasks in its embodiments; Figure 3 This is a schematic diagram of the fault recovery mechanism provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the fault handling method provided in the embodiments of this application. Figure 1 ; Figure 5 This is a flowchart illustrating the fault handling method provided in the embodiments of this application. Figure 2 ; Figure 6 This is a schematic diagram of the network inspection module detection process provided in the embodiments of this application; Figure 7 This is a schematic diagram of a 3D parallel cluster example provided in the embodiments of this application; Figure 8 This is a schematic diagram of tensor parallel group simulated communication provided in an embodiment of this application; Figure 9 This is a schematic diagram of data parallel group simulated communication provided in an embodiment of this application; Figure 10 This is a schematic diagram of pipeline parallel group simulated communication provided in an embodiment of this application; Figure 11 This is a flowchart illustrating the binary search process provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structural composition of the fault handling device provided in the embodiments of this application; Figure 13 This is a schematic structural diagram of a fault handling device provided in an embodiment of this application; Figure 14 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0012] In the following description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. It should also be noted that the terms "first," "second," and "third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. The term "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. It should also be understood that the "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of an association relationship. For example, A instructing B can mean that A directly instructs B, for example, B can obtain information through A; it can also mean that A indirectly instructs B, for example, A instructs C, and B can obtain information through C; it can also mean that there is an association between A and B. It should also be understood that the term "correspondence" mentioned in the embodiments of this application can mean that there is a direct or indirect correspondence between the two, or that there is an association between the two, or that there is an instruction and being instructed, configuration and being configured, etc.

[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0014] One approach typically employs general large-model training and fault location methods. Large models generally refer to deep learning models using the transformer architecture. These models have a massive number of parameters, reaching tens or even hundreds of billions, occupying terabytes of storage space. Currently, the memory of a single node's accelerator card is typically in the gigabyte range. Therefore, 3D parallelism is required, employing strategies such as tensor parallelism, data parallelism, and pipelined parallelism to distribute model parameters across multiple AI devices for training. Training tasks are usually managed in a cloud-native manner. Containerizing training tasks allows for more flexible and elastic horizontal scaling and automated rescheduling on the cluster. Furthermore, large models require massive amounts of training data (terabytes) and enormous AI computing power and network resources. Training clusters for large models typically use thousands or even tens of thousands of GPUs. The aforementioned 3D parallelism techniques generally require data exchange and computation between or within groups of chips. Currently, data transmission within the machine is generally carried out through high-speed interconnection devices, while high-speed data transmission between machines is carried out through Remote Direct Memory Access (RDMA) networks, such as wireless bandwidth technology (infiniband, IB) and RoCEv2 protocols. Figure 1 This is a schematic diagram of a single-track network provided in an embodiment of this application, which is a typical network topology diagram in this scheme, such as... Figure 1 As shown, a single machine typically contains eight AI acceleration chips, namely g1-g8 in the diagram. These eight chips are generally connected within the machine via a high-speed interconnect module, and interconnected between machines via two layers of network devices: leaf and span switches. Figure 1 The architecture includes leaf1, leaf2, and leaf3, and span1 and span2. Leaf and span switches have multiple links for high availability and load balancing; that is, each leaf switch can connect to multiple SPAN switches. However, the AI ​​chip only has a single link to each leaf switch, making it prone to single points of failure and subsequent training job failures. Typically, a communication group in aggregated communication involves multiple nodes and multiple chips. Under existing aggregated communication architectures, the following three problems are common: individual chip failure; failure of one or more network links; and network degradation issues such as reduced bandwidth and increased latency despite normal network links.

[0015] The above three problems will ultimately lead to training task failure. For example, consider 10 AI server nodes, each with 8 AI chips. During training, an all-reduce ensemble communication operation is required. After initialization, these 80 chips form a loop. Then, each chip's memory data is fragmented and sent unidirectionally to its downstream nodes. This process is repeated until all fragments are sent, distributing the data across all chips. If a card fails, a link fails (without load balancing), or bandwidth deteriorates (slow transmission leads to timeouts and failures), other nodes can only wait and eventually time out. It can be seen that current ensemble communication libraries lack strong overall control and low-level hardware awareness, thus failing to quickly locate faults and restart training tasks. For these problems, ensemble communication currently cannot directly pinpoint which chip or network segment failed. Furthermore, recovering from training job failures for large models requires re-establishing cluster connections, recompiling operators, and loading checkpoints, typically requiring waiting for timeouts, cluster re-initialization, and checkpoint loading to restart training, often taking 30 minutes or more. Therefore, a failure would result in a significant waste of the entire cluster's computing resources.

[0016] Figure 2 This is a schematic diagram of the mechanism for creating training tasks provided in an embodiment of this application, such as... Figure 2 As shown, the training task orchestrator creates training tasks and sends them to the scheduler. The scheduler assigns a POD to each AI server. Each AI server includes a device plugin. Figure 3 This is a schematic diagram of the fault recovery mechanism provided in the embodiments of this application, such as... Figure 3 As shown, when the AI ​​server's device components detect a chip failure or an abnormal POD, it reports the anomaly to the scheduler. The scheduler then synchronously reports the fault information to the training task orchestrator, i.e., synchronizes the training task status. The scheduler determines whether it is a hardware failure. If it is a hardware failure, it isolates the faulty hardware and performs rescheduling. If it is not a hardware failure, it checks whether the maximum number of attempts has been exceeded. If the maximum number of attempts has been exceeded, manual analysis and troubleshooting are performed. Even after the cause of the failure is found through manual analysis and troubleshooting, rescheduling is still required. If the maximum number of attempts has not been reached, rescheduling is performed directly. Rescheduling means deleting all PODs of the training task and recreating the PODs. The recreated PODs are then sent to the corresponding AI server. It can be understood that the maximum number of attempts here refers to the maximum number of times the failure has occurred.

[0017] One approach employs a device plugin (POD) network fault localization method. This typically leverages cloud-native scheduling capabilities to automatically launch tasks across the entire cluster. When a device plugin deployed on each AI server node detects a chip or other fault, it notifies the scheduler for rescheduling. Rescheduling generally involves deleting all PODs and then recreating them. This is because the surviving nodes may be out of sync after a fault, potentially lacking data from the faulty node. To achieve mathematical consistency, new PODs must be created, and the process must be restored to a consistent state from the latest checkpoint. This node device monitoring plus layered scheduler intervention strategy addresses the first problem, but it cannot handle network link failures or degradation. For network issues, P2P (point-to-point testing) and aggregated communication testing programs are also provided. These detection scripts require manual execution and complex operations such as providing machine information, configuring passwordless login, and configuring parameters. They often require multiple rounds of elimination testing to accurately pinpoint the network fault. This approach requires experienced operations personnel and time-consuming manual operations, significantly increasing the difficulty of training large models and wasting valuable computing resources.

[0018] In one approach, a bisection method is used for network fault location. As a general method for network fault location, the bisection method has certain advantages, such as the ability to quickly locate fault areas. However, since not all nodes communicate with each other during the training of a large model, performing bisection testing without considering the actual network topology may not be suitable for the characteristics of the training task. For example, the network fault area located by the bisection method may not affect the normal training task, because only certain groups of nodes communicate during the training task. Moreover, the bisection method uses a simple random or ordered approach, which may miss the network topology that actually needs to be tested.

[0019] In other words, these technical solutions typically cannot accurately pinpoint the nodes affected by network faults and network degradation issues, requiring experienced technicians to locate and resolve them. The aforementioned technical solutions suffer from problems such as the inability to automate network location and the need for manual intervention. Fault location can only be determined through localized device-side fault reporting, and the reported fault alarms may or may not affect the actual training task, making it difficult to assess their correlation. Even with switch monitoring technology, factors such as multi-path load balancing prevent precise correlation with training operations. Furthermore, network group testing can only be performed randomly or in a simple order. Therefore, how to achieve the most accurate location of faulty nodes and improve location efficiency becomes a crucial issue. To address this, the following technical solutions, as described in this application, are proposed.

[0020] Figure 4This is a flowchart illustrating the fault handling method provided in the embodiments of this application. Figure 1 ,like Figure 4 As shown, the fault handling method includes the following steps: Step 401: Receive the fault information sent by the faulty server node corresponding to the first task, and retrieve multiple logs corresponding to the first task from the centralized storage area.

[0021] In some implementations, the first task is a large model training task, which corresponds to multiple server nodes. When a corresponding server node fails, the server node that fails is the faulty server node corresponding to the first task.

[0022] In some implementations, when a server node fails, fault information sent by the failed server node is received. For example, the fault information may include information such as job execution failure or code failure.

[0023] In some implementations, when there are multiple faulty server nodes, fault node information sent by multiple faulty server nodes corresponding to the first task will be received.

[0024] It should be noted that in this application, one server corresponds to one node, which can be called a server, a server node, or a node. When a failure occurs, it can be called a faulty server, a faulty server node, or a faulty node.

[0025] In some implementations, the fault information includes fault sender information. For example, the fault sender is a fault server or a first component within the fault server. For example, the first component is a device plugin, but the specific first component can be determined according to the actual situation, and this application does not impose any specific limitations on it.

[0026] In some implementations, logs for the first task are generated during its completion. These logs are centrally stored in real time, meaning they are housed in a centralized storage area. Therefore, to locate a fault, it is necessary to retrieve multiple logs corresponding to the first task from the centralized storage area, analyze the fault based on the logs and fault information, and thus pinpoint the fault. For example, the logs are POD logs. It is understood that this area stores logs for all tasks; to determine a fault in the first task, only the logs corresponding to that task need to be retrieved.

[0027] In some implementations, the multiple logs corresponding to the first task are all the logs corresponding to the first task.

[0028] In some implementations, during the completion of the first task, the first task is divided into multiple subtasks. Therefore, the logs corresponding to the first task include multiple logs corresponding to each subtask. When storing multiple logs, each log is sorted according to the number of the first task and the numbers of the multiple subtasks to facilitate the retrieval of logs for each task in the future. For example, the first task is numbered as Task 1, and the multiple subtasks are numbered as Task 1-1, Task 1-2, Task 1-3, etc. The specific numbering method is not specifically limited in this application.

[0029] Specifically, the logs of each POD for the training task are collected in real time to a centralized storage, such as HDFS or S3 object storage, and sorted by training task and subtask number.

[0030] Here, one server node corresponds to multiple logs, and the first task corresponds to multiple server nodes. Each server node will generate multiple corresponding logs. Therefore, in order to determine the fault type, it is necessary to obtain multiple logs generated by multiple server nodes corresponding to the first task. It can be understood that the multiple servers here include faulty servers as well as non-faulty servers.

[0031] In some implementations, the fault handling method is applied to a fault handling system, which is a system for detecting, locating, and handling network faults. The fault handling system receives fault information sent by the fault server corresponding to the first task and retrieves multiple logs corresponding to the first task from a centralized storage area.

[0032] In some implementations, the fault handling system includes a task orchestrator and a scheduler. The task orchestrator creates a first task, specifically by splitting the first task into multiple subtasks and sending these subtasks to the scheduler. The scheduler then assigns the subtasks to multiple server nodes and allocates a Point of Demand (POD) to each server node. When a server node fails, the scheduler receives fault information from the failed server node corresponding to the first task. Here, the failed server node sends the fault information to the scheduler, or a device component of the failed server node sends the fault information to the scheduler.

[0033] For example, when the first task is a large model training task, the task orchestrator splits the large model training task into multiple training sub-tasks and sends the multiple training sub-tasks to the scheduler. The scheduler assigns the multiple training sub-tasks to multiple AI server nodes and assigns PODs to multiple AI servers. When an AI server fails, the scheduler receives the failure information sent by these failed AI servers, including the AI ​​server's device components discovering a chip failure and reporting it to the scheduler, or the AI ​​server discovering a POD abnormality and reporting it to the scheduler.

[0034] It is understandable that when multiple servers fail, the fault information reported by the multiple servers may not be the same.

[0035] In some implementations, the fault handling system further includes a rule engine log analysis module, which retrieves multiple logs corresponding to the first task from a centralized storage area. For example, when the first task is a large model training task, the rule engine log analysis module retrieves all POD logs corresponding to the large model training task from the centralized storage area.

[0036] Step 402: Determine the fault type of the faulty server node based on the fault information and multiple logs. The fault type includes one of the following: hardware fault type, network abnormal type, and program abnormal type.

[0037] In some implementations, when there are multiple faulty server nodes, the fault type of each faulty server node is determined based on the fault information reported by each faulty server node and multiple logs.

[0038] In some implementations, the fault type of the faulty server node is determined by the sender of the fault information and multiple logs.

[0039] In some implementations, determining the fault type of a faulty server node based on fault information and multiple logs includes: if the fault sender information is a first component of the faulty server node, then the fault type of the faulty server node is determined to be a hardware fault; if the fault sender information is not a first component of the faulty server node, then multiple logs are matched with a first rule set to obtain a matching result, where the rules in the first rule set are used to determine the fault type; and the fault type of the faulty server node is determined based on the matching result. Here, in the first task, if the job container fails or the first component reports fault information, the fault type is determined based on the fault information and multiple logs.

[0040] In some implementations, the first component may be a device plugin in the server node.

[0041] In some implementations, when a device component of a faulty server node sends fault information, the fault type of the faulty server node is determined to be a hardware fault. When the sender of the fault information is not a device component of the faulty server node, the fault type of the faulty server node is determined by analyzing the log information.

[0042] In some implementations, when the scheduler receives fault information from the faulty server node, the scheduler determines the fault type of the faulty server node based on the sender of the fault information.

[0043] In some implementations, analyzing log information to determine the fault type of a faulty server node includes: matching multiple logs with a first set of rules to obtain a matching result, wherein the rules in the first set of rules are used to determine the fault type; and determining the fault type of the faulty server node based on the matching result.

[0044] In some implementations, the fault type of the faulty server node is determined based on fault information and all logs related to the first task.

[0045] Specifically, if the chip or node failure is detected by the device plugin, it indicates a hardware failure; otherwise, it is a POD anomaly. In this case, the POD logs are analyzed, and the cause of the POD failure is matched using a rule engine. If a matching rule is found, a network anomaly is output. For example, if it is a potential network anomaly such as a timeout, a network anomaly is output; otherwise, a training program anomaly is output.

[0046] In some implementations, matching multiple logs with a first rule set includes: reading the content information of multiple logs, whereby the content information includes one or more of the following: log time information, log level information, and log content information; structuring the content information of the multiple logs and aggregating it into a first context; and matching the first context with the first rule set. Here, in order to match the logs with the first rule set, it is necessary to structure the content information of each log and aggregate the structuring log information into a context to facilitate matching with the first rule set.

[0047] In some implementations, determining the fault type of the faulty server node based on the matching results includes: if the first context matches at least one rule in the first rule set, the fault type is determined to be a network anomaly; if the first context does not match any rule in the first rule set, the fault type is determined to be a program anomaly. Here, the rules in the first rule set are all rules used to determine the fault type. If the first context matches any rule, the fault type is determined to be a network anomaly; if no rule matches, the fault type is determined to be a program anomaly.

[0048] In some implementations, the scheduler determines the fault type based on the received fault information. If the fault is a chip fault or server node fault detected by the device component, and the corresponding fault information is uploaded to the scheduler, the scheduler determines it to be a hardware fault. Otherwise, it is considered a POD anomaly. In this case, the rule engine log analysis module analyzes all POD logs corresponding to the current task to determine whether the fault type is a network anomaly or a program anomaly.

[0049] In some implementations, the rule engine log analysis module determines the cause of a POD failure by matching the first rule set with the POD logs. If a match is successful, it is considered a network anomaly; otherwise, it is considered a program anomaly. Specifically, when the first task fails, the rule engine log analysis module reads each POD log corresponding to the first task, performs structured transformation based on fields such as log time, log level, and log content, aggregates the structured log information into a first context, and applies the first context of the processed structured log data to each rule in the first rule set. If any rule in the first rule set is matched, it is considered a network anomaly.

[0050] In some implementations, the rules in the first rule set can be preset, and the rule engine log analysis module reads the preset rules and matches them with the POD logs.

[0051] For example, the pseudocode representation of the rules in the first rule set is as follows: #Rule 1: Filter log entries with the ERROR level that contain the network.*timeout content using regular expressions. re_filter(context.log_content,'network.*timeout').filter(context.log_level,'ERROR').count()>0 #Rule 2, filter matches where reason=[task timeout] and the log level is ERROR. filter(context.log_content,'reason=[task timeout]'').filter(context.log_level,'ERROR').count()>0 #Rule 3, filter matches where the log level is ERROR and the error code is 107020. filter(context.log_content,'error code is 107020').filter(context.log_level,'ERROR').count()>0 Step 403: Determine the fault handling strategy corresponding to the fault type and execute the fault handling strategy. The fault handling strategy is used to locate and resolve the fault.

[0052] Here, for each type of fault, a corresponding fault handling strategy needs to be executed to locate and resolve the fault.

[0053] In some implementations, determining the fault handling strategy corresponding to the fault type includes: when the fault type is a hardware fault, executing a first handling strategy, which is to isolate the faulty node or reschedule the job container of the first task; when the fault type is a program exception, executing a second handling strategy, which is to reschedule the job container of the first task; and when the fault type is a network exception, executing a third handling strategy, which is to isolate the faulty node or reschedule the job container of the first task after locating the fault.

[0054] In some implementations, when the fault type is a hardware fault, the scheduler executes a first processing strategy. Here, hardware fault types include chip-level network faults. Therefore, the scheduler isolates the faulty server node or the chip of the faulty server and then reschedules the task. Rescheduling generally involves deleting all job containers and creating all new job containers. That is, the first processing strategy is to reschedule the job containers of the first task to complete the first task. When the fault type is a program exception, the scheduler executes a second processing strategy. That is, the scheduler attempts to reschedule. Here, when the first task is a large model training task, the program exception here is a training program exception. For the third type of network exception, a third processing strategy is executed, which is to locate the fault, isolate the faulty node, or reschedule the job containers of the first task.

[0055] In some implementations, the fault handling system includes a network inspection module. When the rule engine log analysis module determines that the fault type is a network anomaly, a third processing strategy is executed. That is, after the network inspection module locates the network anomaly range, the scheduler isolates the faulty node or reschedules the job container of the first task.

[0056] In some implementations, a third processing strategy is executed, including: generating a network test task set based on the configuration file and scheduling information of the first task; dividing multiple server nodes into various communication group sets; issuing corresponding network test tasks to each communication group set to obtain test results for each communication group in each set, wherein the network test tasks are used to test the communication status information of each communication group; locating faulty nodes based on the test results of each communication group in each set, and isolating the faulty nodes or rescheduling the job container of the first task. Here, issuing corresponding network test tasks for each communication group constitutes the network test task set.

[0057] In some implementations, the configuration file is a YAML file, and the scheduling information for the first task is the last scheduling information. Specifically, when the fault type is a network anomaly, the network inspection module generates a set of network test tasks, also known as a network inspection task set, based on the YAML description file of the first task and the last scheduling information of the first task. Each network test task in this set is used to test the communication status of a communication group set, locate the scope of the network anomaly, and make further decisions based on the scope of the network anomaly. The scheduler isolates all server nodes and chips within that scope based on the location results, or reschedules them. For example, if there is a network fault or network degradation in a certain area, all server nodes and chips in that area are isolated and an alarm is issued, pending further manual location of the network fault.

[0058] It is understandable that locating the network anomaly here means locating the fault, and isolating the faulty node means isolating all server nodes within the range of the network anomaly.

[0059] In some implementations, the network inspection module divides multiple server nodes into various communication group sets; it issues corresponding network test tasks to each communication group set, obtaining test results for each communication group within each set. These network test tasks are used to test the communication status information of each communication group; and the faulty node is located based on the test results of each communication group within each set. Here, the content and logic of the network test tasks issued by the network inspection module to each communication group are different. For a given communication group, the network inspection module issues the task to each communication group within that group simultaneously.

[0060] In some implementations, the various communication group sets include tensor parallel group sets, data parallel group sets, and pipeline parallel group sets. These three communication groups are also understood to be referred to as 3D parallelism. Specifically, the network test task reads the 3D parallelism strategy of the first task and generates the same network topology as the first task.

[0061] Understandably, the network testing task differs from the first task. The first task involves complex operator operations and network communication, and generally has a longer timeout (30 minutes). In contrast, the network testing task runs a customized network checking program that performs multiple small-scale network communications with a shorter network timeout (1 minute). The program is then scaled proportionally based on the amount of 3D parallel data and the number of nodes. This allows for rapid detection of network problems and further localization.

[0062] In some implementations, when a network failure occurs, to locate the fault, it is necessary to obtain the network topology of the first task, simulate the actual completion of the first task, and determine multiple communication group sets corresponding to multiple server nodes based on the network topology. Then, network test tasks are distributed to each communication group set according to these multiple communication group sets. After receiving the corresponding network test task, each communication group set completes the corresponding network test, obtaining the test results for each communication group. The faulty node is located based on the test results of each communication group within each communication group set. Here, the network test tasks are used to test the communication status of each communication group, obtain communication status results, and determine the communication status within or between communication groups based on these results, thereby identifying the faulty node. In this case, the faulty node is located to determine whether the problem is with the accelerator card network within the communication group set or a larger-scale network problem.

[0063] In some implementations, accelerator cards within a single server node form a tensor parallel group, accelerator cards with the same number on different server nodes form a data parallel group, and multiple data parallel groups and tensor parallel groups form a pipelined parallel group. Here, accelerator cards can also be referred to as chips within the server; for example, AI chips in an AI server.

[0064] Specifically, the test task verification method for different communication parallel groups is as follows: Within a parallel group, the AI ​​chip communication library is used to perform ensemble communication such as all reduce and all gather to simulate the ensemble communication operations used in actual training jobs. Since a GPU may play multiple roles simultaneously under a 3D parallel strategy, and the groups overlap, to facilitate the localization of network problems, the test is divided into three stages, which are tested one by one to simulate network problems in real training scenarios. In addition, the communication methods and data volumes used for different 3D parallel strategies are also different. For example, data parallelism mainly uses all reduce communication, while tensor parallelism mainly uses all gather communication, and pipelined parallelism uses send / receive point-to-point communication. Therefore, the parallel group tests in the three stages need to adopt different ensemble communication methods and data volumes, and corresponding test programs can be customized based on these characteristics.

[0065] In some implementations, locating faulty nodes based on the test results of each communication group within each communication group set includes: if the test results indicate a fault in the corresponding communication group, performing a binary search on the faulty communication group to obtain the binary search result; and determining the faulty node based on the binary search result. The binary search process involves dividing the faulty communication group into two subgroups and issuing network test tasks to both subgroups to test the communication status information of each subgroup. Here, if any accelerator card in a communication group fails to communicate, all accelerator cards in that communication group will report errors. Therefore, it is necessary to further determine whether the problem is with the accelerator cards of individual nodes or a larger-scale network problem. In this case, a binary search method can be used to locate the fault within the communication group.

[0066] In some implementations, the network inspection module first issues a corresponding test task to each tensor parallel group in the tensor parallel group set. Based on the test results of the corresponding test task, it determines whether the tensor parallel group under test is faulty. If the tensor parallel group under test is faulty, a binary search check is performed on the faulty tensor parallel group. If there is no fault, the second stage of inspection is performed, that is, the data parallel group is inspected. A corresponding test task is sent to each data parallel group in the data parallel group set. Based on the test results of the corresponding test task, it determines whether the data parallel group under test is faulty. If the data parallel group under test is faulty, a binary search check is performed on the faulty data parallel group. If there is no fault, the third stage of inspection is performed, that is, the pipeline parallel group is inspected. A corresponding test task is sent to each pipeline parallel group in the pipeline parallel group set. Based on the test results of the corresponding test task, it determines whether the pipeline parallel group under test is faulty. If the pipeline parallel group under test is faulty, a binary search check is performed on the faulty pipeline parallel group. If there is no fault, the testing process ends.

[0067] In some implementations, the network inspection module sends corresponding test tasks to each parallel group and receives the test results. The faulty node is then identified based on the test results.

[0068] In some implementations, when a communication group under inspection is faulty, it is divided into two subgroups. Test tasks identical to those for the original communication group are then issued to both subgroups to test their communication status. Specifically, each communication group is first split in two, with the task settings similar to the original parent task but with half the number of members. The task is then issued again. If both subgroup network checks fail, it indicates the network fault may extend across two subgroups. Therefore, the parent faulty group is recorded, and the binary search is stopped. The parent faulty group is the communication group before splitting. The recorded information includes the tested communication group information and fault markers. If both subgroup network checks succeed, it indicates a larger-scale task involves a potentially faulty switch device. Therefore, the parent group is recorded, and the binary search is exited. If one subgroup network check succeeds and the other fails, it indicates the failed subgroup may contain a faulty switch device. Therefore, the binary search continues.

[0069] In some implementations, determining faulty nodes based on the results of a binary search includes: obtaining a list of faulty region nodes based on the results of the binary search, and locating the faulty node based on the list of faulty region nodes. Here, for subgroups where a fault occurs, the subgroup is recorded and marked as faulty; for subgroups that do not occur, the subgroup is also recorded but not marked as faulty, thus obtaining the list of faulty region nodes. It can be understood that the network inspection module determines faulty nodes based on the results of the binary search.

[0070] In some implementations, the network inspection module generates a network test task by reading the configuration information and physical node information of the first task. The pseudocode for this task is as follows: group_id: int = 1 group_nodes: List[str] =[node1] group_node_ranks: List[int]=[0,1,2,3] master_url: str =”http: / / master:2345” accelerator: str =”gpu” collective_type=”all_gather” check_round: int =3 exit_barrier_timeout: int=60 nproc_per_node: int =4 nodes: int = 15 max_retry_round: int =3 data_size=10240 In this context, "group_id" represents the parallel group to which the current server node belongs, "group_nodes" represents all node members in the current parallel group, "group_node_ranks" represents all accelerator card members in the current group, "master_url" represents the URL of the master node (which needs to report statistical information to the master node after completing the network check task), "collective_type" represents the type of collective communication, "check_round" represents the number of tests, "exit_barrier_timeout" represents the timeout for collective communication, "max_retry_round" represents the maximum number of retries after a failure in collective communication, and "data_size" represents the amount of communication data. Based on this task information, each communication group (also known as each parallel group) attempts to establish connections with other members in the same group after receiving the task information to perform the specified collective communication.

[0071] In some implementations, after the test is completed, each parallel group sends the test results to the network inspection module, which can also be called the network test manager or master node. The network inspection module then generates a final aggregate report after all reports have been collected. An example of the pseudocode representation of the report content of each communication group is as follows: group_id: int = 0 node_rank: int = 0 rank: int =1 init_group_time: float=900 communication_time: float=1029 total_time: float=1929 communication_bandwidth: float=9.95 err_msg: str ='' In this context, "group_id" represents the ID of the current parallel group being tested, "node_rank" represents the ID of the server node in the current parallel group being tested, "rank" represents the identifier of the accelerator card in the current parallel group being tested, "init_group_time" represents the initialization time, i.e., the connection establishment phase of the current parallel group, "communication_time" represents the communication time (understandably, the initialization time and communication time are consistent within the same parallel group), "total_time" is the sum of the initialization time and communication time, "communication_bandwidth" represents the bandwidth, and "err_msg" represents error messages. If there are no error messages, it will be empty; if there are error messages, they will be displayed here to determine if a node is faulty. For example, if it is a data parallel group group0{M0.GPU0-M5.GPU0}, when testing GPU0 of M1, group_id is 0, representing group0, node_rank is 1, representing M1, and rank is 0, representing GPU0 of M1.

[0072] It is understandable that tests are performed on the accelerator cards in each parallel group, and the test results of each accelerator card are included in the reports submitted by each parallel group.

[0073] In some implementations, once a faulty node is detected, the faulty area is reported back to the scheduler, which automatically isolates the faulty area and reports an alarm.

[0074] In some implementations, both the module that detects the fault and the module that analyzes the fault can report alarms. For example, the subject that reports alarms can be the network inspection module, the scheduler, and the rule engine log analysis module, etc. This application does not specifically limit the specific reporting subject.

[0075] In some implementations, the fault handling system includes an alarm module for receiving alarm information reported by other modules, and all alarm information is reported to the alarm module.

[0076] The technical solution of this application embodiment receives fault information sent by a faulty server node corresponding to a first task, and obtains multiple logs corresponding to the first task from a centralized storage area; determines the fault type of the faulty server node based on the fault information and the multiple logs, the fault type including one of hardware fault type, network anomaly type, and program anomaly type; determines a fault handling strategy corresponding to the fault type, and executes the fault handling strategy, which is used to locate and resolve the fault. Thus, by determining the fault type through received fault information and actively obtaining corresponding logs, and determining the fault solution based on the determined fault type, automated and accurate location and handling of network faults are achieved, improving the efficiency of completing the first task.

[0077] Based on the foregoing embodiments, the fault handling method provided by the embodiments of this application will be further described.

[0078] Figure 5 This is a schematic flowchart of the fault handling method provided in the embodiments of this application. Figure 2 ,like Figure 5 As shown, this application embodiment introduces a rule engine log analysis module and a network inspection module. The overall process of the fault handling method is as follows: When a device plugin detects a chip fault or the AI ​​server detects a POD anomaly, it reports the corresponding fault information to the scheduler. The scheduler synchronizes the fault information to the training task orchestrator, that is, the scheduler synchronizes the training task status to the training task orchestrator. When the scheduler receives the fault information, it determines the fault type. First, it determines whether it is a hardware fault. If it is a hardware fault, it isolates the faulty hardware and reschedules. Rescheduling means deleting all PODs of the training task and recreating the PODs. The recreated PODs are then sent to the AI ​​server corresponding to the training task to continue completing the current task. When it is not a hardware fault, the rule engine log analysis module needs to determine whether it is a network fault. If it is not a network fault, it attempts to reschedule. If it is a network fault, the network inspection module locates the fault range. After locating the fault, the scheduler attempts to reschedule.

[0079] Specifically, since training tasks typically employ cloud-native containerization solutions, the training task orchestrator generates jobs from the open-source container orchestration system (Kubenertes). During large model training, if a job container malfunctions or a device plugin reports a fault, the fault information is reported to the scheduler, which then reports it to the training task orchestrator. The training task orchestrator merely displays and synchronizes the status, i.e., it synchronizes the training task status. The scheduler determines the fault type. A job container malfunction is the aforementioned POD (Position Item) anomaly. The fault type determination method is as follows: If the chip or node failure is detected by the device plugin, it indicates a hardware failure (equivalent to the aforementioned hardware anomaly type). Otherwise, it is a POD anomaly. The POD log is analyzed by the rule engine log analysis module. This module uses the rule engine (equivalent to the first rule set mentioned above) to match the cause of the POD failure. If it is a potential network anomaly such as timeout, then the network anomaly (equivalent to the network anomaly type mentioned above) is output. Otherwise, it is a training program exception (equivalent to the aforementioned program exception type).

[0080] The handling strategies for the above three types of faults are as follows: For hardware failures, the faulty node or chip can be isolated and rescheduled. Hardware failures here include chip failures, i.e. network failures, which correspond to isolated chip rescheduling. If the training program malfunctions, a rescheduling attempt will be made. For network anomalies, the network inspection module generates a network test task based on the YAML description file of the current training task to locate the network anomaly. Locating the network anomaly here means locating the anomaly range. Further decisions are made based on the type of network anomaly. The specific decisions are as follows: If there is a network failure or network degradation in a certain area, all nodes and chips in this area are isolated and an alarm is issued, pending further manual location of the network failure.

[0081] It is understandable that after the training task orchestrator distributes training tasks, each AI server performs the relevant training tasks. If a failure occurs, the current status needs to be communicated to the user. Therefore, the training task orchestrator only synchronizes the status and displays the current task status to inform the user. It should also be understood that when a job container fails, the AI ​​server reports the failure. The reported failure information includes the reporting entity, the job process failure, and code failure, etc., which are not specifically limited in this application. Furthermore, it should be understood that when the scheduler analyzes the failure type, it judges based on the reporting entity. If the failure is not reported by a device plugin, it is considered a POD (Physical Device Adapter) failure, and then the log analysis module analyzes the specific cause of the failure. The execution process of the rule engine log analysis module and network inspection module is described in detail below.

[0082] I. Rule Engine Log Analysis Module Since network problems typically manifest as training job anomalies due to timeout errors, a crucial aspect of this application's embodiment is analyzing the causes of failures based on the training task's POD logs. This module utilizes a rule engine to analyze the logs. The specific processing method of this module is as follows: Each POD log from a training task is collected in real-time to a centralized storage (equivalent to the aforementioned centralized storage area), such as HDFS or S3 object storage, and sorted by training task and subtask number. Essentially, this stores logs for all training tasks; when analyzing a fault in a particular task, only the POD log corresponding to that task needs to be retrieved. Each training task and its corresponding subtask has a unique identifier.

[0083] When a training task fails, the logs of each POD corresponding to the training task are read, and the logs are structured according to fields such as log time, log level, and log content. Then, the structured log information is aggregated into a context, which is equivalent to the first context mentioned above.

[0084] This module reads pre-set rules and applies the context of the processed structured log data to each rule. If any of these rules is matched, it is considered a network anomaly. The pseudocode representation of the rules is as described above and will not be repeated here.

[0085] II. Network Inspection Module Figure 6 This is a schematic diagram of the network inspection module detection process provided in the embodiments of this application, such as... Figure 6 As shown, firstly, a network test task (equivalent to the aforementioned set of network test tasks) is generated using the YAML file of the training task and the previous scheduling information of that training task. The previous scheduling information is also the last scheduling information. The network test task is used to perform network checks. Specifically, firstly, a one-stage tensor parallel check is performed, that is, a check is performed on each tensor parallel group to determine if the corresponding tensor parallel group has a fault. If a fault is found, a binary search is performed on the faulty tensor parallel group to obtain the final list of faulty region nodes. If no fault is found, a two-stage data parallel check is performed, that is, a check is performed on each data parallel group to determine... If a fault is found in the corresponding data parallel group, a binary search is performed on the faulty data parallel group to obtain the final list of faulty region nodes. If no fault is found, a three-stage pipeline parallel check is performed, that is, a check is performed on each pipeline parallel group to determine if the corresponding pipeline parallel group is faulty. If a fault is found, a binary search is performed on the faulty pipeline parallel group to obtain the final list of faulty region nodes. If no fault is found, the process ends. Once the final list of faulty region nodes is obtained, the faulty region nodes can be identified. Based on the faulty region nodes, fault isolation or rescheduling is performed, and alarms are issued for relevant information of the faulty region.

[0086] It's understandable that the training task here can also be called a training job, or simply a job. The YAML file contains the configuration information for this training task, and the scheduling information includes the selected nodes (i.e., the AI ​​server), the machine IDs used, and how tasks are allocated. The network testing task, also known as a NetworkCheck job, reads the 3D parallel strategy of the training job and generates the same network topology. Here, "same network topology" means that the simulated network topology is identical to the real network topology. However, unlike the training task, which involves complex operator operations and network communication and typically has a longer timeout (e.g., 30 minutes), the network test runs a customized network checking program. It performs multiple small-scale network communications with shorter timeouts (e.g., 1 minute), and then scales the process proportionally based on the amount of 3D parallel data and the number of nodes. This allows for rapid detection of network problems and further localization. It should also be understood that if a fault is detected in the previous stage, subsequent parallel checks will not be performed.

[0087] The following section provides a detailed explanation of the fault detection process and the binary search process for 3D parallel networks.

[0088] (a) Fault detection in 3D parallel networks Depending on the model training framework used in the training task, the AI ​​chips within a typical AI server node form a tensor parallel group, which can also be called a model parallel group. Different nodes with the same chip number form a data parallel group. Multiple data parallel groups and tensor parallel groups form a pipelined parallel group, which can also be called a model parallel group. Here, the chip number is the identifier of the AI ​​chip. Figure 7 This is a schematic diagram of a 3D parallel cluster example provided in the embodiments of this application, such as... Figure 7 As shown, three types of ensemble communication groups are generated: tensor parallel group, data parallel group, and pipeline parallel group. The specific generation of the ensemble communication groups is as follows: Tensor parallelization groups: M0{GPU0-GPU7}, M1{GPU0-GPU7}, ..., M383{GPU0-GPU7}, Figure 7 There are a total of 384 parallel tensor groups; Data parallel groups: {M0.GPU0-M5.GPU0}, ..., {M0.GPU1-M5.GPU1}, ... Figure 7 There are a total of 512 parallel data groups; Parallel pipeline groups: {M0.GPU0-M5.GPU7}, {M6.GPU0-M11.GPU7}, ... Figure 7 There are a total of 64 parallel pipeline groups; Where M represents Machine, or server node, and M0{GPU0-GPU7} represents cards 0 to 7 of machine 0, that is, the chips in the server node.

[0089] Each communication parallel group uses an AI chip communication library to perform ensemble communication operations such as all reduce and all gather to simulate the ensemble communication operations used in actual training jobs. Here, the communication groups refer to the tensor parallel group, pipeline parallel group, and data parallel group mentioned earlier. Since a GPU may play multiple roles simultaneously under a 3D parallel strategy, and the groups overlap, to facilitate network problem localization, the test is divided into three stages, with each stage tested sequentially to simulate network problems in real training scenarios. Furthermore, the communication methods and data volumes used for different 3D parallel strategies also differ. For example, the data parallel group primarily uses all reduce communication, while the tensor parallel group primarily uses all gather communication, and pipeline parallelism uses send / receive point-to-point communication. Therefore, the testing of the three parallel groups requires different ensemble communication methods and data volumes. Thus, the test program for each communication parallel group can be customized based on these characteristics, i.e., the network test task for each communication group can be determined.

[0090] The first step is the tensor parallel group check in the first phase. Figure 8 This is a schematic diagram of tensor parallel group simulated communication provided in an embodiment of this application, such as... Figure 8 As shown, the network detection master node distributes the test task to each node, that is, to each tensor parallel group, namely M0, M1, ..., Mn. Each node's GPU performs aggregated communication testing. After the test is completed, each node reports a report to the network detection master node. The network detection master node can also be called "NetworkCheckMaster," network test manager, or network detection entity, equivalent to the aforementioned network inspection module. The test task can also be called a "dispatch task," which is the network inspection program customized for the tensor parallel groups. It can be understood that the nodes in the first stage refer to the tensor parallel groups.

[0091] Then comes the second phase of data parallel group inspection. Figure 9 This is a schematic diagram of data parallel group simulated communication provided in an embodiment of this application, such as... Figure 9As shown, the network detection master node distributes the test task to each data parallel group. Each data parallel group performs aggregated communication tests internally. After the test is completed, each data parallel group reports a report to the network detection master node. The network detection master node can also be called "NetworkCheck Master", network test manager or network detection subject, which is equivalent to the aforementioned network check module. The test task can also be called "dishpatch task", which is the network check program customized for the data parallel group.

[0092] Finally, there is the third stage: pipeline parallel group inspection. Figure 10 This is a schematic diagram of pipeline parallel group simulated communication provided in an embodiment of this application, such as... Figure 10 As shown, the network inspection master node distributes test tasks to each pipeline parallel group. Boundary nodes within each pipeline parallel group send / receive point-to-point communication. After the test is completed, each pipeline parallel group reports a report to the network inspection master node. The network inspection master node can also be called the "NetworkCheck Master," network test manager, or network inspection entity, equivalent to the aforementioned network inspection module. The test task can also be called a "dishpatchtask," which is the network inspection program customized for the pipeline parallel groups. It can be understood that the pipeline parallel group here refers to... Figure 7 The second pipeline parallel group is M6-M11. The communication process is intra-group transmission, that is, GPU7 of M6 sends to GPU0 of M7, and so on to achieve point-to-point communication to complete the intra-group test.

[0093] Taking the tensor parallel group task as an example, the network detection master node generates a network detection task (NetworkCheck task) by reading the configuration information and physical node information of the training task (real training task). The pseudocode representation of this task is as shown above and will not be repeated here. This task is also the test task mentioned earlier, the simulated network test task, which is the test task information sent to each group. The configuration information and physical node information include: the number of machines, the number of parallel groups, the logical relationship information in the real topology, the topology generated by the scheduler, and the information of physical nodes, such as the information of AI servers and switches.

[0094] Based on the task information above, each node, upon receiving the task information, attempts to establish a connection with other members of the same group to conduct the specified aggregate communication. After the test, the result is sent to the network detection master node, which then generates the final aggregate report after all reports have been collected. The pseudocode representation of each node's report content is as described above and will not be repeated here. The presence of a fault is determined based on the "err_msg" in the report.

[0095] (II) Two-part inspection process: If any accelerator card in a communication group fails to communicate, all accelerator cards in that group will report errors. Therefore, it is necessary to further determine whether the problem is with the accelerator cards of individual nodes or a larger-scale network issue. In this case, a binary search method can be used to locate the fault within the communication group, which is a process of narrowing down the scope of the fault within the communication group. Figure 11 This is a schematic diagram of the binary search process provided in the embodiments of this application, such as... Figure 11 As shown, it includes the following steps: Step 1101: The task is divided into two sub-communication groups and the task is dispatched.

[0096] First, divide the group into two, set up tasks similar to the parent task but reduce the number of members by half, and then issue the tasks again.

[0097] Step 1102: Determine if both subgroups have failed.

[0098] If neither of the two subgroups fails, proceed to step 1103; otherwise, proceed to step 1104.

[0099] Step 1103: Determine whether one subgroup failed and another subgroup succeeded.

[0100] If one subgroup network check succeeds and the other fails, it indicates that the failed subgroup may contain a faulty switch device. Therefore, continue the binary search process, i.e., execute step 1101 for the failed subgroup and regroup it for testing. If one subgroup fails and the other succeeds, i.e., both subgroups succeed, then execute step 1104.

[0101] Step 1104: Record the abnormal parent group and exit.

[0102] Here, if both subgroup network checks fail, it indicates that the network fault may span two subgroups. Therefore, the parent faulty group is recorded and the binary search is stopped; that is, the abnormal parent group is recorded directly. If both subgroup network checks succeed, it indicates that a larger-scale task spans a potentially faulty switch device. Therefore, the parent group is recorded and the binary search is exited. It's understandable that both successful and unsuccessful subgroup or parent group checks need to be recorded. However, faulty nodes need to be marked as faulty, while fault-free subgroups also need to be recorded, but not marked as faulty.

[0103] Through the above methods, the network inspection module ultimately locates the faulty area and reports it to the scheduler. The scheduler automatically isolates the nodes in the faulty area, reports an alarm, and reschedules the process to continue the training task. Here, the alarm information is sent to the alarm module. The module that discovers or receives the fault information can send the alarm information, such as the network inspection module, the scheduler, and the faulty node, which constitutes a multi-layered, three-dimensional alarm system.

[0104] This application proposes a method and apparatus for large-scale AI chip network fault detection in large-scale model training scenarios. This application addresses the difficulties in existing methods for AI chip cluster network localization, such as the difficulty in locating faulty areas and the need for manual intervention. It automatically and accurately locates network fault areas and automatically isolates nodes affected by the faulty network, improving the efficiency of large-scale model training and increasing chip utilization. This application employs a comprehensive approach, combining hardware fault reporting, log parsing and rule engine fault classification, three-stage network fault simulation testing, and binary search to narrow down the testing scope, to accurately locate problems. This application abstracts the complex network environment of large-scale model training, simplifying it into three stages of aggregate communication with specific data volumes and methods. While maintaining consistency with the network logic of the training job, it quickly uses proportionally scaled small-scale tests for fault localization. After three-stage fault localization, if a communication group fault is found, further network localization is performed using binary search, further narrowing down the faulty nodes and providing valuable reference for subsequent manual network repair, thus shortening the localization time.

[0105] Compared with existing technologies, this application proposes an automated and efficient network fault detection method and apparatus for large model training, which greatly reduces the probability of manual intervention. It provides a comprehensive network fault detection method and enables automated fault isolation and rescheduling. This application can handle not only chip-level network problems but also faults at the leaf and span switch levels. This application performs binary search network localization only after a three-stage network inspection, solving the problem that traditional binary search methods use random or sequential binary search, which is inconsistent with the actual training network topology (actual training involves multiple communication groups, not full connectivity). This application performs binary search within a specific communication group, thus avoiding this problem at its source (a communication group requires full connectivity, so random or sequential binary search is possible).

[0106] This application proposes a more comprehensive method for all-round network fault location, and can analyze the fault area from the perspective of the training task, thereby making it easier for operation and maintenance personnel to narrow down the difficulty of locating the actual fault point based on the fault range. This application generates a network test task topology based on the network topology of the training task to simulate the training task, consistent with actual training. Once the fault group is located, a binary search fault detection method is used to further narrow down the range, making it more intelligent and effective than existing technologies. This application proposes an automated method for detecting network faults and automatically isolating faulty nodes and rescheduling them, greatly reducing the idle time of acceleration chips during network fault handling and effectively improving chip utilization.

[0107] Figure 12 This is a schematic diagram of the structural composition of the fault handling device provided in the embodiments of this application, as shown below. Figure 12 As shown, the fault handling device includes: The acquisition unit 1201 is used to receive fault information sent by the faulty server node corresponding to the first task and acquire multiple logs corresponding to the first task from the centralized storage area. The judgment unit 1202 is used to determine the fault type of the faulty server node based on the fault information and the plurality of logs. The fault type includes one of hardware fault type, network abnormal type and program abnormal type. The processing unit 1203 is used to determine the fault handling strategy corresponding to the fault type and execute the fault handling strategy, which is used to locate and resolve the fault.

[0108] In some implementations, the fault information includes fault sender information; the judgment unit 1202 is used to determine the fault type of the faulty server node as a hardware fault type if the fault sender information is a first component of the faulty server node; if the fault sender information is not a first component of the faulty server node, the multiple logs and a first rule set are matched to obtain a matching result, wherein the rules in the first rule set are used to determine the fault type; and the fault type of the faulty server node is determined according to the matching result.

[0109] In some implementations, the judgment unit 1202 is used to read the content information of the plurality of logs, the content information including one or more of the following: log time information, log level information, and log content information; aggregate the content information of the plurality of logs into a first context after performing a structured transformation; and match the first context with the first rule set.

[0110] In some implementations, the determination unit 1202 is configured to determine the fault type as a network anomaly if the first context successfully matches at least one rule in the first rule set; and to determine the fault type as a program anomaly if the first context does not successfully match all rules in the first rule set.

[0111] In some embodiments, the processing unit 1203 is configured to execute a first processing strategy when the fault type is a hardware fault, wherein the first processing strategy is to isolate the faulty node or reschedule the job container of the first task; execute a second processing strategy when the fault type is a program exception, wherein the second processing strategy is to reschedule the job container of the first task; and execute a third processing strategy when the fault type is a network exception, wherein the third processing strategy is to isolate the faulty node or reschedule the job container of the first task after locating the fault.

[0112] In some implementations, the processing unit 1203 is configured to generate a set of network test tasks based on the configuration file and scheduling information of the first task; divide the plurality of server nodes into multiple communication group sets; issue corresponding network test tasks to each communication group set to obtain test results for each communication group in each communication group set, wherein the network test tasks are used to test the communication status information of each communication group; locate faulty nodes based on the test results of each communication group in each communication group set, and isolate the faulty nodes or reschedule the job container of the first task.

[0113] In some embodiments, the processing unit 1203 is configured to perform a binary search on the faulty communication group if the test result shows that the corresponding communication group is faulty, and obtain the result of the binary search; and determine the faulty node based on the result of the binary search; wherein the binary search process is as follows: the faulty communication group is divided into two subgroups, and network test tasks are sent to the two subgroups to test the communication status information of each subgroup.

[0114] Those skilled in the art should understand that Figure 12 The functions of each unit in the fault handling device shown can be understood by referring to the relevant descriptions of the aforementioned methods. Figure 12 The functions of each unit in the fault handling device shown can be implemented by a program running on a processor or by specific logic circuits.

[0115] Figure 13 This is a schematic structural diagram of a fault handling device 1300 provided in an embodiment of this application. Figure 13The fault handling device 1300 shown includes a processor 1310, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0116] Optionally, such as Figure 13 As shown, the fault handling device 1300 may further include a memory 1320. The processor 1310 can retrieve and run computer programs from the memory 1320 to implement the methods described in this embodiment.

[0117] The memory 1320 can be a separate device independent of the processor 1310, or it can be integrated into the processor 1310.

[0118] Optionally, such as Figure 13 As shown, the fault handling device 1300 may also include a transceiver 1330, and the processor 1310 may control the transceiver 1330 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices.

[0119] The transceiver 1330 may include a transmitter and a receiver. The transceiver 1330 may further include an antenna, and the number of antennas may be one or more.

[0120] The fault handling device 1300 can implement the corresponding processes implemented by the fault handling apparatus in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0121] Figure 14 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 14 The chip 1400 shown includes a processor 1410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0122] Optionally, such as Figure 14 As shown, chip 1400 may further include memory 1420. Processor 1410 can retrieve and run computer programs from memory 1420 to implement the methods described in this embodiment.

[0123] The memory 1420 can be a separate device independent of the processor 1410, or it can be integrated into the processor 1410.

[0124] Optionally, the chip 1400 may also include an input interface 1430. The processor 1410 can control the input interface 1430 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0125] Optionally, the chip 1400 may also include an output interface 1440. The processor 1410 can control the output interface 1440 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0126] This chip can implement the corresponding processes implemented by the fault handling device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0127] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0128] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0129] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0130] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0131] This application also provides a computer program product, including a computer program.

[0132] When executed by a processor, the computer program implements the corresponding processes of the fault handling device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0133] This application also provides a computer-readable storage medium for storing computer programs.

[0134] The computer program causes the computer to execute the corresponding processes implemented by the fault handling device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0136] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0140] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Moreover, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application.

[0142] It should be understood that in the various method embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A fault handling method, characterized in that, The method includes: Receive fault information sent by the faulty server node corresponding to the first task, and retrieve multiple logs corresponding to the first task from the centralized storage area; The fault type of the faulty server node is determined based on the fault information and the multiple logs. The fault type includes one of the following: hardware fault type, network anomaly type, and program anomaly type. Determine the fault handling strategy corresponding to the fault type and execute the fault handling strategy, which is used to locate and resolve the fault.

2. The method according to claim 1, characterized in that, The fault information includes fault sender information; determining the fault type of the faulty server node based on the fault information and the multiple logs includes: If the fault sender information is the first component of the faulty server node, then the fault type of the faulty server node is determined to be a hardware fault type. If the fault sender information is not the first component of the faulty server node, then the multiple logs and the first rule set are matched to obtain a matching result, and the rules in the first rule set are used to determine the fault type; the fault type of the faulty server node is determined according to the matching result.

3. The method according to claim 2, characterized in that, The matching of the multiple logs with the first rule set includes: Read the content information of the plurality of logs, wherein the content information includes one or more of the following: log time information, log level information, and log content information; The content information of the multiple logs is structured and then aggregated into a first context; Match the first context with the first set of rules.

4. The method according to claim 3, characterized in that, Determining the fault type of the faulty server node based on the matching result includes: If the first context matches at least one rule in the first rule set, then the fault type is determined to be a network anomaly type. If the first context does not match any of the rules in the first rule set, then the fault type is determined to be a program exception type.

5. The method according to any one of claims 1 to 4, characterized in that, The fault handling strategy corresponding to the fault type includes: When the fault type is a hardware fault type, the first processing strategy is executed, which is to isolate the faulty node or reschedule the job container of the first task. When the fault type is a program exception type, the second processing strategy is executed, which is to reschedule the first task job container. When the fault type is a network anomaly, a third processing strategy is executed, which is to isolate the faulty node after locating the fault or reschedule the job container of the first task.

6. The method according to claim 5, characterized in that, The execution of the third processing strategy includes: A set of network test tasks is generated based on the configuration file and scheduling information of the first task. The multiple server nodes are divided into various communication group sets; A corresponding network test task is issued to each type of communication group set to obtain the test results of each communication group in each type of communication group set. The network test task is used to test the communication status information of each communication group. Based on the test results of each communication group in each communication group set, locate the faulty node and isolate the faulty node or reschedule the job container of the first task.

7. The method according to claim 6, characterized in that, The step of locating the faulty node based on the test results of each communication group in each communication group set includes: If the test results show that the corresponding communication group is faulty, then the faulty communication group will be subjected to a binary search to obtain the result of the binary search. The fault node is determined based on the results of the binary search. The binary check process involves dividing the faulty communication group into two subgroups and issuing network test tasks to the two subgroups to test the communication status information of each subgroup.

8. A fault handling device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 7.

Images