Fault handling method, device, electronic device and storage medium based on large model

Abstract

The present disclosure provides a fault handling method, device, electronic device, and storage medium based on a large model, which relates to technical fields such as large models, hardware fault handling, and machine learning. The specific implementation scheme is as follows: during the execution of a model training task, fault detection is performed on the task at the business layer; in response to detecting a task fault at the business layer, fault diagnosis is performed on the task to obtain a fault diagnosis result; and fault handling is performed based on the fault diagnosis result. The technology disclosed in the present disclosure can automatically handle faults in model training tasks and can effectively improve fault handling efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as large models, hardware fault processing, and machine learning, and more particularly to a fault processing method, device, electronic device, and storage medium based on a large model. Background Art

[0002] With the rapid development of artificial intelligence, training large models has become a very important task.

[0003] Large model training often leads to hardware failures, task stalls, and other issues, resulting in training interruptions. Existing techniques require algorithm engineers to manually monitor model training tasks and manually resume training when interrupted. Summary of the Invention

[0004] The present disclosure provides a fault handling method, device, electronic device and storage medium based on a large model.

[0005] According to one aspect of the present disclosure, a fault handling method based on a large model is provided, comprising:

[0006] During the execution of the model training task, fault detection is performed on the task at the business layer;

[0007] In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result;

[0008] Based on the fault diagnosis result, fault processing is performed.

[0009] According to another aspect of the present disclosure, a large model-based fault handling device is provided, comprising:

[0010] A fault detection module is used to detect faults in the model training task at the business layer during the execution of the task;

[0011] a fault diagnosis module, configured to, in response to detecting a task fault at the service layer, perform fault diagnosis on the task and obtain a fault diagnosis result;

[0012] The fault processing module is used to perform fault processing based on the fault diagnosis result.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned aspects and any possible implementation method when executed by a processor.

[0019] The technology disclosed in this disclosure can automatically handle faults during model training tasks and can effectively improve fault handling efficiency.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0027] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0028] Figure 7 is a block diagram of an electronic device for implementing the method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0031] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0032] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0033] In actual scenarios, during the model training process, algorithm engineers are required to manually monitor the training tasks. Usually, by the time a training failure is discovered manually, the model training task has been interrupted for several hours or even more than ten hours. The fault handling efficiency is extremely low, which seriously affects the model training efficiency.

[0034] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a fault handling method based on a large model, which may specifically include the following steps:

[0035] S101. During the execution of the model training task, perform fault detection on the task at the business layer;

[0036] The executor of the large model-based fault handling method of this embodiment can be a large model-based fault handling device, which can be set in the fault-tolerant controller at the central end of the model training system to promptly handle faults in the model training task.

[0037] In this embodiment, in order to promptly discover faults in the model training task, the faults are promptly detected at the business layer.

[0038] The fault detection of this embodiment can be achieved by configuring an agent component at the business layer.

[0039] S102, in response to detecting a task fault at the business layer, performing fault diagnosis on the task and obtaining a fault diagnosis result;

[0040] S103: Perform fault processing based on the fault diagnosis result.

[0041] In this embodiment, when a task failure is detected in the business layer, the task can be diagnosed in a timely manner to obtain a fault diagnosis result, and then the fault can be processed according to the fault diagnosis result to restore the model training task in a timely manner, thereby improving the fault handling efficiency and the model training efficiency, and also reducing the waste of computing resources.

[0042] The large-model-based fault handling method of this embodiment performs fault detection at the business layer to facilitate timely fault detection. When a fault is detected, it effectively diagnoses the fault and obtains an accurate diagnosis result. Finally, it memorizes the diagnosis result and handles the fault promptly and effectively. Compared with the existing manual monitoring model training tasks, this method can handle faults promptly, accurately, and effectively, effectively improving fault handling efficiency, avoiding unnecessary waste of computing resources, and effectively improving the training efficiency of model training tasks.

[0043] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the fault handling method based on the large model of this embodiment, in the above Figure 1 Based on the technical solution of the embodiment shown, the technical solution of the present disclosure is further and more effectively described by taking the task failure detected at the business layer as an example. Figure 2 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0044] S201. During the execution of the model training task, check whether the task is stuck at the business layer; if so, execute step S202; otherwise, if not, continue checking;

[0045] When an anomaly occurs during model training, one of the most direct manifestations is a task stall. In this embodiment, during business layer fault detection, by detecting whether the model training task being executed by the business layer is stalled, it is possible to promptly and accurately detect whether an anomaly has occurred during the execution of the business layer task.

[0046] For example, in this embodiment, the specific implementation of step S201 may include the following two situations:

[0047] Scenario 1: During the execution of the model training task, the business layer detects whether the utilization of each graphics processor (GPU) executing the task continues to reach the first preset threshold or the second preset threshold within a preset time period before the current moment, and the bandwidth continues to reach the third preset threshold; for example, in this scenario, the first preset threshold can be 100%, the second preset threshold can be 0, and the third preset threshold is 0; if so, it is determined that the task is stuck, that is, the task is determined to be faulty; otherwise, it is determined that the task is not stuck.

[0048] Specifically, during the training process, if the utilization of each GPU executing the task has been continuously 100% within the preset time period before the current moment, and the bandwidth has been continuously 0; it can be considered that each GPU has stopped executing the task and the task is stuck. Or if the utilization of some GPUs executing the task has been continuously 100% within the preset time period before the current moment, the utilization of individual GPUs has been continuously 0 within the preset time period before the current moment, and the bandwidth has been continuously 0; it can be considered that some GPUs have stopped executing the task, and some GPUs have encountered abnormalities and have not yet executed the task, and the task is stuck. Scenario 2: During the execution of the model training task, the business layer detects whether the logs of each GPU executing the task have not been refreshed within the preset time period before the current moment; if so, it is determined that the task is stuck, that is, it is determined that the task has failed; otherwise, it is determined that the task is not stuck.

[0049] Specifically, during normal task execution, each GPU executing the task will automatically refresh its log. If a task is interrupted due to a fault, the logs of each GPU executing the task will no longer be refreshed. Therefore, in this embodiment, by detecting whether each GPU's log is refreshed normally, it is possible to detect whether a task is abnormal. To achieve accurate detection, in this embodiment, the preset time period must be greater than the maximum log refresh rate.

[0050] For example, the preset time period in this embodiment can be set according to actual needs, for example, it can be 2 minutes, 1 minute or other time lengths, which are not limited here.

[0051] In this embodiment, step S201 is used to detect faults at the service layer. Figure 1 An implementation of step S101 of the illustrated embodiment.

[0052] Since the business layer is the specific location where tasks are executed, usually, when a task execution fails, it will be manifested at the business layer. For example, a task stuck is a manifestation of task execution interruption. Therefore, in this embodiment, by detecting whether a task is stuck at the business layer, the failure can be perceived sensitively and promptly at the business layer.

[0053] In actual application scenarios, there are many manifestations of task stagnation at the business layer. This embodiment uses the above two manifestations, Situation 1 and Situation 2, as examples to accurately and efficiently detect whether a task is stagnate at the business layer. In actual applications, task stagnation can also be detected based on other manifestations of task stagnation, which will not be detailed here.

[0054] In this embodiment, a set of agent components can be implemented in a sidecar manner to achieve fault detection on the business layer side. For example, when implementing step S201, a Hang Detect Agent can be set to detect whether the task is stuck.

[0055] S202, collect the communication times of each process in each communication group executing the task; execute step S203;

[0056] Specifically, the communication times of each process in each of the multiple processes in each of the multiple communication groups executing the task can be collected. Specifically, the communication times of each process in each communication group can be collected by setting an Agent component.

[0057] S203, based on the number of communications of the processes in the communication group executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; and execute step S204;

[0058] Specifically, the task may be diagnosed for fault based on the number of communications of each of the multiple processes in each of the multiple communication groups that execute the task.

[0059] For example, the specific implementation of step S203 may include the following steps:

[0060] (a1) For each communication group, obtain the faulty process whose communication times are less than the communication times of other processes in the communication group;

[0061] (b1) Detect whether the GPUs corresponding to the faulty processes in different communication groups are the same;

[0062] (c1) in response to the fact that the GPUs corresponding to the faulty processes in different communication groups are the same, determining that the GPUs corresponding to the faulty processes in the different communication groups are the faulty GPUs, and determining that the node where the faulty GPU is located is the faulty node;

[0063] In actual application scenarios, the communication times of each process in the same communication group should be consistent. If the communication times of a process in the same communication group are less than those of other processes, it can be considered that the GPU corresponding to the process is suspected to have failed. To further determine whether the GPU has failed, it is possible to further detect that processes in other communication groups that have less communication times than other processes also point to the GPU. In this case, it can be determined that the GPU is a failed GPU and the node where the GPU is located is the failed node. The node in this embodiment refers to the physical machine node where the GPU is located. In actual application scenarios, multiple GPUs can be configured on a node.

[0064] (d1) In response to the fact that the GPUs corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0065] That is, if the GPUs corresponding to the faulty processes in any two different communication groups are not the same, the faulty GPU cannot be determined, and the fault diagnosis result can be determined as being unable to determine the faulty node.

[0066] Further optionally, in one embodiment of the present disclosure, step S202 and step S203 may be replaced by the following step S202′ and step S203′:

[0067] S202', collecting the communication duration and the completion status of the communication operator of each process in each communication group executing the task; executing step S203'; the completion status of the communication is completed or not completed, indicating whether the current communication operator has completed the communication or not completed the communication within the communication duration;

[0068] S203', based on the communication duration of the communication operator of each process in each communication group executing the task and the completion status of the communication, the task is fault diagnosed to obtain a fault diagnosis result; and step S204 is executed;

[0069] For example, in a specific implementation, for each communication group, it is possible to detect whether there is a communication operator whose communication duration reaches a preset duration threshold but the communication process is not completed; if so, the process is determined to be a faulty process, the GPU corresponding to the faulty process is the faulty GPU, and the node where the faulty GPU is located is the faulty node; wherein the preset duration threshold can be set according to the needs of the actual scenario, for example, it can be 5s, 6s, or other time lengths, which are not limited here.

[0070] S204, checking whether the fault diagnosis result can determine the fault node, if not, executing step S205; if yes, executing step S210;

[0071] S205, collecting call stack information of each GPU executing the task; executing step S206;

[0072] S206. Based on the call stack information of each GPU executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; and execute step S207.

[0073] Specifically, the call stack information of each GPU that is normally executing a task should be consistent. In this embodiment, based on the call stack information of each GPU executing a task, it can be detected whether the call stack information of each GPU is consistent. If the call stack information of an individual GPU is inconsistent with the call stack information of other GPUs, it is considered that the individual GPU has a fault, and the fault of the individual GPU can be identified in the fault diagnosis result. If the call stack information of all GPUs is consistent, the fault diagnosis result is determined to be that the faulty node cannot be determined.

[0074] In this embodiment, steps S202-S203 and steps S205-S206 are the same as those described above. Figure 1 The implementation of step S102 in the illustrated embodiment.

[0075] Steps S202-S203 and steps S205-S206 are two different fault diagnosis schemes. These two schemes can exist independently. Regardless of which method is used, the fault can be accurately detected. In actual application scenarios, one of them can be selected according to needs. Alternatively, when used together, steps S205-S206 can be placed before steps S202-S203, or steps S202-S203 and steps S205-S206 can be performed simultaneously. In short, the above two fault diagnosis schemes can be used to perform fault diagnosis and locate the fault node in the fault diagnosis results as much as possible.

[0076] S207, further checking whether the fault diagnosis result can determine the fault node, if not, executing step S208; if yes, executing step S210;

[0077] S208, detecting whether the infrastructure layer of each node executing the task has a fault; if so, executing step S209; otherwise, executing step S213;

[0078] The node in this embodiment refers to the hardware computing device to which the GPU that executes the task is attached, which can also be called a computing node. Multiple GPU cards can be inserted into one hardware computing device.

[0079] The fault diagnosis performed in the above steps S202-S203 and steps S205-S206 is performed based on the information of the business layer. If the fault node cannot be determined in the fault diagnosis of the business layer, step S208 can be used to perform fault diagnosis at the infrastructure layer.

[0080] For example, when step S208 is specifically implemented, it may include at least one of the following steps:

[0081] (a2) Detecting whether kernel abnormalities occur in each node executing the task;

[0082] (b2) detecting whether any system component anomalies occur at each node executing the task; or

[0083] (c2) Detect whether any hardware failure occurs on each node executing the task.

[0084] Specifically, when the step S208 is implemented, the detection of whether the infrastructure layer of each node sends a fault can also be achieved by configuring the Agent component.

[0085] The kernel exceptions in this embodiment may include kernel crash, kernel deadlock, kernel OOM and other faults, which will cause node notReady after occurrence, thereby causing training interruption. In this embodiment, the kernel detection agent (KernelDetectAgent) component can be implemented to trace whether the kernel is abnormal, and the heartbeat signal sent to the fault-tolerant controller at the center end carries the detection result of whether the kernel is abnormal. In this way, the fault handling device on the fault-tolerant controller side at the center end can know in real time whether the kernel is abnormal based on the heartbeat signal reported by the KernelDetectAgent component. In this embodiment, the KernelDetectAgent component periodically sends a heartbeat signal to the fault handling device. For example, the period can be 30s, 25s or 35s, etc., which is not limited here.

[0086] In actual applications, the system software layer includes core components of a single machine, such as the k8s kubelet, container-runtime, and nvidia-container-toolkit. Exceptions in these components may also cause training interruptions, so fault detection of such software is necessary. In one embodiment of the present disclosure, a system detection agent component can be set up to detect anomalies in system components. When anomalies are found, they are reported to the fault-tolerant controller at the central end. In this way, the fault handling device on the fault-tolerant controller side of the central end can obtain information about component anomalies in real time.

[0087] In practical applications, node failures are the most common point of failure during training, especially for GPUs and RDMA network cards. Fluctuations in ambient temperature, voltage, and power can often trigger hardware failures. Therefore, in one embodiment of the present disclosure, corresponding Agent components can be configured to detect anomalies in these devices. When an anomaly occurs, it is reported to the central fault-tolerant controller through events, metrics, node conditions, and other methods. This allows the fault handling device on the central fault-tolerant controller to receive real-time information about the node failure.

[0088] In this embodiment, at the infrastructure layer, it is possible to accurately and comprehensively detect whether a fault occurs at the infrastructure layer from three dimensions: the kernel, system components, and node hardware, and to effectively perform fault detection at the infrastructure layer.

[0089] S209. In response to the presence of a node having a fault in the infrastructure layer, update the diagnosis result to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result; and execute step S210.

[0090] S210, shield the faulty node; execute step S211;

[0091] The shielding in this embodiment includes the following two methods:

[0092] Explicit blocking: After fault diagnosis, if a faulty node can be accurately located, cluster-level blocking will be performed, and all subsequent tasks cannot be scheduled to this faulty node.

[0093] Implicit blocking: When only suspected faulty nodes are found after fault diagnosis, task-level node blocking is performed without actual blocking. This ensures that the current task will no longer be scheduled to the suspected faulty node, while other tasks can still run on the suspected faulty node.

[0094] S211, configure a replacement node for the failed node; execute step S212;

[0095] Specifically, a replacement node may be configured for the failed node based on nodes with idle available GPUs in the resource pool.

[0096] S212: Reschedule the tasks on the failed node on the replacement node, and end.

[0097] S213, restart the task; execute step S214;

[0098] S214: If the task still fails after the restart, reschedule the task.

[0099] In actual application scenarios, some faults will recover after a restart. Therefore, in this embodiment, when the faulty node is uncertain and the fault location cannot be accurately located, you can first try restarting the task. This may resolve the fault and effectively improve fault handling efficiency. After restarting, return to step S201 to continue fault detection and processing. If a task fault is detected again, the task can be directly rescheduled and executed using a new node to ensure accurate fault handling.

[0100] Optionally, in one embodiment of the present disclosure, for scenario 1 of step S201, when it is detected that a task is stuck, the following steps may be used to diagnose the fault:

[0101] (a3) business characteristic information of the collection task;

[0102] For example, the utilization rate of each graphics processor executing a task within a preset time period before the current moment may be collected.

[0103] (b3) Based on the business characteristic information of the task, perform fault diagnosis on the task and obtain the fault diagnosis result;

[0104] For example, the specific implementation of step (b3) may include the following steps:

[0105] (a4) based on the utilization of each GPU of the task in a preset time period before the current moment, detecting whether there is a utilization equal to a second preset threshold;

[0106] (b4) in response to the presence of a utilization rate equal to a second preset threshold, determining that a GPU corresponding to the utilization rate equal to the second preset threshold is a faulty GPU, and a node where the faulty GPU is located is a faulty node;

[0107] (c4) In response to the absence of a utilization ratio equal to the second preset threshold, determining that the diagnosis result is that the faulty node cannot be determined.

[0108] The second preset threshold of this embodiment may be 0. If the utilization of a particular GPU remains 0 for a preset time period during task execution, it may be considered that a failure has occurred in the particular GPU and the task cannot be executed, resulting in task interruption.

[0109] Using steps (a4)-(c4), fault diagnosis can also be accurately achieved. Optionally, in one embodiment of the present disclosure, in case 1 of step S201, steps (a4)-(c4) can be placed before step S202. If the fault node cannot be determined according to step (c4), step S202 and subsequent diagnosis can be further performed to further improve the comprehensiveness and accuracy of fault diagnosis.

[0110] The large-model-based fault handling method of this embodiment adopts the above-mentioned solution, which can comprehensively, accurately and timely perform fault detection at the business layer and infrastructure layer, and handle the fault based on the detection results without human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; it can also avoid long-term interruption of training tasks and waste of computing resources.

[0111] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; the fault handling method based on the large model of this embodiment, in the above Figure 1 Based on the technical solution of the embodiment shown, the technical solution of the present disclosure is further and more effectively described by taking the task failure detected at the business layer as an example. Figure 3 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0112] S301: During the execution of the model training task, detect whether the performance of the task has degraded at the business layer. If so, execute step S302; otherwise, continue detecting.

[0113] For example, the specific implementation of step S301 may include the following steps:

[0114] (a5) During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles;

[0115] (b5) collecting an average value of the number of tokens processed per second by at least one graphics processor executing the task during the iteration period as a performance indicator for the corresponding iteration period;

[0116] Specifically, for each iteration cycle, the at least one GPU selected to execute the task can be determined based on pre-defined rules. Preferably, this step is implemented by collecting the average number of tokens processed per second by all GPUs executing the task during each iteration cycle as the performance indicator for the corresponding iteration cycle.

[0117] (c5) Based on the performance indicators of the iteration cycle, detect whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest historical iteration cycle reaches a preset ratio threshold; if so, execute step (d5); otherwise, determine that no task failure is detected, and return to continue detection; for example, the preset ratio threshold of this embodiment can be set according to actual experience, such as 30%, 25% or other ratio values, which are not limited here.

[0118] (d5) Determine whether the performance of the task has degraded, i.e., determine whether the task has failed.

[0119] In actual applications, task performance degradation is also a manifestation of task anomaly. In this embodiment, by detecting whether task performance degradation is detected at the business layer, it is also possible to timely and accurately detect whether anomalies occur during the execution of business layer tasks.

[0120] It should be noted that in this embodiment, the task failure is determined when the performance degradation of the task is detected as an example. In actual application scenarios, the detection of task failure can adopt step S301 of this embodiment and the above-mentioned step S302 at the same time. Figure 2 In step S201 of the illustrated embodiment, as long as any one of the methods detects a task failure, the task failure is determined; if the task failure cannot be determined after all the methods are used for detection, subsequent methods are further used for detection.

[0121] S302, collecting business feature information of the task;

[0122] For example, the time consumption of each stage in the last iteration cycle and the historical iteration cycles of the nearest neighbor may be specifically collected.

[0123] S303, based on the business characteristic information of the task, perform fault diagnosis on the task and obtain a fault diagnosis result;

[0124] For example, the specific implementation of step S303 may include the following steps:

[0125] (a6) For the same stage, detect whether the time taken for the last iteration cycle increases by more than a preset ratio relative to the time taken for the historical iteration cycles of the nearest neighbor;

[0126] In this embodiment, at a certain stage, the time consumed by the last iteration cycle increases by more than a preset proportion relative to the time consumed by the historical iteration cycle of the nearest neighbor. Specifically, the time consumed by this stage of the last iteration cycle, relative to the time consumed by this stage of the historical iteration cycle of the nearest neighbor, increases by a proportion greater than or equal to a preset proportion.

[0127] (b6) in response to the fact that the duration of the last iteration cycle of the phase increases by more than a preset proportion relative to the duration of the historical iteration cycles of the nearest neighbor, obtaining a GPU that executes the task of the phase as a faulty GPU, and a node where the faulty GPU is located as a faulty node;

[0128] (c6) In response to the fact that the time taken by the last iteration cycle of the stage relative to the historical iteration cycles of the nearest neighbor does not increase by a preset proportion, determining that the diagnosis result is that the faulty node cannot be determined.

[0129] By adopting the above steps (a6)-(c6), fault diagnosis can be performed accurately and effectively.

[0130] S304, checking whether the fault diagnosis result can determine the fault node, if not, executing step S305; if yes, executing step S313;

[0131] S305, collect the communication times of each process in each communication group executing the task; execute step S306;

[0132] S306. Based on the communication times of each process in each communication group executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; then execute step S307.

[0133] S307, checking whether the fault diagnosis result can determine the fault node, if not, executing step S308; if yes, executing step S313;

[0134] S308, collecting call stack information of each GPU executing the task; executing step S309;

[0135] S309, based on the call stack information of each GPU executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; and execute step S310;

[0136] S310, further checking whether the fault diagnosis result can determine the fault node, if not, executing step S311; if yes, executing step S313;

[0137] S311, detecting whether the infrastructure layer of each node executing the task has a fault; if so, executing step S312; otherwise, executing step S316;

[0138] S312. In response to the presence of a node having a fault in the infrastructure layer, update the diagnosis result to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result; and execute step S313.

[0139] S313, shield the faulty node; execute step S314;

[0140] S314. Configure a replacement node for the failed node; execute step S315;

[0141] S315: Reschedule the tasks on the failed node on the replacement node, and end.

[0142] S316, restart the task; execute step S317;

[0143] S317: If the task still fails after the restart, reschedule the task.

[0144] The implementation of the above steps S305-S317 can refer to the above Figure 2Steps S202 to S214 of the illustrated embodiment will not be described in detail here.

[0145] The large-model-based fault handling method of this embodiment adopts the above-mentioned solution, which can comprehensively, accurately and timely perform fault detection at the business layer and infrastructure layer, and handle the fault based on the detection results without human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; it can also avoid long-term interruption of training tasks and waste of computing resources.

[0146] Figure 4 This is a schematic diagram of the fourth embodiment of the present disclosure; the fault handling method based on the large model of this embodiment takes fault detection during abnormal exit as an example to further and more effectively describe the technical solution of the present disclosure. Figure 4 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0147] S401. If a task exits abnormally during execution, obtain the logs of each GPU executing the task;

[0148] S402: Perform fault diagnosis based on the logs of each GPU to obtain a fault diagnosis result;

[0149] For example, fault diagnosis can be performed based on the logs of each GPU to identify the faulty GPU and determine that the node where the faulty GPU is located is the faulty node.

[0150] In actual application scenarios, when an abnormal exit occurs, the log can contain fault information. For example, it can include two types of abnormalities: computing abnormalities and communication abnormalities.

[0151] Computational anomalies: These are usually manifested as CUDA Runtime Error exceptions in the log. You can configure the CudaRuntime Detect Agent component to detect and analyze various CUDA Runtime error patterns to determine whether there are any computational anomalies.

[0152] Similar to computing anomalies, communication anomalies are often reflected in logs by the NVIDIA Collective Communication Library (NCCL). By configuring the NCCLDetect Agent component, common NCCL failure patterns can be accumulated and analyzed to detect communication anomalies.

[0153] Based on the logs of each GPU, it is possible to diagnose whether a computing anomaly or a communication anomaly occurs. It should be noted that if a computing anomaly is detected in the log of a certain GPU, the GPU is determined to be a faulty GPU. If a communication anomaly is detected in the log of a certain GPU, it will be further identified whether the communication anomaly refers to a current-end communication anomaly, a peer-end communication anomaly, or whether it is impossible to determine which end has a communication anomaly. If the log indicates a current-end communication anomaly, the GPU on the current end can be considered a faulty GPU; if the log indicates a peer-end communication anomaly, the GPU on the peer end can be considered a faulty GPU; if the log indicates that it is impossible to determine which end has a communication anomaly, it can be considered that the faulty GPU cannot be determined, or both GPUs at both ends can be considered faulty GPUs at the same time.

[0154] S403, checking whether the fault diagnosis result can determine the fault node, if not, executing step S404; if yes, executing step S406;

[0155] S404: Detect whether the infrastructure layer of each node executing the task has a fault; if so, execute step S405; otherwise, execute step S409;

[0156] S405. In response to the presence of a node having a fault in the infrastructure layer, update the diagnosis result to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result; and execute step S406.

[0157] S406, shield the faulty node; execute step S407;

[0158] S407, configure a replacement node for the failed node; execute step S408;

[0159] S408: Restart the task and reschedule the task on the failed node on the replacement node, and end.

[0160] S409, restart the task; execute step S410;

[0161] S410: If the task still exits abnormally after restarting, reschedule the task.

[0162] The implementation of the above steps S404-S410 can refer to the above Figure 2 Steps S208 - S214 of the illustrated embodiment will not be described in detail here.

[0163] The large-model-based fault handling method of this embodiment adopts the above-mentioned scheme, and can comprehensively, accurately and timely perform fault detection at the framework layer and infrastructure layer when a task exits abnormally, and handle the fault based on the detection results without the need for human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; it can also avoid long-term interruption of training tasks and waste of computing resources.

[0164] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 5 As shown, this embodiment provides a fault handling device 500 based on a large model, including:

[0165] A fault detection module 501 is used to perform fault detection on the task at the business layer during the execution of the model training task;

[0166] A fault diagnosis module 502 is configured to, in response to detecting a task fault at the business layer, perform fault diagnosis on the task and obtain a fault diagnosis result;

[0167] The fault processing module 503 is configured to perform fault processing based on the fault diagnosis result.

[0168] The large model-based fault handling device 500 of this embodiment implements the fault handling principle and technical effect by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0169] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure; Figure 6 As shown, the fault handling device 600 based on the large model of this embodiment, in the above Figure 5 Based on the technical solutions of the illustrated embodiments, the technical solutions of the present disclosure are further described in more detail. Figure 6 As shown, the fault handling device 600 based on the large model of this embodiment includes the above Figure 5 Modules with the same name and function are shown as follows: a fault detection module 601 , a fault diagnosis module 602 , and a fault processing module 603 .

[0170] In this embodiment, the fault diagnosis module 602 is used to:

[0171] The communication times of the processes in the communication group that executes the task are collected; specifically, the communication times of each process in the multiple processes in each communication group in the multiple communication groups that execute the task can be collected.

[0172] Based on the communication times of the processes in the communication group that executes the task, the task is diagnosed for a fault and a fault diagnosis result is obtained. Specifically, based on the communication times of each of the multiple processes in each of the multiple communication groups that execute the task, the task is diagnosed for a fault and a fault diagnosis result is obtained.

[0173] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0174] For each communication group, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group;

[0175] Check whether the graphics processors corresponding to the faulty processes in different communication groups are the same;

[0176] In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in the different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node;

[0177] In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0178] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0179] Collecting call stack information of each graphics processor executing the task;

[0180] Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

[0181] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602′ is configured to:

[0182] Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors;

[0183] In response to the presence of a graphics processor having call stack information different from that of other graphics processors, determining that the graphics processor having call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node;

[0184] In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

[0185] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602′ is configured to:

[0186] In response to detecting the task failure at the service layer, collecting service feature information of the task;

[0187] Based on the service feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

[0188] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is configured to:

[0189] During the execution of the model training task, the business layer detects whether the task is successful.

[0190] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is configured to:

[0191] During the execution of the model training task, detecting at the business layer whether the utilization of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold;

[0192] If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

[0193] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0194] In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

[0195] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0196] Based on the utilization rates of each graphics processor of the task within a preset time period before the current moment, detecting whether there is a utilization rate equal to the second preset threshold;

[0197] In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node;

[0198] In response to the absence of a utilization rate equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

[0199] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is configured to:

[0200] During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment;

[0201] If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

[0202] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is configured to:

[0203] During the execution of the model training task, the business layer detects whether the performance of the task has degraded.

[0204] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0205] During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles;

[0206] An average value of the number of symbols processed per second by at least one graphics processor executing the task within the iteration cycle is collected as a performance indicator for the corresponding iteration cycle. Specifically, for each iteration cycle, an average value of the number of symbols processed per second by all graphics processors executing the task within the iteration cycle can be collected as the performance indicator for the corresponding iteration cycle.

[0207] Based on the performance indicators of each iteration cycle, detect whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold;

[0208] If so, it is determined that the performance of the task has degraded, that is, the task is determined to have failed.

[0209] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0210] In response to detecting the task failure at the business layer, the time consumed at each stage in the last iteration cycle and the nearest neighbor historical iteration cycles is collected.

[0211] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0212] For the same stage, check whether the time taken for the last iteration cycle increases by more than a preset ratio relative to the historical iteration cycle of the nearest neighbor;

[0213] In response to a time increase of more than a preset ratio between the last iteration cycle of the stage and the historical iteration cycles of the nearest neighbor, obtaining a graphics processor executing the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located as a faulty node;

[0214] In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycles of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0215] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is further configured to:

[0216] If the fault node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task has a fault;

[0217] In response to the presence of a node having a fault in the infrastructure layer, the diagnosis result is updated to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result.

[0218] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to perform at least one of the following:

[0219] Detecting whether kernel abnormalities occur in each node executing the task;

[0220] Detecting whether any system component anomalies occur at each node executing the task; or

[0221] Detect whether hardware failure occurs in each node executing the task.

[0222] Further optionally, as Figure 6 As shown, in one embodiment of the present disclosure, the fault handling device 600 further includes:

[0223] The log acquisition module 604 is used to acquire the logs of each graphics processor executing the task if the task exits abnormally during execution;

[0224] The fault diagnosis module 602 is further configured to perform fault diagnosis based on the logs of each graphics processor, and determine that the fault diagnosis result is that the fault node cannot be determined.

[0225] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is configured to:

[0226] Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined to be the faulty node.

[0227] Further optionally, as Figure 6 As shown, in one embodiment of the present disclosure, the fault processing module 603 includes:

[0228] A shielding unit 6031 is configured to shield a faulty node when a faulty node is identified in the fault diagnosis result;

[0229] A configuration unit 6032 is configured to configure a replacement node for the failed node;

[0230] The node rescheduling unit 6033 is configured to reschedule the tasks on the failed node on the replacement node.

[0231] Further optionally, as Figure 6 As shown, in one embodiment of the present disclosure, the fault processing module 603 further includes:

[0232] A restart unit 6034 is configured to restart the task when the fault diagnosis result indicates that the fault node cannot be determined;

[0233] The task rescheduling unit 6035 is further configured to reschedule the task if the task still fails after restart.

[0234] The large model-based fault handling device 600 of this embodiment implements the fault handling principle and technical effect by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0235] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0236] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0237] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0238] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0239] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0240] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the above-mentioned methods of the present disclosure described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the above-mentioned methods of the present disclosure by any other appropriate means (e.g., by means of firmware).

[0241] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0242] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0243] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0244] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0245] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0246] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0247] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0248] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A fault handling method based on a large model, comprising: During the execution of the model training task, fault detection is performed on the task at the business layer; In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result; Performing fault processing based on the fault diagnosis result; In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result includes: Collect the number of communications of processes in the communication group that executes the task; Performing fault diagnosis on the task based on the number of communications of the processes in the communication group that executes the task, and obtaining a fault diagnosis result; During the execution of the model training task, fault detection is performed on the task at the business layer, including: During the execution of the model training task, the business layer detects whether the task is successful; During the execution of the model training task, the business layer detects whether the task is successful, including: During the execution of the model training task, detecting at the business layer whether the utilization of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

2. The method according to claim 1, wherein Performing fault diagnosis on the task based on the number of communications of the processes in the communication group that executes the task, and obtaining a fault diagnosis result, including: For the communication group, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group; Check whether the graphics processors corresponding to the faulty processes in different communication groups are the same; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in the different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

3. The method according to claim 1, wherein In response to detecting the task failure at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result may also include the following steps: Collecting call stack information of each graphics processor executing the task; Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

4. The method according to claim 3, wherein: Performing fault diagnosis on the task based on call stack information of each graphics processor executing the task to obtain a fault diagnosis result includes: Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors; In response to the presence of a graphics processor with call stack information different from that of other graphics processors, determining that the graphics processor with call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

5. The method according to claim 1, wherein In response to detecting the task failure at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result may also include the following steps: In response to detecting the task failure at the service layer, collecting service feature information of the task; Based on the service feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

6. The method according to claim 1, wherein In response to detecting the task failure at the service layer, collecting service feature information of the task, including: In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

7. The method according to claim 6, wherein: Performing fault diagnosis on the task based on the service feature information of the task to obtain a fault diagnosis result includes: Based on the utilization rates of each graphics processor of the task within a preset time period before the current moment, detecting whether there is a utilization rate equal to the second preset threshold; In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node; In response to the absence of a utilization rate equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

8. The method according to claim 1, wherein During the execution of the model training task, the business layer checks whether the task is successful, including: During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

9. The method according to claim 5, wherein: During the execution of the model training task, fault detection is performed on the task at the business layer, including: During the execution of the model training task, the business layer detects whether the performance of the task has degraded.

10. The method according to claim 9, wherein: During the execution of the model training task, the business layer detects whether the performance of the task has degraded, including: During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles; collecting an average value of the number of symbols processed per second by at least one graphics processor executing the task during the iteration period as a performance indicator for the corresponding iteration period; Based on the performance indicator of the iteration cycle, detecting whether the rate of decrease of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold; If so, it is determined that the performance of the task has degraded, that is, the task is determined to have failed.

11. The method according to claim 10, wherein: In response to detecting the task failure at the service layer, collecting service feature information of the task, including: In response to detecting the task failure at the business layer, the time consumed at each stage in the last iteration cycle and the nearest neighbor historical iteration cycles is collected.

12. The method according to claim 11, wherein Performing fault diagnosis on the task based on the service feature information of the task to obtain a fault diagnosis result includes: For the same stage, check whether the time taken for the last iteration cycle increases by more than a preset ratio relative to the historical iteration cycle of the nearest neighbor; In response to a time increase of more than a preset ratio between the last iteration cycle of the stage and the historical iteration cycles of the nearest neighbor, obtaining a graphics processor executing the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located as a faulty node; In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycles of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

13. The method according to claim 1, 3 or 5, wherein In response to detecting the task fault, performing fault diagnosis on the task to obtain a fault diagnosis result, further comprising: If the fault node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task has a fault; In response to the presence of a node having a fault in the infrastructure layer, the diagnosis result is updated to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result.

14. The method according to claim 13, wherein Detecting whether a failure occurs in the infrastructure layer of each node executing the task, including at least one of the following: Detecting whether kernel abnormalities occur in each node executing the task; Detecting whether any system component anomalies occur at each node executing the task; or Detect whether hardware failure occurs in each node executing the task.

15. The method according to claim 13, wherein Before detecting whether a failure occurs in the infrastructure layer of each node executing the task, the method further includes: If the task exits abnormally during execution, obtaining logs of each graphics processor executing the task; A fault diagnosis is performed based on the logs of each graphics processor, and it is determined that a faulty node cannot be determined as a result of the fault diagnosis.

16. The method according to claim 15, wherein The method further comprises: Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined to be the faulty node.

17. The method according to any one of claims 1-12 and 14-16, wherein: Based on the fault diagnosis result, perform fault handling, including: When a faulty node is identified in the fault diagnosis result, shielding the faulty node; Configuring a replacement node for the failed node; The tasks on the failed node are rescheduled on the replacement node.

18. The method according to claim 17, wherein: Performing fault processing based on the fault diagnosis result also includes: When the fault diagnosis result indicates that the fault node cannot be determined, restarting the task; If the task still fails after the restart, the task is rescheduled.

19. A fault handling device based on a large model, comprising: A fault detection module is used to detect faults in the model training task at the business layer during the execution of the task; a fault diagnosis module, configured to, in response to detecting a task fault at the service layer, perform fault diagnosis on the task and obtain a fault diagnosis result; A fault processing module, configured to perform fault processing based on the fault diagnosis result; Wherein, the fault diagnosis module is used to: Collect the number of communications of processes in the communication group that executes the task; Performing fault diagnosis on the task based on the number of communications of the processes in the communication group that executes the task, and obtaining a fault diagnosis result; The fault detection module is used to: During the execution of the model training task, the business layer detects whether the task is successful; Wherein, the fault detection module is used to: During the execution of the model training task, detecting at the business layer whether the utilization of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

20. The device according to claim 19, wherein The fault diagnosis module is used to: For the communication group, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group; Check whether the graphics processors corresponding to the faulty processes in different communication groups are the same; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in the different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

21. The apparatus according to claim 19, wherein The steps performed by the fault diagnosis module can also be replaced by the following steps: Collecting call stack information of each graphics processor executing the task; Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

22. The device according to claim 21, wherein The fault diagnosis module is used to: Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors; In response to the presence of a graphics processor with call stack information different from that of other graphics processors, determining that the graphics processor with call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

23. The apparatus according to claim 19, wherein The steps performed by the fault diagnosis module can also be replaced by the following steps: In response to detecting the task failure at the service layer, collecting service feature information of the task; Based on the service feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

24. The apparatus according to claim 19, wherein The fault diagnosis module is used to: In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

25. The apparatus according to claim 24, wherein The fault diagnosis module is used to: Based on the utilization rates of each graphics processor of the task within a preset time period before the current moment, detecting whether there is a utilization rate equal to the second preset threshold; In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node; In response to the absence of a utilization rate equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

26. The apparatus according to claim 19, wherein The fault detection module is used to: During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

27. The apparatus according to claim 23, wherein The fault detection module is used to: During the execution of the model training task, the business layer detects whether the performance of the task has degraded.

28. The apparatus according to claim 27, wherein The fault diagnosis module is used to: During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles; collecting an average value of the number of symbols processed per second by at least one graphics processor executing the task during the iteration period as a performance indicator for the corresponding iteration period; Based on the performance indicator of the iteration cycle, detecting whether the rate of decrease of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold; If so, it is determined that the performance of the task has degraded, that is, the task is determined to have failed.

29. The apparatus according to claim 28, wherein The fault diagnosis module is used to: In response to detecting the task failure at the business layer, the time consumed at each stage in the last iteration cycle and the nearest neighbor historical iteration cycles is collected.

30. The apparatus according to claim 29, wherein The fault diagnosis module is used to: For the same stage, check whether the time taken for the last iteration cycle increases by more than a preset ratio relative to the historical iteration cycle of the nearest neighbor; In response to a time increase of more than a preset ratio between the last iteration cycle of the stage and the historical iteration cycles of the nearest neighbor, obtaining a graphics processor executing the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located as a faulty node; In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycles of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

31. The apparatus of claim 19, 21 or 23, wherein The fault diagnosis module is further used to: If the fault node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task has a fault; In response to the presence of a node having a fault in the infrastructure layer, the diagnosis result is updated to identify the node having a fault in the infrastructure layer as a faulty node in the diagnosis result.

32. The apparatus according to claim 31, wherein The fault diagnosis module is configured to perform at least one of the following: Detecting whether kernel abnormalities occur in each node executing the task; Detecting whether any system component anomalies occur at each node executing the task; or Detect whether hardware failure occurs in each node executing the task.

33. The apparatus according to claim 31, wherein The device further comprises: A log acquisition module, configured to acquire logs of each graphics processor executing the task if the task exits abnormally during execution; The fault diagnosis module is further configured to perform fault diagnosis based on the logs of each graphics processor, and determine that the fault diagnosis result is that the fault node cannot be determined.

34. The apparatus according to claim 33, wherein The fault diagnosis module is used to: Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined to be the faulty node.

35. The device according to any one of claims 19-30, 32-34, wherein: The fault processing module includes: a shielding unit, configured to shield a faulty node when a faulty node is identified in the fault diagnosis result; A configuration unit, configured to configure a replacement node for the failed node; A node rescheduling unit is used to reschedule tasks on the failed node on the replacement node.

36. The apparatus of claim 35, wherein: The fault processing module further includes: A restart unit, configured to restart the task when the fault diagnosis result indicates that the fault node cannot be determined; The task rescheduling unit is further configured to reschedule the task if the task still fails after restart.

37. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-18.

39. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images