Fault processing method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a fault processing method and device based on a large model, electronic equipment and a storage medium, and relates to the technical fields of large models, hardware fault processing, machine learning and the like. According to the specific implementation scheme, in the execution process of a model training task, fault detection is conducted on the task in a service layer; when the task fault is detected in response to the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result; and performing fault processing based on the fault diagnosis result. According to the technology disclosed by the invention, the fault can be automatically processed in the model training task, and the fault processing efficiency can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as large models, hardware fault processing, and machine learning, and more particularly to a fault processing method, device, electronic device, and storage medium based on a large model. Background Art

[0002] With the rapid development of artificial intelligence, the training of large models has become a very important task.

[0003] Large model training often encounters hardware failures, task deadlocks, and other phenomena, resulting in training task interruptions. In the prior art, algorithm engineers are required to manually monitor model training tasks and manually resume training when training is interrupted. Summary of the invention

[0004] The present disclosure provides a fault handling method, device, electronic device and storage medium based on a large model.

[0005] According to one aspect of the present disclosure, a fault handling method based on a large model is provided, comprising:

[0006] During the execution of the model training task, fault detection is performed on the task at the business layer;

[0007] In response to detecting the task failure at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result;

[0008] Based on the fault diagnosis result, fault processing is performed.

[0009] According to another aspect of the present disclosure, a fault handling device based on a large model is provided, comprising:

[0010] A fault detection module is used to perform fault detection on the task at the business layer during the execution of the model training task;

[0011] A fault diagnosis module, configured to perform fault diagnosis on the task in response to detecting the task fault at the service layer, and obtain a fault diagnosis result;

[0012] The fault processing module is used to perform fault processing based on the fault diagnosis result.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and any possible implementation method.

[0019] The technology disclosed in the present invention can automatically handle faults in model training tasks and can effectively improve fault handling efficiency.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0027] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0028] Figure 7 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0031] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0032] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0033] In actual scenarios, during the model training process, algorithm engineers are required to manually monitor the training tasks. Usually, when a training failure is discovered manually, the model training task has been interrupted for several hours or even more than ten hours. The fault handling efficiency is extremely low, which seriously affects the training efficiency of the model.

[0034] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a fault handling method based on a large model, which may specifically include the following steps:

[0035] S101. During the execution of the model training task, fault detection is performed on the task at the business layer;

[0036] The executor of the large model-based fault handling method of this embodiment may be a large model-based fault handling device, which may be arranged in a fault-tolerant controller at the central end of the model training system to promptly handle faults in the model training task.

[0037] In this embodiment, in order to promptly discover faults in the model training task, the faults are promptly detected at the business layer.

[0038] The fault detection of this embodiment can be implemented by configuring an agent component at the business layer.

[0039] S102, in response to detecting a task failure at the business layer, diagnose the task failure and obtain a failure diagnosis result;

[0040] S103: Perform fault processing based on the fault diagnosis result.

[0041] In this embodiment, when a task failure is detected in the business layer, the task can be diagnosed in a timely manner to obtain a fault diagnosis result, and then the fault can be processed according to the fault diagnosis result to restore the model training task in a timely manner, thereby improving the fault handling efficiency, thereby improving the model training efficiency, and reducing the waste of computing resources.

[0042] The fault handling method based on the large model of this embodiment performs fault detection at the business layer to timely discover faults; and when a fault is detected, the fault is effectively diagnosed to obtain accurate fault diagnosis results; finally, the fault diagnosis results are memorized to handle the fault in a timely and effective manner. Compared with the manual monitoring model training task in the prior art, the fault can be handled in a timely, accurate and effective manner, effectively improving the fault handling efficiency, avoiding unnecessary waste of computing resources, and effectively improving the training efficiency of the model training task.

[0043] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; the fault handling method based on the large model of this embodiment, in the above Figure 1 Based on the technical solution of the embodiment shown, the technical solution of the present disclosure is further and more effectively described by taking the task failure detected at the business layer as an example. Figure 2 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0044] S201. During the execution of the model training task, whether the task is stuck is detected at the business layer; if it is, execute step S202; otherwise, if it is not stuck, continue to detect;

[0045] When an exception occurs during model training, one of the most direct manifestations is that the task is stuck. In this embodiment, when detecting faults at the business layer, by detecting whether the model training task being executed by the business layer is stuck, it is possible to timely and accurately perceive whether an exception occurs during the execution of the business layer task.

[0046] For example, in this embodiment, the specific implementation of step S201 may include the following two situations:

[0047] Scenario 1: During the execution of the model training task, the business layer detects whether the utilization rate of each graphics processor (Graphics Processing Unit; GPU) executing the task continues to reach the first preset threshold or the second preset threshold within a preset time period before the current moment, and the bandwidth continues to reach the third preset threshold; for example, in this scenario, the first preset threshold may be 100%, the second preset threshold may be 0, and the third preset threshold is 0; if so, it is determined that the task is stuck, that is, it is determined that the task has failed; otherwise, it is determined that the task is not stuck.

[0048] Specifically, during the training process, if the utilization of each GPU executing the task has been continuously 100% within the preset time period before the current moment, and the bandwidth has been continuously 0; it can be considered that each GPU has stopped executing the task and the task is stuck. Or if the utilization of some GPUs executing the task has been continuously 100% within the preset time period before the current moment, the utilization of individual GPUs has been continuously 0 within the preset time period before the current moment, and the bandwidth has been continuously 0; it can be considered that some GPUs have stopped executing the task, and some GPUs have experienced an abnormality and have not yet executed the task, and the task is stuck. Scenario 2: During the execution of the model training task, the business layer detects whether the logs of each GPU executing the task have not been refreshed within the preset time period before the current moment; if so, it is determined that the task is stuck, that is, it is determined that the task is faulty; otherwise, it is determined that the task is not stuck.

[0049] Specifically, during the normal execution of a task, each GPU executing the task will inevitably refresh the log. If a task is abnormally interrupted due to a fault, the log of each GPU executing the task will inevitably no longer be refreshed. Therefore, in this embodiment, by detecting whether the log of each GPU is refreshed normally, it can be perceived whether the task is abnormal. In order to achieve accurate detection, in this embodiment, the preset time period set needs to be greater than the maximum log refresh frequency.

[0050] For example, the preset time period in this embodiment can be set according to actual needs, for example, it can be 2 minutes, 1 minute or other time lengths, which are not limited here.

[0051] In this embodiment, step S201 is used to detect faults at the business layer. Figure 1 An implementation of step S101 of the illustrated embodiment.

[0052] Since the business layer is the specific location of task execution, usually, when a task execution fails, it can be manifested in the business layer. For example, if a task is stuck, it is a manifestation of task execution interruption. Therefore, in this embodiment, by detecting whether a task is stuck at the business layer, the failure can be keenly and timely perceived at the business layer.

[0053] In actual application scenarios, there are many manifestations of task stagnation at the business layer. In this embodiment, the above two manifestations of situation 1 and situation 2 are taken as examples to accurately and efficiently detect whether a task is stagnate at the business layer. In actual applications, it is also possible to detect whether a task is stagnate based on other manifestations of task stagnation, which will not be described one by one here.

[0054] In this embodiment, a set of agent components can be implemented in a sidecar manner to realize fault perception on the business layer side. For example, when the step S201 is implemented, a Hang Detect Agent can be set to detect whether the task is stuck.

[0055] S202, collecting the communication times of each process in each communication group executing the task; executing step S203;

[0056] Specifically, the communication times of each process in each of the multiple processes in each of the multiple communication groups executing the task may be collected. Specifically, the communication times of each process in each communication group may be collected by setting an Agent component.

[0057] S203, based on the communication times of the processes in the communication group executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; execute step S204;

[0058] Specifically, the task may be diagnosed for a fault based on the number of communications of each of the multiple processes in each of the multiple communication groups that execute the task.

[0059] For example, when the step S203 is specifically implemented, it may include the following steps:

[0060] (a1) For each communication group, obtain the faulty process whose communication times are less than the communication times of other processes in the communication group;

[0061] (b1) Detect whether the GPUs corresponding to the faulty processes in different communication groups are the same;

[0062] (c1) in response to the fact that the GPUs corresponding to the faulty processes in different communication groups are the same, determining that the GPUs corresponding to the faulty processes in the different communication groups are the faulty GPUs, and the node where the faulty GPU is located is the faulty node;

[0063] In actual application scenarios, the communication times of each process in the same communication group should be consistent. If the communication times of a process in the same communication group are less than that of other processes, it can be considered that the GPU corresponding to the process is suspected to have a fault. In order to further determine whether the GPU has a fault, it can be further detected that the processes in other communication groups whose communication times are less than that of other processes also point to the GPU, then it can be determined that the GPU is a faulty GPU, and the node where the GPU is located is a faulty node. The node of this embodiment refers to the physical machine node where the GPU is located. In actual application scenarios, multiple GPUs can be configured on a node.

[0064] (d1) In response to the fact that the GPUs corresponding to the faulty processes in different communication groups are not the same, determining that the diagnosis result is that the faulty node cannot be determined.

[0065] That is to say, if the GPUs corresponding to the faulty processes in any two different communication groups are not the same, the faulty GPU cannot be determined at this time, and the fault diagnosis result can be determined as the faulty node cannot be determined.

[0066] Further optionally, in one embodiment of the present disclosure, step S202 and step S203 may be replaced by the following step S202' and step S203':

[0067] S202', collecting the communication duration of the communication operator of each process in each communication group executing the task and the completion status of the communication; executing step S203'; the completion status of the communication is completed or not completed, indicating that the current communication operator has completed the communication or not completed the communication within the communication duration;

[0068] S203', based on the communication duration of the communication operator of each process in each communication group executing the task and the completion status of the communication, perform fault diagnosis on the task to obtain a fault diagnosis result; execute step S204;

[0069] For example, in a specific implementation, for each communication group, it can be detected whether there is a communication operator whose communication duration reaches a preset duration threshold but a process of not completing the communication; if so, the process is determined to be a faulty process, the GPU corresponding to the faulty process is a faulty GPU, and the node where the faulty GPU is located is a faulty node; wherein the preset duration threshold can be set according to the needs of the actual scenario, for example, it can be 5s, 6s, or other time lengths, which are not limited here.

[0070] S204, checking whether the fault diagnosis result can determine the faulty node, if not, executing step S205; if yes, executing step S210;

[0071] S205, collecting call stack information of each GPU executing the task; executing step S206;

[0072] S206, based on the call stack information of each GPU executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; execute step S207;

[0073] Specifically, the call stack information of each GPU that normally executes the task should be consistent. In this embodiment, based on the call stack information of each GPU that executes the task, it can be detected whether the call stack information of each GPU is consistent. If the call stack information of an individual GPU is inconsistent with the call stack information of other GPUs, it is considered that the individual GPU has a fault, and then the fault of the individual GPU can be identified in the fault diagnosis result. If the call stack information of all GPUs is consistent, it is determined that the faulty node cannot be determined.

[0074] Steps S202-S203 and steps S205-S206 in this embodiment are the same as those described above. Figure 1 The implementation manner of step S102 of the illustrated embodiment.

[0075] Steps S202-S203 and steps S205-S206 are two different fault diagnosis schemes, which can exist independently. No matter which method is used, the fault can be accurately detected. In actual application scenarios, one of them can be selected according to needs. Or when used together, steps S205-S206 can also be located before steps S202-S203, or steps S202-S203 and steps S205-S206 can also be performed simultaneously. In short, the above two fault diagnosis schemes can be used for fault diagnosis, and the faulty node can be located in the fault diagnosis result as much as possible.

[0076] S207, further detecting whether the fault diagnosis result can determine the faulty node, if not, executing step S208; if yes, executing step S210;

[0077] S208, detecting whether the infrastructure layer of each node executing the task fails; if so, executing step S209; otherwise, executing step S213;

[0078] The node in this embodiment refers to the hardware computing device to which the GPU that executes the task is attached, which can also be called a computing node. Multiple GPU cards can be inserted into one hardware computing device.

[0079] The fault diagnosis performed in the above steps S202-S203 and steps S205-S206 is performed based on the information of the business layer. If the fault node cannot be determined in the fault diagnosis of the business layer, step S208 can be used to perform fault diagnosis at the infrastructure layer.

[0080] For example, when step S208 is specifically implemented, it may include at least one of the following steps:

[0081] (a2) Detect whether kernel abnormalities occur in each node executing the task;

[0082] (b2) detecting whether any system component abnormality occurs at each node executing the task; or

[0083] (c2) Check whether any hardware failure occurs on each node executing the task.

[0084] Specifically, when the step S208 is implemented, the detection of whether the infrastructure layer of each node sends a fault can also be implemented by configuring the Agent component.

[0085] The kernel exceptions in this embodiment may include kernel crash, kernel deadlock, kernel OOM and other faults, which will cause node notReady after occurrence, thereby causing training interruption. In this embodiment, the kernel detection agent (KernelDetect Agent) component can be implemented to track (trace) whether the kernel is abnormal, and the heartbeat signal sent to the fault-tolerant controller at the center end carries the detection result of whether the kernel is abnormal. In this way, the fault handling device on the fault-tolerant controller side of the center end can know in real time whether the kernel is abnormal based on the heartbeat signal reported by the Kernel Detect Agent component. In this embodiment, the Kernel Detect Agent component periodically sends a heartbeat signal to the fault handling device. For example, the period can be 30s, 25s or 35s, etc., which is not limited here.

[0086] In actual applications, the system software layer includes core components of a single machine such as k8s kubelet, container-runtime, and nvidia-container-toolkit. When these components are abnormal, they may also cause training interruptions, so it is necessary to perform fault detection on such software. In one embodiment of the present disclosure, a system detection agent (System Detect Agent) component can be set to perform abnormality detection on system components, and when an abnormality is found, it is reported to the fault-tolerant controller at the center end. In this way, the fault handling device on the fault-tolerant controller side of the center end can obtain the message of component abnormality in real time.

[0087] In practical applications, node failure is the most common failure point in the training process, especially for GPU and RDMA network cards. Fluctuations in ambient temperature and voltage power often cause hardware failures. Therefore, in one embodiment of the present disclosure, these devices can be configured with corresponding Agent components to perform abnormality detection. When an abnormality occurs, it is reported to the fault-tolerant controller at the center end through events, metrics, node conditions, etc. In this way, the fault handling device at the fault-tolerant controller side of the center end can obtain node failure messages in real time.

[0088] In this embodiment, at the infrastructure layer, it is possible to accurately and comprehensively detect whether a fault occurs at the infrastructure layer from three dimensions: the kernel, system components, and node hardware, and to effectively perform fault detection at the infrastructure layer.

[0089] S209, in response to the existence of a node with a fault in the infrastructure layer, updating the diagnosis result to identify the node with a fault in the infrastructure layer as a faulty node in the diagnosis result; executing step S210;

[0090] S210, shielding the faulty node; executing step S211;

[0091] The shielding of this embodiment includes the following two methods:

[0092] Explicit shielding: After fault diagnosis, if a faulty node can be accurately located, the node will be shielded at the cluster level, and all subsequent tasks cannot be scheduled to this faulty node.

[0093] Implicit shielding: When only suspected faulty nodes can be found after fault diagnosis, task-level node shielding is performed without actual shielding. This ensures that the current task will no longer be scheduled to the suspected faulty node, while other tasks can still run on the suspected faulty node.

[0094] S211, configuring a replacement node for the failed node; executing step S212;

[0095] Specifically, a replacement node may be configured for the failed node based on nodes with idle available GPUs in the resource pool.

[0096] S212: Reschedule the tasks on the failed node on the replacement node, and end.

[0097] S213, restart the task; execute step S214;

[0098] S214: If the task still fails after restarting, reschedule the task.

[0099] In actual application scenarios, some faults will recover after restarting. Therefore, in this embodiment, when the faulty node is uncertain and the fault location cannot be accurately located, you can try to restart the task first, which may solve the fault and effectively improve the efficiency of fault handling. After restarting, return to step S201 to continue fault detection and processing. If the task fault is detected again, the task can be directly rescheduled at this time, and a new node can be used to execute the task to ensure the accuracy of fault handling.

[0100] Optionally, in an embodiment of the present disclosure, for situation 1 of step S201, when it is detected that the task is stuck, the following steps may be used to diagnose the fault:

[0101] (a3) Business characteristic information of the collection task;

[0102] For example, the utilization rate of each graphics processor executing a task within a preset time period before the current moment may be specifically collected.

[0103] (b3) Based on the business feature information of the task, perform fault diagnosis on the task and obtain the fault diagnosis result;

[0104] For example, the specific implementation of step (b3) may include the following steps:

[0105] (a4) based on the utilization of each GPU of the task in a preset time period before the current moment, detecting whether there is a utilization equal to a second preset threshold;

[0106] (b4) in response to the existence of a utilization rate equal to the second preset threshold, determining that a GPU corresponding to the utilization rate equal to the second preset threshold is a faulty GPU, and a node where the faulty GPU is located is a faulty node;

[0107] (c4) In response to the absence of a utilization ratio having a size equal to the second preset threshold, determining that the diagnosis result is that the faulty node cannot be determined.

[0108] The second preset threshold of this embodiment may be 0. If the utilization rate of an individual GPU continues to be 0 within a preset time period during task execution, it may be considered that the individual GPU has a fault and cannot execute the task, resulting in task interruption.

[0109] By adopting the method of steps (a4)-(c4), it is also possible to accurately diagnose the fault. Optionally, in one embodiment of the present disclosure, in the case of step S201, the steps (a4)-(c4) can be placed before step S202. If the fault node cannot be determined according to step (c4), further performing step S202 and subsequent diagnosis can further improve the comprehensiveness and accuracy of fault diagnosis.

[0110] The large model-based fault handling method of this embodiment adopts the above-mentioned scheme, which can comprehensively, accurately and timely perform fault detection at the business layer and the infrastructure layer, and handle the fault based on the detection results without human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; and it can also avoid long-term interruption of training tasks and waste of computing resources.

[0111] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; the fault handling method based on the large model of this embodiment, in the above Figure 1 Based on the technical solution of the embodiment shown, the technical solution of the present disclosure is further and more effectively described by taking the task failure detected at the business layer as an example. Figure 3 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0112] S301. During the execution of the model training task, whether the performance of the task is reduced is detected at the business layer. If so, step S302 is executed; otherwise, the detection is continued;

[0113] For example, when step S301 is specifically implemented, it may include the following steps:

[0114] (a5) During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles;

[0115] (b5) collecting an average value of the number of tokens processed per second by at least one graphics processor executing a task in an iteration cycle as a performance indicator of the corresponding iteration cycle;

[0116] Specifically, for each iteration cycle, the at least one graphics processor selected to execute the task can be determined according to a pre-set rule. Preferably, when this step is implemented, the average number of symbols (tokens) processed per second by all graphics processors executing tasks in each iteration cycle can be collected as the performance indicator of the corresponding iteration cycle.

[0117] (c5) Based on the performance index of the iteration cycle, detect whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest historical iteration cycle reaches a preset ratio threshold; if so, execute step (d5); otherwise, determine that no task failure is detected, and return to continue detection; for example, the preset ratio threshold of this embodiment can be set according to actual experience, such as 30%, 25% or other ratio values, which are not limited here.

[0118] (d5) Determine whether the performance of the task has degraded, i.e., determine whether the task has failed.

[0119] In actual applications, the performance degradation of a task is also a manifestation of task abnormality. In this embodiment, by detecting whether the performance of a task is degraded at the business layer, it is also possible to timely and accurately perceive whether an abnormality occurs during the execution of the business layer task.

[0120] It should be noted that in this embodiment, the task failure is determined when the performance of the task is detected to be degraded. In actual application scenarios, the task failure detection can be performed by using both step S301 of this embodiment and the above-mentioned step S302. Figure 2 In step S201 of the illustrated embodiment, as long as any one method detects a task failure, the task failure is determined; if the task failure cannot be determined after all methods are used for detection, subsequent methods are further used for further detection.

[0121] S302, collecting business feature information of the task;

[0122] For example, the time consumption of each stage in the last iteration cycle and the historical iteration cycle of the nearest neighbor may be specifically collected.

[0123] S303, based on the business characteristic information of the task, perform fault diagnosis on the task to obtain a fault diagnosis result;

[0124] For example, when the step S303 is specifically implemented, it may include the following steps:

[0125] (a6) for the same stage, detecting whether the time consumption of the last iteration cycle relative to the historical iteration cycle of the nearest neighbor increases by more than a preset ratio;

[0126] In this embodiment, at a certain stage, the time consumption of the last iteration cycle increases by more than a preset proportion relative to the time consumption of the historical iteration cycle of the nearest neighbor. Specifically, the time consumption of this stage in the last iteration cycle, relative to the time consumption of this stage in the historical iteration cycle of the nearest neighbor, increases by a proportion greater than or equal to a preset proportion.

[0127] (b6) in response to the time consumption of the last iteration cycle of the stage increasing by more than a preset proportion relative to the historical iteration cycle of the nearest neighbor, obtaining a GPU that executes the task of the stage as a faulty GPU, and a node where the faulty GPU is located as a faulty node;

[0128] (c6) In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycles of the nearest neighbor does not increase by a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0129] By adopting the above steps (a6)-(c6), fault diagnosis can be performed accurately and effectively.

[0130] S304, checking whether the fault diagnosis result can determine the faulty node, if not, executing step S305; if yes, executing step S313;

[0131] S305, collecting the communication times of each process in each communication group executing the task; executing step S306;

[0132] S306, based on the communication times of each process in each communication group executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; execute step S307;

[0133] S307, checking whether the fault diagnosis result can determine the faulty node, if not, executing step S308; if yes, executing step S313;

[0134] S308, collecting call stack information of each GPU executing the task; executing step S309;

[0135] S309, based on the call stack information of each GPU executing the task, perform fault diagnosis on the task to obtain a fault diagnosis result; execute step S310;

[0136] S310, further detecting whether the fault diagnosis result can determine the faulty node, if not, executing step S311; if yes, executing step S313;

[0137] S311, detecting whether the infrastructure layer of each node executing the task fails; if so, executing step S312; otherwise, executing step S316;

[0138] S312, in response to the existence of a node with a fault in the infrastructure layer, updating the diagnosis result to identify the node with a fault in the infrastructure layer as a faulty node in the diagnosis result; executing step S313;

[0139] S313, shielding the faulty node; executing step S314;

[0140] S314, configuring a replacement node for the failed node; executing step S315;

[0141] S315: Reschedule the tasks on the failed node on the replacement node, and end.

[0142] S316, restart the task; execute step S317;

[0143] S317: If the task still fails after restarting, reschedule the task.

[0144] The implementation of the above steps S305-S317 can refer to the above Figure 2Steps S202 - S214 of the illustrated embodiment are not described in detail here.

[0145] The large model-based fault handling method of this embodiment adopts the above-mentioned scheme, which can comprehensively, accurately and timely perform fault detection at the business layer and the infrastructure layer, and handle the fault based on the detection results without human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; and it can also avoid long-term interruption of training tasks and waste of computing resources.

[0146] Figure 4 is a schematic diagram of the fourth embodiment of the present disclosure; the fault handling method based on the large model of the present embodiment takes fault detection during abnormal exit as an example to further and more effectively describe the technical solution of the present disclosure. Figure 4 As shown, the fault handling method based on the large model of this embodiment may specifically include the following steps:

[0147] S401, if the task is abnormally exited during execution, obtain the log of each GPU executing the task;

[0148] S402, performing fault diagnosis based on the logs of each GPU to obtain a fault diagnosis result;

[0149] For example, fault diagnosis may be performed based on the logs of each GPU to identify the faulty GPU, and to determine that the node where the faulty GPU is located is the faulty node.

[0150] In actual application scenarios, when an abnormal exit occurs, the log can carry fault information, for example, it can include two situations: calculation abnormality and communication abnormality.

[0151] For computing exceptions: Usually, there are exceptions of the CUDA Runtime Error type thrown in the log. You can configure the CudaRuntime Detect Agent component to detect and analyze the error patterns of various CUDA Runtimes to determine whether there are computing exceptions.

[0152] For communication anomalies, similar to computing anomalies, the NVIDIA Collective Communication Library (NCCL) communication library usually reflects the error content in the log. You can configure the NCCLDetect Agent component to accumulate and precipitate common failure patterns of NCCL to detect communication anomalies.

[0153] Based on the logs of each GPU, it is possible to diagnose whether a computing anomaly or a communication anomaly occurs. It should be noted that if a computing anomaly is detected in the log of a GPU, the GPU is determined to be a faulty GPU. If a communication anomaly is detected in the log of a GPU, it will be further identified whether the communication anomaly refers to a current-end communication anomaly, a peer-end communication anomaly, or whether it is impossible to determine which end has a communication anomaly. If the log indicates that the current-end communication anomaly is present, the GPU on the current end can be considered a faulty GPU; if the log indicates that the peer-end communication anomaly is present, the GPU on the peer-end can be considered a faulty GPU; if the log indicates that it is impossible to determine which end has a communication anomaly, it can be considered that the faulty GPU cannot be determined, or both GPUs at both ends can be considered faulty GPUs at the same time.

[0154] S403, checking whether the fault diagnosis result can determine the faulty node, if not, executing step S404; if yes, executing step S406;

[0155] S404, detecting whether the infrastructure layer of each node executing the task fails; if so, executing step S405; otherwise, executing step S409;

[0156] S405. In response to the existence of a node with a fault in the infrastructure layer, update the diagnosis result to identify the node with a fault in the infrastructure layer as a faulty node in the diagnosis result; execute step S406;

[0157] S406, shielding the faulty node; executing step S407;

[0158] S407, configuring a replacement node for the failed node; executing step S408;

[0159] S408: Restart the task and reschedule the task on the failed node on the replacement node, and end.

[0160] S409, restart the task; execute step S410;

[0161] S410: If the task still exits abnormally after restarting, reschedule the task.

[0162] The implementation of the above steps S404-S410 can refer to the above Figure 2 Steps S208 - S214 of the illustrated embodiment are not described in detail here.

[0163] The large model-based fault handling method of this embodiment adopts the above-mentioned scheme, and can comprehensively, accurately and timely perform fault detection at the framework layer and the infrastructure layer when the task exits abnormally, and handle the fault based on the detection result without human intervention, which can effectively improve the fault handling efficiency and thus effectively improve the model training efficiency; and it can also avoid long-term interruption of training tasks and waste of computing resources.

[0164] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 5 As shown, this embodiment provides a fault handling device 500 based on a large model, including:

[0165] A fault detection module 501 is used to perform fault detection on the task at the business layer during the execution of the model training task;

[0166] A fault diagnosis module 502, configured to perform fault diagnosis on the task in response to detecting the task fault at the business layer, and obtain a fault diagnosis result;

[0167] The fault processing module 503 is used to perform fault processing based on the fault diagnosis result.

[0168] The large model-based fault handling device 500 of this embodiment implements the fault handling implementation principle and technical effect by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0169] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure; Figure 6 As shown, the fault handling device 600 based on the large model of this embodiment, in the above Figure 5 Based on the technical solutions of the illustrated embodiments, the technical solutions of the present disclosure are further described in more detail. Figure 6 As shown, the fault handling device 600 based on the large model of this embodiment includes the above Figure 5 The modules with the same name and function are shown as follows: a fault detection module 601 , a fault diagnosis module 602 , and a fault processing module 603 .

[0170] In this embodiment, the fault diagnosis module 602 is used to:

[0171] The communication times of the processes in the communication group executing the task are collected; specifically, the communication times of each process in the multiple processes in each communication group in the multiple communication groups executing the task may be collected.

[0172] Based on the communication times of the processes in the communication group executing the task, the task is diagnosed to obtain a fault diagnosis result. Specifically, based on the communication times of each of the multiple processes in each of the multiple communication groups executing the task, the task is diagnosed to obtain a fault diagnosis result.

[0173] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0174] For each of the communication groups, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group;

[0175] Detect whether the graphics processors corresponding to the faulty processes in different communication groups are the same;

[0176] In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node;

[0177] In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0178] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0179] Collecting call stack information of each graphics processor executing the task;

[0180] Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

[0181] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602′ is used to:

[0182] Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors;

[0183] In response to the existence of a graphics processor with call stack information different from that of other graphics processors, determining that the graphics processor with call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node;

[0184] In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

[0185] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602′ is used to:

[0186] In response to detecting the task failure at the service layer, collecting service feature information of the task;

[0187] Based on the business feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

[0188] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is used to:

[0189] During the execution of the model training task, it is detected at the business layer whether the task is completed.

[0190] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is used to:

[0191] During the execution of the model training task, detecting at the business layer whether the utilization rate of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold;

[0192] If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

[0193] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0194] In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

[0195] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0196] Based on the utilization of each graphics processor of the task in a preset time period before the current moment, detecting whether there is a utilization equal to the second preset threshold;

[0197] In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node;

[0198] In response to the absence of a utilization ratio equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

[0199] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is used to:

[0200] During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment;

[0201] If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

[0202] Further optionally, in one embodiment of the present disclosure, the fault detection module 601 is used to:

[0203] During the execution of the model training task, whether the performance of the task has declined is detected at the business layer.

[0204] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0205] During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles;

[0206] An average value of the number of symbols processed per second by at least one graphics processor executing the task within the iteration cycle is collected as a performance indicator of the corresponding iteration cycle; specifically, for each iteration cycle, an average value of the number of symbols processed per second by all graphics processors executing the task within the iteration cycle can be collected as the performance indicator of the corresponding iteration cycle.

[0207] Based on the performance indicators of each iteration cycle, detect whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold;

[0208] If so, it is determined that the performance of the task is degraded, that is, the task is determined to be faulty.

[0209] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0210] In response to detecting the task failure at the business layer, the time consumption of each stage in the last iteration cycle and the nearest neighbor historical iteration cycle is collected.

[0211] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0212] For the same stage, check whether the time consumption of the last iteration cycle relative to the historical iteration cycle of the nearest neighbor increases by more than a preset ratio;

[0213] In response to the time consumption of the last iteration cycle of the stage increasing by more than a preset proportion relative to the historical iteration cycle of the nearest neighbor, obtaining a graphics processor that executes the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node;

[0214] In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycle of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

[0215] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is further configured to:

[0216] If the faulty node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task fails;

[0217] In response to the existence of a node where a fault occurs at the infrastructure layer, the diagnosis result is updated to identify the node where a fault occurs at the infrastructure layer as a faulty node in the diagnosis result.

[0218] Further optionally, in an embodiment of the present disclosure, the fault diagnosis module 602 is configured to perform at least one of the following:

[0219] Detecting whether kernel abnormality occurs in each node executing the task;

[0220] Detecting whether a system component abnormality occurs at each node executing the task; or

[0221] Detect whether each node executing the task has a hardware failure.

[0222] Further optionally, if Figure 6 As shown, in one embodiment of the present disclosure, the fault handling device 600 further includes:

[0223] The log acquisition module 604 is used to acquire the logs of each graphics processor executing the task if the task is abnormally exited during execution;

[0224] The fault diagnosis module 602 is further configured to perform fault diagnosis based on the logs of each of the graphics processors, and determine that the fault diagnosis result is that the faulty node cannot be determined.

[0225] Further optionally, in one embodiment of the present disclosure, the fault diagnosis module 602 is used to:

[0226] Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined as the faulty node.

[0227] Further optionally, if Figure 6 As shown, in one embodiment of the present disclosure, the fault processing module 603 includes:

[0228] A shielding unit 6031 is used to shield a faulty node when a faulty node is identified in the fault diagnosis result;

[0229] A configuration unit 6032 is used to configure a replacement node for the failed node;

[0230] The node rescheduling unit 6033 is used to reschedule the tasks on the failed node on the replacement node.

[0231] Further optionally, if Figure 6 As shown, in one embodiment of the present disclosure, the fault processing module 603 further includes:

[0232] A restart unit 6034, configured to restart the task when the fault diagnosis result indicates that the fault node cannot be determined;

[0233] The task rescheduling unit 6035 is further used to reschedule the task if the task still fails after restarting.

[0234] The large model-based fault handling device 600 of this embodiment implements the fault handling implementation principle and technical effect by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0235] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0236] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0237] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0238] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0239] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0240] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the above-mentioned methods of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0241] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0242] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0243] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0244] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0245] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0246] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0247] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0248] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A fault handling method based on a large model, comprising: During the execution of the model training task, fault detection is performed on the task at the business layer; In response to detecting the task failure at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result; Based on the fault diagnosis result, fault processing is performed.

2. The method according to claim 1, wherein: In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result includes: Collect the communication times of processes in the communication group that execute the task; Based on the communication times of the processes in the communication group that executes the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

3. The method according to claim 2, wherein: Based on the communication times of the processes in the communication group that executes the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result, including: For the communication group, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group; Detect whether the graphics processors corresponding to the faulty processes in different communication groups are the same; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

4. The method according to claim 1, wherein: In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result includes: Collecting call stack information of each graphics processor executing the task; Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

5. The method according to claim 4, wherein: Based on the call stack information of each graphics processor executing the task, the task is diagnosed for a fault, and a fault diagnosis result is obtained, including: Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors; In response to the existence of a graphics processor with call stack information different from that of other graphics processors, determining that the graphics processor with call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

6. The method according to claim 1, wherein: In response to detecting the task fault at the business layer, performing fault diagnosis on the task to obtain a fault diagnosis result includes: In response to detecting the task failure at the service layer, collecting service feature information of the task; Based on the business feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

7. The method according to claim 6, wherein: During the execution of the model training task, fault detection is performed on the task at the business layer, including: During the execution of the model training task, it is detected at the business layer whether the task is completed.

8. The method according to claim 7, wherein: During the execution of the model training task, the business layer detects whether the task is successful, including: During the execution of the model training task, detecting at the business layer whether the utilization rate of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

9. The method according to claim 8, wherein: In response to detecting the task failure at the service layer, collecting service feature information of the task, including: In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

10. The method according to claim 9, wherein: Based on the service feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result, including: Based on the utilization of each graphics processor of the task in a preset time period before the current moment, detecting whether there is a utilization equal to the second preset threshold; In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node; In response to the absence of a utilization ratio equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

11. The method according to claim 7, wherein: During the execution of the model training task, the business layer detects whether the task is successful, including: During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

12. The method according to claim 6, wherein: During the execution of the model training task, fault detection is performed on the task at the business layer, including: During the execution of the model training task, whether the performance of the task has declined is detected at the business layer.

13. The method according to claim 12, wherein: During the execution of the model training task, the business layer detects whether the performance of the task has degraded, including: During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles; collecting an average value of the number of symbols processed per second by at least one graphics processor executing the task in the iteration period as a performance indicator of the corresponding iteration period; Based on the performance index of the iteration cycle, detecting whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold; If so, it is determined that the performance of the task is degraded, that is, the task is determined to be faulty.

14. The method according to claim 13, wherein: In response to detecting the task failure at the service layer, collecting service feature information of the task, including: In response to detecting the task failure at the business layer, the time consumed at each stage in the last iteration cycle and the nearest neighbor historical iteration cycles is collected.

15. The method according to claim 14, wherein: Based on the service feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result, including: For the same stage, check whether the time consumption of the last iteration cycle relative to the historical iteration cycle of the nearest neighbor increases by more than a preset ratio; In response to the time consumption of the last iteration cycle of the stage increasing by more than a preset proportion relative to the historical iteration cycle of the nearest neighbor, obtaining a graphics processor that executes the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycle of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

16. The method according to claim 2, 4 or 6, wherein: In response to detecting the task fault, performing fault diagnosis on the task to obtain a fault diagnosis result, further comprising: If the faulty node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task fails; In response to the existence of a node where a fault occurs at the infrastructure layer, the diagnosis result is updated to identify the node where a fault occurs at the infrastructure layer as a faulty node in the diagnosis result.

17. The method according to claim 16, wherein: Detecting whether a failure occurs at the infrastructure layer of each node executing the task includes at least one of the following: Detecting whether kernel abnormality occurs in each node executing the task; Detecting whether a system component abnormality occurs at each node executing the task; or Detect whether each node executing the task has a hardware failure.

18. The method according to claim 16, wherein: Before detecting whether a failure occurs at the infrastructure layer of each node executing the task, the method further includes: If the task is abnormally exited during execution, obtaining logs of each graphics processor executing the task; Based on the logs of each of the graphics processors, a fault diagnosis is performed, and it is determined that a faulty node cannot be determined as a result of the fault diagnosis.

19. The method according to claim 18, wherein: The method further comprises: Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined as the faulty node.

20. The method according to any one of claims 1-15, 17-19, wherein: Based on the fault diagnosis result, the fault processing is performed, including: When a faulty node is identified in the fault diagnosis result, shielding the faulty node; Configuring a replacement node for the failed node; The tasks on the failed node are rescheduled on the replacement node.

21. The method according to claim 20, wherein: Performing fault processing based on the fault diagnosis result also includes: When the fault diagnosis result indicates that the faulty node cannot be determined, restarting the task; If the task still fails after restarting, the task is rescheduled.

22. A fault handling device based on a large model, comprising: A fault detection module is used to perform fault detection on the task at the business layer during the execution of the model training task; A fault diagnosis module, configured to perform fault diagnosis on the task in response to detecting the task fault at the service layer, and obtain a fault diagnosis result; The fault processing module is used to perform fault processing based on the fault diagnosis result.

23. The device according to claim 22, wherein: The fault diagnosis module is used for: Collect the communication times of processes in the communication group that execute the task; Based on the communication times of the processes in the communication group that executes the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

24. The device according to claim 23, wherein: The fault diagnosis module is used for: For the communication group, obtaining a faulty process whose communication times are less than the communication times of other processes in the communication group; Detect whether the graphics processors corresponding to the faulty processes in different communication groups are the same; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are the same, determining that the graphics processors corresponding to the faulty processes in different communication groups are the faulty graphics processors, and the node where the faulty graphics processor is located is the faulty node; In response to the fact that the graphics processors corresponding to the faulty processes in different communication groups are not the same, it is determined that the diagnosis result is that the faulty node cannot be determined.

25. The device according to claim 22, wherein: The fault diagnosis module is used for: Collecting call stack information of each graphics processor executing the task; Based on the call stack information of each graphics processor executing the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

26. The device according to claim 25, wherein The fault diagnosis module is used for: Based on the call stack information of each graphics processor, detecting whether there is a graphics processor with different call stack information from other graphics processors; In response to the existence of a graphics processor with call stack information different from that of other graphics processors, determining that the graphics processor with call stack information different from that of other graphics processors is a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the absence of a graphics processor having call stack information different from that of other graphics processors, it is determined that the diagnosis result is that a faulty node cannot be determined.

27. The device according to claim 22, wherein: The fault diagnosis module is used for: In response to detecting the task failure at the service layer, collecting service feature information of the task; Based on the business feature information of the task, a fault diagnosis is performed on the task to obtain a fault diagnosis result.

28. The device according to claim 27, wherein The fault detection module is used to: During the execution of the model training task, it is detected at the business layer whether the task is completed.

29. The device according to claim 28, wherein The fault detection module is used to: During the execution of the model training task, detecting at the business layer whether the utilization rate of each graphics processor executing the task has continuously reached the first preset threshold or the second preset threshold within a preset time period before the current moment, and whether the bandwidth has continuously reached the third preset threshold; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

30. The device according to claim 29, wherein: The fault diagnosis module is used for: In response to detecting the task failure at the service layer, the utilization rate of each graphics processor executing the task within a preset time period before the current moment is collected.

31. The device according to claim 30, wherein The fault diagnosis module is used for: Based on the utilization of each graphics processor of the task in a preset time period before the current moment, detecting whether there is a utilization equal to the second preset threshold; In response to the existence of a utilization rate equal to the second preset threshold, determining that the graphics processor corresponding to the utilization rate equal to the second preset threshold is a faulty graphics processor, and the node where the faulty graphics processor is located is a faulty node; In response to the absence of a utilization ratio equal to the second preset threshold, it is determined that the diagnosis result is that a faulty node cannot be determined.

32. The apparatus of claim 28, wherein: The fault detection module is used to: During the execution of the model training task, the business layer detects whether the logs of each graphics processor executing the task have not been refreshed within a preset time period before the current moment; If so, it is determined that the task is stuck, that is, it is determined that the task has failed.

33. The apparatus of claim 27, wherein: The fault detection module is used to: During the execution of the model training task, whether the performance of the task has declined is detected at the business layer.

34. The device according to claim 33, wherein: The fault diagnosis module is used for: During the execution of the model training task, the task is divided according to the training iteration cycle of the model before the current moment to obtain multiple iteration cycles; collecting an average value of the number of symbols processed per second by at least one graphics processor executing the task in the iteration period as a performance indicator of the corresponding iteration period; Based on the performance index of the iteration cycle, detecting whether the decline rate of the performance of the last iteration cycle before the current moment relative to the performance of the nearest neighbor historical iteration cycle reaches a preset ratio threshold; If so, it is determined that the performance of the task is degraded, that is, the task is determined to be faulty.

35. The device according to claim 34, wherein The fault diagnosis module is used for: In response to detecting the task failure at the business layer, the time consumed at each stage in the last iteration cycle and the nearest neighbor historical iteration cycles is collected.

36. The device according to claim 35, wherein The fault diagnosis module is used for: For the same stage, check whether the time consumption of the last iteration cycle relative to the historical iteration cycle of the nearest neighbor increases by more than a preset ratio; In response to the time consumption of the last iteration cycle of the stage increasing by more than a preset proportion relative to the historical iteration cycle of the nearest neighbor, obtaining a graphics processor that executes the task of the stage as a faulty graphics processor, and a node where the faulty graphics processor is located is a faulty node; In response to the fact that the time consumption of the last iteration cycle of the stage relative to the historical iteration cycle of the nearest neighbor does not increase by more than a preset proportion, it is determined that the diagnosis result is that the faulty node cannot be determined.

37. The apparatus of claim 23, 25 or 27, wherein: The fault diagnosis module is also used for: If the faulty node cannot be determined in the fault diagnosis result, detecting whether the infrastructure layer of each node executing the task fails; In response to the existence of a node where a fault occurs at the infrastructure layer, the diagnosis result is updated to identify the node where a fault occurs at the infrastructure layer as a faulty node in the diagnosis result.

38. The device according to claim 37, wherein The fault diagnosis module is used to perform at least one of the following: Detecting whether kernel abnormality occurs in each node executing the task; Detecting whether system component anomalies occur in each node executing the task; or Detect whether each node executing the task has a hardware failure.

39. The device according to claim 37, wherein: The device also includes: A log acquisition module, used for acquiring the logs of each graphics processor executing the task if the task is abnormally exited during execution; The fault diagnosis module is further used to perform fault diagnosis based on the logs of each of the graphics processors, and determine that the fault diagnosis result is that the fault node cannot be determined.

40. The device according to claim 39, wherein The fault diagnosis module is used for: Based on the logs of the graphics processors, fault diagnosis is performed to determine the faulty graphics processor, and the node where the faulty graphics processor is located is determined as the faulty node.

41. The device according to any one of claims 22-36, 38-40, wherein: The fault processing module comprises: A shielding unit, configured to shield a faulty node when a faulty node is identified in the fault diagnosis result; A configuration unit, configured to configure a replacement node for the failed node; A node rescheduling unit is used to reschedule the tasks on the failed node on the replacement node.

42. The device according to claim 41, wherein The fault processing module further includes: A restart unit, used to restart the task when the fault node cannot be determined in the fault diagnosis result; The task rescheduling unit is further used to reschedule the task if the task still fails after restarting.

43. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 21.

44. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-21.

45. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 21.

Images