Fault positioning method, device and equipment and computer readable storage medium

Abstract

The invention relates to a fault positioning method, device and equipment and a computer readable storage medium, and the method comprises the steps: carrying out the health detection of a target computing power node in at least two computing power nodes under the condition of receiving a model training request, the health detection comprises at least one of arithmetic unit detection, communication link detection and model training detection; under the condition that the detection result of the health detection represents that the target computing power node has no fault, responding to the model training request, and executing a model training task through the target computing power node; under the condition that the model training task is interrupted, the fault computing power node in the target computing power node is positioned according to a training log, and the training log is a log output in the process of executing the model training task. According to the invention, faults occurring before and in the process of executing the model training task by the target computing power node can be positioned, and the accuracy of fault positioning is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a fault positioning method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] With the increasing demand for model training and inference, more and more cloud vendors begin to choose to provide services for intelligent computing business customers in the form of platform as a service (PAAS), that is, to build a computing power platform for model training based on graphics processing unit (GPU) computing power.

[0003] Taking large model training as an example, when the computing power platform is abnormal due to abnormal underlying computing power resource hardware or network during large model training based on the PAAS computing power platform, the training task will be interrupted. In the related technology, when the model training task is interrupted during the process, the cause of the interruption of the training task is analyzed, and there is a case of failure of fault positioning, and the accuracy of fault positioning is low. SUMMARY

[0004] In order to solve the above technical problems, the present disclosure provides a fault positioning method, device, equipment and computer readable storage medium to solve the problem of low accuracy of fault positioning when the computing power platform executes the model training task and the training is interrupted.

[0005] In a first aspect, the embodiments of the present disclosure provide a fault positioning method applied to a computing power platform, the computing power platform comprising at least two computing power nodes, each computing power node of the at least two computing power nodes being deployed with at least one operation unit, and the fault positioning method comprising: In the case of receiving a model training request, performing health detection on a target computing power node in the at least two computing power nodes, wherein the health detection comprises at least one of operation unit detection, communication link detection and model training detection, the operation unit detection is used to detect the operation ability of each operation unit in the target computing power node, the communication link detection is used to detect the communication ability of each operation unit in the target computing power node, and the model training detection is used to detect the model training ability of the target computing power node; in the case that the detection result of the health detection represents that the target computing power node is fault-free, responding to the model training request, performing a model training task through the target computing power node; in the case that the model training task is interrupted, positioning a fault computing power node in the target computing power node according to a training log, the training log being a log output during the execution of the model training.

[0006] In some optional embodiments, performing health detection on a target computing power node in the at least two computing power nodes comprises: The operation unit detection, the communication link detection, and the model training detection are sequentially performed on the target computing power node.

[0007] In some optional embodiments, in the case of interruption of the model training task, after locating the faulty computing power node in the target computing power node according to the training log, the fault locating method further comprises: In the case of locating the faulty computing power node, outputting a fault prompt information, the fault prompt information being used to prompt processing of the faulty computing power node; or, in the case of not locating the faulty computing power node, returning to perform the health detection processing on the computing power platform.

[0008] In some optional embodiments, the operation unit detection comprises: For each operation unit of the at least two operation units in the target computing power node, inputting a preset matrix to the operation unit, and obtaining a floating point calculation throughput of the operation unit; controlling the operation unit to load a kernel function, and obtaining a loading duration of the kernel function; in the case that the floating point calculation throughput is greater than a throughput threshold, and the loading duration is less than a duration threshold, determining that the operation ability of the operation unit is in a non-faulty state; or, in the case that the floating point calculation throughput is less than or equal to the throughput threshold, and / or the loading duration is greater than or equal to the duration threshold, determining that the operation ability of the operation unit is in a faulty state.

[0009] In some optional embodiments, the communication link detection comprises: For each operation unit of the at least two operation units in the target computing power node, performing a communication test, and determining a test result of the operation unit; wherein the communication test comprises at least one of the following: full-reduction communication test, reduction-divergence communication test, full-collection communication test, and full-exchange communication test.

[0010] In some optional embodiments, the model training test comprises: Training a preset framework model through the target computing power node, and obtaining a floating point operation per second of the target computing power node and a model floating point operation utilization rate; in the case that the floating point operation per second is greater than a number threshold, and the model floating point operation utilization rate is greater than a utilization rate threshold, determining that the model training ability of the target computing power node is in a non-faulty state; or, in the case that the floating point operation per second is less than or equal to the number threshold, and / or the model floating point operation utilization rate is greater than or equal to the utilization rate threshold, determining that the model training ability of the target computing power node is in a faulty state.

[0011] In some optional embodiments, in the case of receiving the model training request, after performing the health detection processing on the target computing power node of the at least two computing power nodes, the fault locating method further comprises: In a case where the detection result of the health detection processing indicates that the target computing power node is faulty, a fault prompt information is output, and the fault prompt information is used to prompt processing of the faulty computing power node; wherein the fault prompt information includes at least one of the following: identification information of the faulty computing power node, identification information of a faulty operation unit in the faulty computing power node, running capability fault information, communication capability fault information, and model training capability fault information.

[0012] In a second aspect, the embodiments of the present disclosure provide a fault locating device, applied to a computing power platform, the computing power platform including at least two computing power nodes, each of the at least two computing power nodes being deployed with at least one operation unit, and the fault locating device including: a detection module, configured to, in a case where a model training request is received, perform health detection processing on a target computing power node in the at least two computing power nodes, wherein the health detection processing includes at least one of the following: operation unit detection, communication link detection, and model training detection, the operation unit detection being used to detect operation capability of each operation unit in the target computing power node, the communication link detection being used to detect communication capability of each operation unit in the target computing power node, and the model training detection being used to detect model training capability of the target computing power node; a training module, configured to, in a case where a detection result of the health detection processing indicates that the target computing power node is not faulty, perform a model training task through the target computing power node in response to the model training request; and a locating module, configured to, in a case where the model training task is interrupted, locate a faulty computing power node in the target computing power node according to a training log, the training log being a log output in a process of performing the model training task.

[0013] In a third aspect, the embodiments of the present disclosure provide an electronic device, including: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the fault locating method as described in the first aspect.

[0014] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium, having a computer program stored thereon, the computer program being executed by a processor to implement the fault locating method as described in the first aspect.

[0015] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the fault locating method as described above.

[0016] The method, device, equipment and computer readable storage medium provided by the embodiments of the present disclosure can perform active health detection on the target computing power nodes participating in training in the computing power platform before starting to execute the model training task of the model training request when the computing power platform receives the model training request, so as to detect whether the target computing power nodes in the computing power platform are abnormal or faulty before starting to execute the model training task. The model training task corresponding to the model training request is performed only when the detection result of the health detection represents that all the target computing power nodes are normal. The probability of training abnormal interruption in the model training process can be significantly reduced. In the case of training interruption in the model training process, the passive method of training log analysis is used to locate the faulty computing power nodes in the abnormal state, and the abnormality or fault of the target computing power nodes in the process of executing the model training task is located. The present application performs active health detection before the computing power platform starts to execute the model training task, and performs training log analysis when the model training task is interrupted, so as to locate the fault of the target computing power nodes before starting to execute the model training task and in the process of executing the task, and improve the accuracy of fault location. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure together with the specification.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0019] Figure 1 A schematic diagram of an application scenario provided by the embodiments of the present disclosure; Figure 2 A flowchart of a fault location method provided by the embodiments of the present disclosure; Figure 3 A flowchart of a fault location method provided by another embodiment of the present disclosure; Figure 4 A structural schematic diagram of a fault location device provided by the embodiments of the present disclosure; Figure 5 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] In order to enable a more clear understanding of the above-mentioned purposes, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0021] In the following description, many specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure can also be practiced without the specific details that are described herein; obviously, the embodiments described in the specification are only a part of the embodiments of the present disclosure, not all the embodiments.

[0022] The embodiments of the present disclosure provide a fault locating method, which will be introduced below in combination with specific embodiments.

[0023] Figure 1 A schematic diagram of an application scenario provided by the embodiments of the present disclosure is shown in Figure 1 The computing power platform 100 in the application scenario includes at least two computing power nodes 110, and each computing power node 110 includes at least one operation unit 111. Exemplarily, the computing power platform 100 can be a PSSA computing power platform.

[0024] When the computing power platform 100 performs a large model training task, the target computing power node required by the large model training task can be automatically selected from the at least two computing power nodes 110, and the target computing power node is a container participating in the distributed training task. It can be understood that the embodiments of the present disclosure are introduced by taking the model as a large model.

[0025] Exemplarily, each computing power node 110 can be a physical server, and each operation unit 111 can be a GPU (graphics processing unit) deployed on a physical server. It can be understood that the fault locating method provided by the embodiments of the present disclosure can also be applied to other scenarios.

[0026] Figure 2 A flowchart of the fault locating method provided by the embodiments of the present disclosure is shown in Figure 1 The fault locating method can be applied to the application scenario shown in Figure 1 The fault locating method will be introduced below in combination with the application scenario shown in Figure 2 The fault locating method provided by the embodiments of the present disclosure is applied to a computing power platform, the computing power platform includes at least two computing power nodes, and each computing power node of the at least two computing power nodes is deployed with at least one operation unit. The specific steps of the fault locating method include the following: S201, in the case of receiving a model training request, performing health detection on a target computing power node in the at least two computing power nodes.

[0027] The health detection includes at least one of operation unit detection, communication link detection and model training detection. The operation unit detection is used to detect the operation capability of each operation unit in the target computing power node. The communication link detection is used to detect the communication capability of each operation unit in the target computing power node. The model training detection is used to detect the model training capability of the target computing power node.

[0028] In this embodiment, the model training request is used to call the computing power resources in the computing power platform to train the model. For example, the computing power platform can train a large language model (LLM).

[0029] In this embodiment, when the computing power platform receives the model training request, the target computing power node to be called in at least two computing power nodes in the computing power platform can be determined according to the model training request. The number of target nodes is at least one.

[0030] For example, the computing power platform performs task demand analysis on the model training task corresponding to the model training request, and determines the computing resources, storage space, network bandwidth and the like required to execute the model training task. The computing power platform evaluates the computing power node resources, and determines the real-time computing power (for example, GPU utilization and the like), available memory, current load and the like in each computing power node. Then, the computing power platform performs computing power resource scheduling based on the task demand analysis result and the computing power node resource evaluation result, and determines the target nodes to be called in the plurality of computing power nodes.

[0031] In this embodiment, after the computing power platform determines the target computing power nodes to be called, the target computing power nodes need to be health detected before starting to execute the model training task by the target computing power nodes, so as to determine whether the target computing power nodes for executing the model training task are in a healthy state.

[0032] Specifically, the operation unit detection in the health detection is to detect the operation capability of at least one operation unit in each target computing power node, and to determine whether the operation capability of each operation unit meets the demand of the model training task. The communication link detection in the health detection is to detect the communication capability of at least one operation unit in each target computing power node, and to determine whether the communication capability of each operation unit meets the demand of the model training task. The model training detection in the health detection is to detect the target computing power node, to train a preset model by the target computing power node, and to determine whether the model training capability of the target computing power node meets the demand of the model training task.

[0033] S202, in the case that the detection result of the health detection represents that the target computing power node is fault-free, the model training task is executed by the target computing power node in response to the model training request.

[0034] In this embodiment, when each health detection performed by the computing power platform meets the requirements of the model training task, the detection result of the health detection indicates that each target computing power node is fault-free.

[0035] For example, when the health detection includes operation unit detection, communication link detection, and model training detection, if the operation unit detection result, the communication link detection result, and the model training detection result all meet the requirements of the model training task, it is determined that each target computing power node required to perform the model training task is fault-free; if at least one of the operation unit detection result, the communication link detection result, and the model training detection result does not meet the requirements of the model training task, it is determined that the target computing power node required to perform the model training task has a fault.

[0036] In this embodiment, when the detection result indicates that the target computing power node is fault-free, the computing power platform determines that the hardware of the target computing power node can meet the requirements of the model training task before performing the model training task, and at this time, the computing power platform starts to perform the model training task. Through the above health detection, the model training task interruption caused by hardware failure before the computing power platform starts to perform the model training task can be reduced, and the stability of the target computing power node participating in the training is maximized.

[0037] S203, in the case of interruption of the model training task, locating the fault computing power node in the target computing power node according to a training log.

[0038] The training log is a log output in the process of performing the model training task.

[0039] In this embodiment, the computing power platform starts to perform the model training task, and the computing power platform can generate and output the training log in the process of performing the model training task. Since there is also a case of model training task interruption caused by hardware exception in actual distributed training, when the model training is interrupted, the fault computing power node in the target computing power node is located according to the training log output in the training process.

[0040] In this embodiment, the training log records the all-around information of the model training task in detail. For example, the training log includes a general training log, a log of a communication library monitoring and diagnosis tool, and a log of a distributed debugging tool. Specifically as follows: The general training log includes abnormal interruption information and stuck related logs. For example, scanning error information, warning information, and abnormal stack tracking in the training log to determine whether there is a fault computing power node or a code level error; when the training is stuck but not crashed, focusing on the progress index stagnation, i.e., the loss function is not updated for a long time, to determine whether there is a fault computing power node stuck or a communication timeout in the communication link.

[0041] The log of the communication library monitoring and diagnosis tool includes keep-alive, abnormal positioning, and communicator state tracking. The keep-alive monitoring is monitoring information that the communication library monitoring and diagnosis tool establishes a connection between all participating processes and exchanges alive signals periodically. The communicator state tracking is tracking information that the communication library monitoring and diagnosis tool can provide the global state of all communicators and their internal processes. The abnormal positioning log is related information that the communication library monitoring and diagnosis tool locates abnormal computing nodes.

[0042] The log of the distributed debugging tool includes function call stack records recorded by the tool continuously, communication group state, and communication summary analysis. When the model training task is stuck, by analyzing the call stack at the time of suspension, it can be seen which specific function the process is stuck in, so as to distinguish whether the cause of the stall is a computing problem or a communication problem. The communication group state and communication summary analysis can provide more detailed communication event sequences and state information than general logs.

[0043] In the embodiments of the present disclosure, when the computing platform receives a model training request, before starting to execute the model training task of the model training request, the target computing nodes participating in training in the computing platform are actively detected for health, so as to detect whether the target computing nodes in the computing platform have abnormalities or faults before starting to execute the model training task. When the detection result of the health detection indicates that all target computing nodes are normal, the model training task corresponding to the model training request is performed, which can significantly reduce the probability of training abnormal interruption in the model training process. When the training interruption occurs in the model training process, the passive method of training log analysis is used to locate the fault computing nodes in the abnormal state, and the abnormalities or faults of the target computing nodes occurring in the process of executing the model training task are located. The present application actively detects the health before the computing platform starts to execute the model training task, and performs training log analysis when the model training task is interrupted, so as to locate the faults occurring before the target computing nodes start to execute the model training task and in the process of executing the task, and improve the accuracy of fault location.

[0044] In some embodiments of the present disclosure, the health detection on the target computing node in the at least two computing nodes includes sequentially performing operation unit detection, communication link detection, and model training detection on the target computing node.

[0045] In this embodiment, in the process of performing health detection on the target computing power node in the computing power platform, detection needs to be performed in the order of operation unit detection, communication link detection, and model training detection. Among them, the detection purpose of model training detection is to judge whether the model training capability of the target computing power node meets the requirements of the model training task, the communication link detection is to judge whether the communication capability of the operation unit in the target computing power node meets the requirements of the model training task, and the operation unit detection is to judge whether the operation capability of the operation unit in the target computing power node meets the requirements of the model training task.

[0046] It can be understood that when the operation capability of the operation unit in the target computing power node does not meet the requirements of the model training task, it may cause the communication capability detected by the communication link detection to not meet the requirements, and it may cause the model training capability detected by the model training detection to not meet the requirements. When the communication capability of the operation unit in the target computing power node meets the requirements of the model training task, it does not affect the judgment of the operation capability of the operation unit, but it may cause the model training capability detected by the model training detection to not meet the requirements. When the operation capability and the communication capability of the operation unit in the target computing power node both meet the requirements, there is still a possibility that the model training capability of the target computing power node does not meet the requirements. If the upper-layer model training detection is performed first, if the case of not meeting the requirements occurs, it is not possible to accurately determine whether the model training capability does not meet the requirements due to insufficient operation capability or insufficient communication capability.

[0047] In the embodiments of the present disclosure, the health detection is performed in the order from bottom to top of operation unit detection, communication link detection, and model training detection, which can improve the health detection efficiency and reduce the possibility of repeated testing.

[0048] In some embodiments of the present disclosure, in the case of interruption of the model training task, after the fault computing power node in the target computing power node is located according to the training log, the fault locating method further comprises: In the case of locating the fault computing power node, outputting fault prompt information, the fault prompt information is used to prompt processing of the fault computing power node; or, in the case of not locating the fault computing power node, returning to perform health detection processing on the target computing power node.

[0049] In this embodiment, when the model training is started and the model training task is abnormally interrupted, the fault computing power node in the target computing power node is located according to the real-time output training log.

[0050] In the case of successful positioning, the fault computing power node in the target computing power node can be determined, at which time, the fault prompt information is output, and the fault prompt information can prompt the maintenance personnel of the computing power platform to process the fault computing power node.

[0051] Exemplarily, the fault prompt information includes prompt information prompting the user sending the model training request, and prompt information prompting the maintenance personnel of the computing power platform. It is judged whether the target computing power node has a hot backup machine. If it is judged that the computing power node does not have a hot backup machine, the computing power platform outputs the “training failure” prompt information to the user terminal. If it is judged that the computing power node has a hot backup machine, the computing power platform outputs the “replace the hot backup machine” prompt information or the “disconnect the target computing power node” prompt information to the maintenance terminal.

[0052] In this embodiment, since the computing power platform depends on the actual distributed training framework in the passive positioning fault mode of the training log, there is a possibility that the fault computing power node cannot be located or cannot be directly located. Therefore, when the model training task is in an interrupted state and the fault computing power node is not located, the computing power platform triggers the active health detection process again, so as to attempt to locate the fault again and further improve the efficiency of locating the abnormality occurring in the process of executing the model training task.

[0053] It should be noted that, after triggering the active health detection process again, if the fault computing power node is detected, the maintenance personnel processes the fault computing power node, and then returns to the process of performing the active health detection. If the health detection process after processing the fault computing power node can still locate the fault computing power node, the above process is repeatedly performed until the fault computing power node can no longer be located. If the health detection process after processing the fault computing power node can no longer locate the fault computing power node, the model training task is continuously executed until the model training is completed.

[0054] In the embodiments of the present disclosure, when the training model task is executed, the training interruption occurs, and the fault computing power node is attempted to be located through the training log. When the fault computing power node is located, the fault prompt information is output, so as to prompt the maintenance personnel to process the fault computing power node in time. When the fault computing power node is not located, the active health detection process of the computing power platform is returned to be executed, so as to locate the fault computing power node and reduce the waste of computing power caused by long-time interruption of the training task.

[0055] In some embodiments of the present disclosure, the operation unit detects, comprising: For each operation unit in the target computing power node, a preset matrix is input to the operation unit, and the floating point calculation throughput of the operation unit is obtained. The operation unit is controlled to load the kernel function, and the loading time of the kernel function is obtained. In the case that the floating point calculation throughput is greater than the throughput threshold, and the loading time is less than the time threshold, it is determined that the operation ability of the operation unit is in a fault-free state. Or, in the case that the floating point calculation throughput is less than or equal to the throughput threshold, and / or the loading time is greater than or equal to the time threshold, it is determined that the operation ability of the operation unit is in a fault state.

[0056] In this embodiment, the operation unit detects the operation capability of each operation unit, and therefore, operation unit detection needs to be performed on each operation unit in the target computing power node. The operation unit detection includes preset matrix operation detection and Kernel loading delay detection, which will be described in detail as follows: The preset matrix operation detection: whether the floating point calculation throughput of the operation unit when calculating the preset standard matrix reaches the throughput threshold is determined. Specifically, the preset matrix is input to the operation unit, and when the operation unit operates on the preset matrix, the floating point calculation throughput of the operation unit is obtained, and the floating point calculation throughput is compared with the throughput threshold. If the floating point calculation throughput is greater than the throughput threshold, it is determined that the operation unit passes the preset matrix operation detection; if the floating point calculation throughput is less than or equal to the throughput threshold, it is determined that the operation unit fails the preset matrix operation detection.

[0057] The Kernel loading delay detection: whether the loading time length of the budget unit loading the kernel function is less than the time length threshold is determined. Specifically, the operation unit loads the preset kernel function, and the starting time and completion time of the operation unit loading the kernel function are recorded, and the loading time length is determined according to the starting time and the completion time. If the loading time length is less than the time length threshold, it is determined that the operation unit passes the Kernel loading delay detection; if the loading time length is greater than or equal to the time length threshold, it is determined that the operation unit fails the Kernel loading delay detection.

[0058] In the process of detecting each operation unit in the target computing power node, if the operation unit passes the preset matrix operation detection and the Kernel loading delay detection, that is, the floating point calculation throughput is greater than the throughput threshold, and the loading time length is less than the time length threshold, it is determined that the operation capability of the operation unit is in a normal fault-free state; if the operation unit fails at least one of the preset matrix operation detection and the Kernel loading delay detection, that is, the floating point calculation throughput is less than or equal to the throughput threshold, and / or the loading time length is greater than or equal to the time length threshold, it is determined that the operation capability of the operation unit is in an abnormal fault state.

[0059] In the embodiments of the present disclosure, the preset matrix operation detection and the Kernel loading delay detection are performed on each operation unit in the target computing power node, and the operation unit with normal preset matrix operation detection and Kernel loading delay detection is determined as the operation unit in the fault-free state, and the operation unit with at least one of the preset matrix operation detection and the Kernel loading delay detection in the abnormal state is determined as the operation unit in the fault state, thereby improving the accuracy of detecting whether the operation capability of each operation unit meets the model training requirement.

[0060] In some embodiments of the present disclosure, the communication link detection includes: For each operation unit in the target computing power node, a communication test is performed, and a test result of the operation unit is determined; wherein the communication test includes at least one of the following: all-reduce communication test, reduce-scatter communication test, all-gather communication test, and all-to-all communication test.

[0061] In this embodiment, when the target computing power node is subjected to the communication link detection, the core detection target is whether the communication capability of all operation units participating in the distributed training of the model training task meets the requirements of the model training task. Therefore, it is necessary to perform a communication test on each operation unit, and the communication test includes at least one of the all-reduce communication test, the reduce-scatter communication test, the all-gather communication test, and the all-to-all communication test.

[0062] Exemplarily, the all-reduce communication test, the reduce-scatter communication test, the all-gather communication test, and the all-to-all communication test are four standard source language communication test tools of the set communication library. The all-reduce communication test, the reduce-scatter communication test, the all-gather communication test, and the all-to-all communication test are described as follows: All-reduce communication test: control all operation units to cooperatively complete a reduction operation, which can be a summation operation, a maximum value operation, etc. If all operation units obtain complete results, it is determined that all operation units pass the test.

[0063] Reduce-scatter communication test: each operation unit first performs a reduction operation, which can be a summation operation, a maximum value operation, etc., and then scatters the result to different operation units in blocks, and each operation unit finally obtains a part of the result, then it is determined that all units pass the test.

[0064] All-gather communication test: each operation unit provides a block of data, and finally all nodes collect the data of other nodes to form complete global data, then it is determined that all units pass the test.

[0065] All-to-all communication test: each operation unit sends its data block to all other operation units, and at the same time receives data blocks from all other operation units, then it is determined that all units pass the test.

[0066] Exemplarily, in the process of communication link detection, if it is detected that the full-reduction communication test, the reduction-divergence communication test, the full-collection communication test, and the full-exchange communication test of a certain operation unit all pass, it is determined that the communication capability of the operation unit is in a normal state; if it is detected that one of the full-reduction communication test, the reduction-divergence communication test, the full-collection communication test, and the full-exchange communication test of a certain operation unit does not pass, it is determined that the communication capability of the operation unit is in a fault state.

[0067] In the embodiments of the present disclosure, at least one of the full-reduction communication test, the reduction-divergence communication test, the full-collection communication test, and the full-exchange communication test is performed on each operation unit in the target computing power node, and whether the operation unit has a communication fault is determined according to the test result, thereby improving the detection efficiency and detection accuracy of the communication capability of the communication link.

[0068] In some embodiments of the present disclosure, the model training test comprises: The preset framework model is trained by the target computing power node, and the floating point operation number per second and the model floating point operation utilization rate of the target computing power node are obtained; in the case that the floating point operation number per second is greater than the number threshold and the model floating point operation utilization rate is greater than the utilization rate threshold, it is determined that the model training capability of the target computing power node is in a fault-free state; or, in the case that the floating point operation number per second is less than or equal to the number threshold, and / or the model floating point operation utilization rate is greater than or equal to the utilization rate threshold, it is determined that the model training capability of the target computing power node is in a fault state.

[0069] In this embodiment, when the target computing power node is tested for model training, the preset framework model needs to be trained by the target computing power node, so as to simulate the state of the target computing power node performing the model training task, and based on the floating point operation number per second and the model floating point operation utilization rate collected in the training process, it is determined whether the model training capability of the target computing power node meets the requirements of the model training task.

[0070] Specifically, when the target computing power node trains the preset framework model, the floating point operation number per second (TFLOPS, tera floating-point operations per second) and the model floating point budget utilization rate (MFU, Model FLOPS Utilization) are obtained in the training process, wherein the floating point operation number per second is used to measure the theoretical peak computing capability of the operation unit, and the efficiency of utilizing hardware theoretical computing power when measuring the preset framework model.

[0071] In a case where the number of floating point operations per second is greater than the number threshold and the model floating point operation utilization rate is greater than the utilization threshold, it is determined that the model training capability of the target computing power node can meet the demand of the model training task, i.e., in a fault-free state; in a case where the number of floating point operations per second is less than or equal to the number threshold and / or the model floating point operation utilization rate is greater than or equal to the utilization threshold, it is determined that the model training capability of the target computing power node cannot meet the demand of the model training task, i.e., in a fault state.

[0072] Exemplarily, the model framework of the preset framework model matches the model framework of the model training task, and the preset framework model includes a model based on Megatron (an open source deep learning framework).

[0073] In the embodiments of the present disclosure, by controlling the target computing power node to train the preset framework model, the simulation training of the model training task can be performed, the number of floating point operations per second and the model floating point operation utilization rate are obtained in the simulation training process, and whether the model training capability of the target computing power node meets the demand of the model training task is judged through the number of floating point operations per second and the model floating point operation utilization rate, thereby improving the accuracy of the evaluation of the model training capability of the target computing power node, so that the model training capability of the target computing power node can be evaluated in time before the target computing power node starts to execute the model task.

[0074] In some embodiments of the present disclosure, after performing the health detection processing on the target computing power node in the at least two computing power nodes in a case where the model training request is received, the fault locating method further comprises: In a case where the detection result of the health detection processing represents that the target computing power node is in a fault state, fault prompt information is output, and the fault prompt information is used to prompt the maintenance personnel to process the fault computing power node; wherein the fault prompt information comprises at least one of the following: identification information of the fault computing power node, identification information of the fault operation unit in the fault computing power node, running capability fault information, communication capability fault information, and model training capability fault information.

[0075] In this embodiment, when the detection result of the active health detection processing performed by the computing power platform represents that the target computing power node is in a fault state, fault prompt information is output, and the fault prompt information can prompt the maintenance personnel to process the fault computing power node.

[0076] Specifically, the maintenance personnel can locate the fault computing power node in the target computing power node based on the identification information of the fault computing power node; the maintenance personnel can locate the fault operation unit in the target computing power node through the identification information of the fault operation unit; and the maintenance personnel can determine the fault type of the fault computing power node or the fault operation unit according to the running capability fault information, the communication capability fault information, and the model training capability fault information, so as to facilitate the targeted processing by the maintenance personnel.

[0077] It should be noted that the fault prompt information output in the active health detection processing stage and the fault prompt information output in the passive training log analysis stage can be the same or different.

[0078] In the embodiments of the present disclosure, the fault prompt information includes identification information of a fault computing power node, and at least one of identification information of a fault operation unit in the fault computing power node, running capability fault information, communication capability fault information, and model training capability fault information. The fault prompt information can be used by maintenance personnel to locate the fault, and further plan a solution for processing the located fault, thereby improving the accuracy of fault location and simplifying the operation steps of the computing power platform maintenance.

[0079] Figure 3 A flowchart of a fault locating method provided by another embodiment of the present disclosure is shown in FIG. 3, the fault locating method is executed by a computing power platform, and the fault locating method includes the following steps. Figure 3 Step 301, receiving a model training request.

[0080] In this embodiment, the model training request is used to call the computing power resources in the computing power platform to train the model.

[0081] Step 302, selecting a target computing power node from a plurality of computing power nodes according to a scheduling strategy, and pulling training code and a dataset, configuring an environment, and checking dependencies through a running initialization container.

[0082] In this embodiment, the initialization container (init container) is a special container used to run before the start of the main application container, and is mainly used to perform initialization tasks to prepare the required environment or conditions for the start of the application container. When there are multiple initialization containers, the multiple initialization containers need to be executed in sequence, and the main container is started after successful completion.

[0083] Step 303, performing active health detection.

[0084] In this embodiment, the active health detection includes operation unit detection, communication link detection, and model training detection.

[0085] Further, the active health detection includes sequentially performing the operation unit detection, the communication link detection, and the model training detection.

[0086] Step 304, determining whether there is a fault computing power node, and if the result is yes, executing step 305, otherwise executing step 307.

[0087] Step 305, determining whether the standby machine is being replaced, and if the result is yes, executing step 306, otherwise executing step 312. ​

[0088] In this embodiment, the computing power platform determines whether the backup machine is being replaced. If the determination is no, it is determined that there is no backup machine in the current target node, and thus a training failure prompt information is directly output. If the determination is yes, it is determined that the maintenance personnel are replacing the hot backup machine, and a step of receiving a corresponding notification is performed.

[0089] In step 306, a hot backup machine replacement completion notification or a failed computing power node blocking notification is received, and step 302 is returned to be executed.

[0090] In this embodiment, a failure prompt information needs to be output in step 306 to prompt the maintenance personnel of the computing power platform to manually perform the hot backup machine replacement operation or perform the blocking operation of the failed computing power node. After the maintenance personnel manually perform the hot backup machine replacement operation or perform the blocking operation of the failed computing power node, the maintenance personnel notify the computing power platform that the processing of the failed computing power node is completed, and then step 302 is returned to be executed to perform the re-distribution of the target computing power node.

[0091] In step 307, the model training task is executed.

[0092] In step 308, it is determined whether the training is interrupted. If the determination is yes, step 309 is executed, otherwise, step 311 is executed.

[0093] In step 309, the training log analysis is performed to locate the failure.

[0094] In step 310, it is determined whether the failed computing power node is located. If the determination is yes, step 304 is executed, otherwise, step 303 is executed.

[0095] In step 311, the trained model is output.

[0096] In step 312, a training failure prompt information is output.

[0097] In the embodiments of the present disclosure, before the computing power platform starts to execute the model training task, active health detection is performed to locate the failed computing power node, so that the failed computing power node can be processed in time before the training task starts, and the probability of abnormal interruption in the subsequent execution of the model training task is reduced. If the training is interrupted during the execution of the model training task by the computing power platform, passive failure positioning can be performed through the training log. If the failure positioning based on the training log fails, the model training task is suspended, and the health detection is returned to be executed, so that the failed computing power node can be located in time, and the waste of computing power resources of the computing power platform caused by the interruption of the model training task is reduced.

[0098] Figure 4A structural schematic diagram of a fault locating apparatus provided by an embodiment of the present disclosure. The fault locating apparatus can be a to-be-upgraded device as described in the above embodiments, or the fault locating apparatus can be a component or assembly in the to-be-upgraded device. The apparatus provided by an embodiment of the present disclosure can perform the processing flow provided by the fault locating method embodiment, as shown in Figure 4 As shown in FIG. 4, the fault locating apparatus 40 is applied to a computing power platform, the computing power platform includes at least two computing power nodes, each of the at least two computing power nodes is deployed with at least one operation unit, and the fault locating apparatus 40 includes a detection module 41, a training module 42, and a locating module 43. The detection module 41 is configured to, in response to receiving a model training request, perform health detection processing on a target computing power node in the at least two computing power nodes, where the health detection processing includes at least one of operation unit detection, communication link detection, and model training detection, the operation unit detection is configured to detect operation capability of each operation unit in the target computing power node, the communication link detection is configured to detect communication capability of each operation unit in the target computing power node, and the model training detection is configured to detect model training capability of the target computing power node.

[0099] The training module 42 is configured to, in response to the model training request, perform a model training task by the target computing power node, in a case where a detection result of the health detection processing indicates that the target computing power node is fault-free.

[0100] The locating module 43 is configured to, in a case where the model training task is interrupted, locate a fault computing power node in the target computing power node according to a training log, the training log being a log output in a process of performing the model training task.

[0101] In some embodiments of the present disclosure, the detection module 41 is specifically configured to sequentially perform the operation unit detection, the communication link detection, and the model training detection on the target computing power node.

[0102] In some embodiments of the present disclosure, the fault locating apparatus 40 further includes: An output module is configured to, in a case where the fault computing power node is located, output fault prompt information, the fault prompt information being used to prompt processing of the fault computing power node.

[0103] The detection module 41 is configured to, in a case where the fault computing power node is not located, return to perform the health detection processing on the computing power platform.

[0104] In some embodiments of the present disclosure, the fault locating apparatus 40 further includes: An input module is configured to input a preset matrix to each operation unit in at least two operation units in the target computing power node.

[0105] The acquisition module is configured to acquire a floating-point calculation throughput of the operation unit.

[0106] The control module is configured to control the operation unit to load the kernel function and acquire a loading duration of the kernel function.

[0107] The determination module is configured to determine that the operation capability of the operation unit is in a failure state in a case where the floating-point calculation throughput is greater than the throughput threshold and the loading duration is less than the duration threshold.

[0108] The determination module is configured to determine that the operation capability of the operation unit is in a failure state in a case where the floating-point calculation throughput is less than or equal to the throughput threshold and / or the loading duration is greater than or equal to the duration threshold.

[0109] In some embodiments of the present disclosure, the detection module 41 is further configured to perform a communication test on each of the at least two operation units in the target computing power node and determine a test result of the operation unit.

[0110] The communication test includes at least one of a full-contract communication test, a contract divergence communication test, a full-collection communication test, and a full-exchange communication test.

[0111] In some embodiments of the present disclosure, the training module 42 is configured to train a preset framework model by the target computing power node.

[0112] The fault locating apparatus 40 further includes: The acquisition module is configured to acquire a floating-point operation frequency per second of the target computing power node and a model floating-point operation utilization rate.

[0113] The determination module is configured to determine that the model training capability of the target computing power node is in a failure state in a case where the floating-point operation frequency per second is less than or equal to the frequency threshold and / or the model floating-point operation utilization rate is greater than or equal to the utilization rate threshold.

[0114] The determination module is configured to determine that the model training capability of the target computing power node is in a failure state in a case where the floating-point operation frequency per second is less than or equal to the frequency threshold and / or the model floating-point operation utilization rate is greater than or equal to the utilization rate threshold.

[0115] In some embodiments of the present disclosure, the fault locating apparatus 40 further includes: The output module is configured to output a fault prompt information in a case where the detection result of the health detection processing indicates that the target computing power node is in a failure state, and the fault prompt information is used to prompt processing of the fault computing power node.

[0116] The fault prompt information includes at least one of the following: identification information of the fault computing power node, identification information of a fault operation unit in the fault computing power node, running capability fault information, communication capability fault information, and model training capability fault information.

[0117] Figure 4 The fault positioning device of the illustrated embodiment can be used to implement the technical solutions of the fault positioning method embodiments described above, and has similar implementation principles and technical effects, which will not be described here again.

[0118] Figure 5 A structural schematic diagram of an electronic device is provided in the embodiments of the present disclosure. The electronic device can be the device to be upgraded as described in the above embodiments. The electronic device provided in the embodiments of the present disclosure can execute the processing flow provided in the fault positioning method embodiments, such as Figure 5 As shown, the device 50 includes a memory 51, a processor 52, a computer program, and a communication interface 53. The computer program is stored in the memory 51 and is configured to be executed by the processor 52 to perform the method as described above.

[0119] In addition, the embodiments of the present disclosure also provide a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the fault positioning method described in the above embodiments.

[0120] In addition, the embodiments of the present disclosure also provide a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed by a processor to implement the fault positioning method as described above.

[0121] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0122] The foregoing is merely illustrative of the various implementations of the present disclosure and the general principles thereof. Numerous modifications can be made to these illustrations, and equivalents can be substituted therefor, without departing from the scope of the present disclosure. The specific embodiments commensurate with the specific application are intended to be illustrative only and not limiting of the scope of the application as set forth in the following claims.

Claims

1. A fault location method, characterized in that, Applied to a computing power platform, the computing power platform including at least two computing power nodes, each of the at least two computing power nodes having at least one computing unit deployed on it, the fault location method includes: Upon receiving a model training request, a health check is performed on the target computing node among the at least two computing power nodes. The health check includes at least one of the following: computing unit detection, communication link detection, and model training detection. The computing unit detection is used to detect the computing power of each computing unit in the target computing power node, the communication link detection is used to detect the communication power of each computing unit in the target computing power node, and the model training detection is used to detect the model training capability of the target computing power node. If the health detection result indicates that the target computing node is fault-free, the model training task is executed through the target computing node in response to the model training request. In the event of an interruption in the model training task, the faulty computing node in the target computing node is located based on the training log, which is the log output during the execution of the model training task.

2. The fault location method according to claim 1, characterized in that, Performing a health check on the target computing node among the at least two computing nodes includes: The target computing node is sequentially subjected to the following steps: computing unit detection, communication link detection, and model training detection.

3. The fault location method according to claim 1, characterized in that, In the event of an interruption in the model training task, after locating the faulty computing node among the target computing power nodes based on the training logs, the fault location method further includes: Upon locating the faulty computing node, a fault prompt message is output, which is used to prompt for processing of the faulty computing node; or, If the faulty computing node is not located, return to performing the health check process on the target computing node.

4. The fault location method according to claim 1, characterized in that, The detection of the arithmetic unit includes: For each computing unit in the target computing node, a preset matrix is ​​input to the computing unit, and the floating-point computing throughput of the computing unit is obtained; The system controls the computing unit to load the kernel function and obtains the loading time of the kernel function. If the floating-point computing throughput is greater than the throughput threshold and the loading time is less than the time threshold, it is determined that the computing power of the computing unit is in a fault-free state. or, If the floating-point computation throughput is less than or equal to the throughput threshold, and / or the loading time is greater than or equal to the duration threshold, the computing power of the computing unit is determined to be in a fault state.

5. The fault location method according to claim 1, characterized in that, The communication link detection includes: For each computing unit in the target computing node, a communication test is performed, and the test result of the computing unit is determined; The communication test includes at least one of the following: full protocol communication test, protocol divergence communication test, full collection communication test, and full switching communication test.

6. The fault location method according to claim 1, characterized in that, The model training and testing include: The preset framework model is trained using the target computing power node, and the number of floating-point operations per second and the floating-point operation utilization rate of the model are obtained. If the number of floating-point operations per second is greater than the number threshold and the model floating-point operation utilization rate is greater than the utilization rate threshold, it is determined that the model training capability of the target computing node is in a fault-free state. or, If the number of floating-point operations per second is less than or equal to the number of operations threshold, and / or the model floating-point operation utilization rate is greater than or equal to the utilization rate threshold, the model training capability of the target computing node is determined to be in a fault state.

7. The fault location method according to any one of claims 1 to 5, characterized in that, Upon receiving a model training request, after performing a health check on the target computing node among the at least two computing nodes, the fault location method further includes: If the detection result of the health detection process indicates that the target computing power node is faulty, a fault prompt message is output, which is used to prompt the faulty computing power node to be processed. The fault indication information includes at least one of the following: the identification information of the faulty computing power node, the identification information of the faulty computing unit in the faulty computing power node, the operation capability fault information, the communication capability fault information, and the model training capability fault information.

8. A fault location device, characterized in that, The fault location device is applied to a computing power platform, which includes at least two computing power nodes, each of which has at least one computing unit deployed on it. The detection module is used to perform health detection processing on the target computing power node among the at least two computing power nodes when a model training request is received. The health detection processing includes at least one of the following: computing unit detection, communication link detection, and model training detection. The computing unit detection is used to detect the computing power of each computing unit in the target computing power node, the communication link detection is used to detect the communication power of each computing unit in the target computing power node, and the model training detection is used to detect the model training capability of the target computing power node. The training module is configured to, in response to the model training request, execute a model training task through the target computing node when the detection result of the health detection process indicates that the target computing node is fault-free. The localization module is used to locate the faulty computing node among the target computing nodes based on the training log when the model training task is interrupted. The training log is the log output during the execution of the model training task.

9. An electronic device, characterized in that, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images