Fault detection method, computing device, storage medium, and program product

Abstract

This disclosure provides a fault detection method, computing device, storage medium, and program product, relating to the field of artificial intelligence technology. The fault detection method includes: acquiring event information generated by a target system running an artificial intelligence task; generating target dependency data based on the event information; and determining whether the target system has a fault that blocks the execution of the artificial intelligence task based on the target dependency data. The target system may include one or more processing units, and each processing unit includes one or more processing queues. The target dependency data may include global dependency data and local dependency data corresponding to the target processing queues. The local dependency data includes data dependencies and / or control dependencies, and the global dependency data includes communication dependencies between different processing units and / or different processing queues within the target system.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a fault detection method, computing device, storage medium, and program product. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, AI models are becoming increasingly large and complex, requiring higher and higher hardware capabilities. Deploying AI models and running AI tasks on large-scale distributed cluster computing systems has become the mainstream approach. For example, large-scale distributed AI tasks can be built based on distributed frameworks such as PyTorch. During the execution of AI tasks, unexpected task hangs may occur due to operator execution anomalies. If task hangs are not detected and handled in a timely manner, computing resources (such as GPUs) will be inefficiently occupied, delaying task processing progress and affecting the application of AI models.

[0003] Therefore, how to reliably and efficiently detect faults such as task suspension during the operation of artificial intelligence tasks has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the present disclosure proposes a new technical solution for fault detection.

[0005] According to a first aspect of the present disclosure, a fault detection method is provided, the method comprising:

[0006] The system acquires event information generated by the target system during the execution of an artificial intelligence task; the target system includes one or more processing units, each processing unit includes one or more processing queues, and the event information includes relevant information generated during the execution of the operators of the artificial intelligence task.

[0007] Target dependency data is generated based on the event information; the target dependency data includes global dependency data and local dependency data corresponding to the target processing queue, the target processing queue includes the processing queue that generates the event information and the processing queue associated with the event information, the local dependency data includes data dependencies and / or control dependencies, and the global dependency data includes communication dependencies between different processing units and / or different processing queues in the target system.

[0008] Based on the target dependency data, determine whether the target system has a fault that blocks the operation of the artificial intelligence task.

[0009] Optionally, the target dependency data is generated or updated periodically; the step of generating the target dependency data based on the event information includes:

[0010] Retrieve target dependency data at the start of the current period;

[0011] Incremental dependency data for the current period is generated based on the event information of the target processing queue in the current period;

[0012] At the end of the current period, the incremental dependency data is merged into the target dependency data to obtain the updated target dependency data.

[0013] Optionally, the target dependency data is stored in the memory of each of the processing units; merging the incremental dependency data into the target dependency data to obtain the updated target dependency data includes:

[0014] Each of the aforementioned processing units synchronizes the incremental dependency data of one or more target processing queues within its unit, and updates the global dependency data and the local dependency data corresponding to the target processing queues; or,

[0015] Each of the processing units sends the incremental dependency data generated by the processing unit in the current period to other processing units in the target system, so that the other processing units can update the target dependency data.

[0016] Optionally, determining whether the target system has experienced a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes:

[0017] When a directed dependency edge from the source node to the target node is added to the target dependency data, a depth-first search algorithm is executed in the target dependency data to determine whether there is a path from the target node to the source node that would cause a directed cycle in the target dependency data; if a directed cycle appears in the target dependency data, it is determined that the target system has a circular dependency fault that blocks the operation of the artificial intelligence task.

[0018] Optionally, determining whether the target system has experienced a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes:

[0019] If the communication dependency meets the communication failure detection start condition, determine the task execution status and task execution duration of multiple communication nodes corresponding to the communication dependency.

[0020] For each communication node, if the task execution status of the communication node is "in execution" and the task execution duration is greater than the first duration threshold corresponding to the communication dependency, then the communication node is determined to be an abnormal node.

[0021] If the proportion of the abnormal nodes among the multiple communication nodes is greater than a preset abnormal proportion threshold, then it is determined that the target system has a communication dependency failure that blocks the operation of the artificial intelligence task.

[0022] Optionally, the communication fault detection activation conditions include one or more of the following:

[0023] The duration of the communication operation corresponding to the communication dependency is greater than or equal to the second duration threshold.

[0024] The current time is greater than or equal to the preset periodic detection time interval since the last communication failure detection.

[0025] The target dependency data includes newly added communication dependencies;

[0026] The obtained event information is a new communication event.

[0027] Optionally, determining whether the target system has experienced a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes:

[0028] Based on the target dependency data, determine the target operator that is in the running state;

[0029] If the target operator meets the timeout fault detection conditions, it is determined that the target system has experienced a timeout fault that blocks the execution of the artificial intelligence task; the timeout fault detection conditions are determined based on the runtime of the target operator, the tensor size of the target operator, and the processor utilization of the target processing unit running the target operator.

[0030] Optionally, after determining, based on the target dependency data, that the target system has encountered a fault that is blocking the execution of the artificial intelligence task, the method further includes:

[0031] Based on the fault type that blocks the execution of the artificial intelligence task and the processor utilization of the target system, fault prompt information is generated. The fault prompt information is used to indicate the reason for the fault that the artificial intelligence task is blocked from running. The fault type includes one or more of timeout faults, circular dependency faults, and communication dependency faults.

[0032] Optionally, the artificial intelligence task is a task of training an artificial intelligence model; the global dependency data also includes iterative boundary relationships, which include one or more of the data dependencies, control dependencies, or communication dependencies between each training iteration of the artificial intelligence task.

[0033] Optionally, acquiring event information generated by the target system during the execution of an artificial intelligence task includes:

[0034] The event information is obtained based on the extended Berkeley packet filter probes deployed in the artificial intelligence task; wherein, the extended Berkeley packet filter probes include one or more of data dependency probes, control dependency probes, and communication dependency probes; wherein:

[0035] The data dependency probe is deployed within the memory processing function provided by the target system. The event information obtained based on the data dependency probe includes the source address and target address of the memory copy event performed by the memory processing function. The data dependency relationship is generated based on the source address and the target address.

[0036] The control dependency probes are deployed at the execution entry and exit points of the operator. The event information obtained based on the control dependency probes includes the current operator and the preceding operator in the operator call stack. The control dependency relationship is generated based on the current operator and the preceding operator.

[0037] The communication dependency probe is deployed within the collection communication library function provided by the target system. The event information obtained based on the communication dependency probe includes a list of communication objects within the collection communication library function, and the communication dependency relationship is generated based on the list of communication objects.

[0038] According to a second aspect of the present disclosure, a computing device is provided, including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform the method as described in the first aspect.

[0039] According to a third aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0040] According to a fourth aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0041] The fault detection method provided in this embodiment determines data dependencies, control dependencies, and communication dependencies through task event information. This covers various fault modes that block the operation of artificial intelligence tasks, improving the reliability of fault detection. Furthermore, it achieves flexible management of dependencies through hierarchical global and local dependency data, and analyzes and processes only event-related local and global dependency data, thus improving fault detection efficiency. Therefore, the solution based on this embodiment can improve the efficiency and reliability of fault detection in artificial intelligence task scenarios.

[0042] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0044] Figure 1 This is a schematic diagram of a target system for performing artificial intelligence tasks provided in an embodiment of this disclosure;

[0045] Figure 2 This is a schematic flowchart of a fault detection method provided in an embodiment of this disclosure;

[0046] Figure 3 This is a block diagram of a fault detection system that implements a fault detection method according to an embodiment of this disclosure;

[0047] Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Detailed Implementation

[0048] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0049] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0050] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0051] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0052] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0053] First, the application scenarios of the embodiments of this disclosure will be described. The embodiments of this disclosure can be applied to fault detection scenarios for artificial intelligence tasks, particularly fault detection of task suspension. Artificial intelligence tasks may include tasks that perform inference or application based on artificial intelligence models, or tasks that train artificial intelligence models. This artificial intelligence task can be executed by the target system. Figure 1This is a schematic diagram of a target system for performing artificial intelligence tasks according to an embodiment of this disclosure. The target system may include multiple processing units, such as processing unit 1, processing unit 2, or more processing units. Each processing unit may include one or more processing queues (e.g., Streams). For example, processing unit 1 may include processing queue 11, processing queue 12, or more processing queues; processing unit 2 may include processing queue 21, processing queue 22, or more processing queues. A processing queue may execute one or more operators (e.g., Ops). For example, processing queue 11 may include operator 111, operator 112, or more operators; processing queue 12 may include operator 121, operator 122, or more operators; processing queue 21 may include operator 211, operator 212, or more operators; and processing queue 22 may include operator 221, operator 222, or more operators. An operator may be the smallest logical unit in the target system that performs a specific computational task, such as a code block or function that performs a specific computational task (e.g., matrix operation). In some examples, the target system may be a distributed cluster computing system. The different processing units in this target system can perform aggregated communication. For example, aggregated communication between processing units can be achieved through aggregated communication module 9. Each processing unit can send local weights generated during model training through the aggregated communication module and receive global weights through the aggregated communication module. This aggregated communication module may include software modules deployed on each processing unit. The processing units in this target system may include a Graphics Processing Unit (GPU), a General-Purpose Graphics Processing Unit (GPGPU), a Neural-network Processing Unit (NPU), or a Tensor Processing Unit (TPU). This processing unit can also be referred to as a Rank. In some examples, the target system may also include a Central Processing Unit (CPU), and the aforementioned aggregated communication module may further include some software modules deployed on this CPU. The CPU can be used to manage the processing units, such as controlling each processing unit to execute corresponding computational tasks.

[0054] The processing units and queues within this target system can execute corresponding operators to perform artificial intelligence tasks. However, due to the complexity of the target system, unexpected task suspensions may occur during the execution of AI tasks due to operator execution anomalies. Taking training tasks as an example, training tasks may be unexpectedly suspended due to operator execution anomalies. If training task suspensions are not detected and handled in a timely manner, it will not only lead to the ineffective use of computing resources (such as GPUs) but also delay model delivery and deployment cycles, severely impacting the application and development of AI models.

[0055] In related technologies, fault detection for the suspended state of artificial intelligence tasks can be achieved through log analysis, code instrumentation, external sampling, and other methods.

[0056] One approach, log analysis, involves embedding numerous log-printing statements into the code implementing the AI ​​model to record the start and end times of critical code execution and the states of related variables. However, this approach relies entirely on post-event collection and manual comparison of timestamps from massive amounts of log files to identify abnormal pauses, resulting in low fault detection efficiency.

[0057] Code instrumentation can directly modify the underlying source code of an AI model, inserting performance counters and state machine tracing logic at the entry / exit points of critical functions. While this approach can obtain more accurate data, it is tightly coupled to specific framework versions, resulting in extremely high maintenance costs. Each framework upgrade may require weeks of adaptation and refactoring, and improper instrumentation can easily introduce secondary failures such as memory leaks or runtime crashes. This approach suffers from poor stability and cannot achieve reliable fault detection.

[0058] External sampling methods can utilize system-level tools (such as perf and nvprof) to perform millisecond-level periodic sampling of process CPU stacks, GPU execution states, or network traffic. However, due to their coarse sampling granularity, they may fail to capture critical events (such as GPU kernel startup and NCCL communication primitives), resulting in large event location errors, a high false alarm rate, and an inability to achieve reliable fault detection.

[0059] Therefore, the fault detection technologies in this field are insufficient to achieve reliable and efficient fault detection for artificial intelligence task scenarios.

[0060] To address the problems of related technologies, this disclosure provides a fault detection method that acquires event information generated by a target system running an artificial intelligence task; generates target dependency data based on the event information; and determines whether the target system has a fault that blocks the operation of the artificial intelligence task based on the target dependency data. The target system may include one or more processing units, and each processing unit includes one or more processing queues. The target dependency data may include global dependency data and local dependency data corresponding to target processing queues. The target processing queues include processing queues that generate event information and processing queues associated with the event information. The local dependency data includes data dependencies and / or control dependencies, and the global dependency data includes communication dependencies between different processing units and / or different processing queues within the target system. By determining data dependencies, control dependencies, and communication dependencies through task event information, this method can cover various fault modes that block the operation of artificial intelligence tasks, improving the reliability of fault detection. Furthermore, the hierarchical dependency data (global and local) enables flexible management of dependencies, and analyzing only event-related local and global dependency data also improves fault detection efficiency. Therefore, the solution based on this embodiment can improve the efficiency and reliability of fault detection in artificial intelligence task scenarios.

[0061] Figure 2 This is a schematic flowchart of a fault detection method provided in an embodiment of this disclosure. The fault detection method can be... Figure 1 The target system shown can be executed, or it can be executed by any computing device. For example... Figure 2 As shown, the fault detection method of this embodiment may include the following steps S210 to S230.

[0062] Step S210: Obtain event information generated by the target system under test when running artificial intelligence tasks.

[0063] The target system may include one or more processing units, and each processing unit may include one or more processing queues. For example, the target system may be as follows: Figure 1 The target system is shown. This target system can be a combination of hardware and software, including one or more processing units and computer programs deployed on those processing units.

[0064] In some examples, the aforementioned event information includes relevant information generated during the execution of operators in an artificial intelligence task. This relevant information may include, for example, the time when the operator starts and / or ends execution, the operator's input parameters, the operator's output parameters, and so on.

[0065] In some examples, the processing queues described above can run operators in artificial intelligence tasks. Different operators within the same processing queue are executed serially, while operators within different processing queues can be executed in parallel.

[0066] The artificial intelligence task can include tasks that involve reasoning or application based on an artificial intelligence model, or tasks that involve training an artificial intelligence model.

[0067] In some examples, the above event information can be obtained based on target probes deployed in artificial intelligence tasks.

[0068] In some examples, the target probe can be an Extended Berkeley Packet Filter (eBPF) probe, which enables a non-intrusive runtime tracing method for AI tasks. For instance, the target program for the AI ​​task is a compiled binary file. The compiled binary file (e.g., libATen.so, libcudart.so, libnccl.so, etc.) can be dynamically hooked using eBPF's user-space probe (uprobe) and / or user-space return probe (uretprobe) mechanisms, injecting security probes at the operator entry and / or exit points of the binary file. CPU register parameters are captured by accessing the register context structure (pt_regs), zero-copy user-space memory is safely read using the BPF safe read user memory function (bpf_probe_read_user), and finally, efficient event streaming from kernel space to user space is achieved through the BPF efficient streaming output ring buffer (bpf_perf_output). The entire process requires no modification to the AI ​​software's framework source code or restarting the training task.

[0069] In other examples, the target probe may include other types of probes, such as Kprobes / Kretprobes probes, which can be used to dynamically insert probes at the entry and return points of kernel functions. Another example is User Statically Defined Tracepoints (USDTs), which can be manually inserted into the program to achieve custom event collection.

[0070] Step S220: Generate target dependency data based on event information.

[0071] The target dependency data may include global dependency data and local dependency data corresponding to the target processing queue. The target processing queue may include a processing queue that generates event information and a processing queue associated with the event information.

[0072] In some examples, the local dependency data may include data dependencies and / or control dependencies. The global dependency data may include communication dependencies between different processing units and / or different processing queues in the target system.

[0073] In some examples, this global dependency data can be referred to as the upper-level skeleton graph, and it can be stored in memory. The local dependency data can be referred to as the lower-level detail graph. Local dependency data corresponding to the target processing queue can be stored in memory. Other processing queues in the target system besides the target processing queue can also have local dependency data, which can be stored on disk. This allows for on-demand loading of local dependency data, reducing memory usage. Furthermore, the target dependency data can include only the local and global dependency data stored or loaded into memory, reducing the amount of data traversed during fault detection and improving fault detection efficiency.

[0074] In some examples, event information can be sharded to generate local dependency data for each processing queue. This enables parallel processing between processing queues, improving event processing efficiency. Furthermore, hash sharding can be performed based on processing units and processing queues to create multiple fault detection processes. Each fault detection process can correspond to one or more processing queues, allowing different fault detection processes to process in parallel.

[0075] In some examples, the global dependency data mentioned above can retain only the communication dependencies between communication nodes across processing units and / or across processing queues, which can compress the scale by more than 90% compared to the full dependency data.

[0076] In some examples, the AI ​​task is to train an AI model, which may perform multiple training iterations to optimize the model's parameters. The global dependency data may also include iteration boundary relationships, which include one or more of the following: data dependencies, control dependencies, or communication dependencies between different training iterations of the AI ​​task. This allows for further preservation of iteration boundary nodes in the global dependency data for the training task, improving the reliability of fault detection. Optionally, the aforementioned local dependency data may include data dependencies and / or control dependencies within the current training iteration, and this local dependency data can be loaded on demand based on the iteration progress.

[0077] In some examples, the target dependency data can be constructed based on a directed graph, where directed edges can represent any of the following: data dependencies, control dependencies, or communication dependencies.

[0078] In some examples, target probes (eBPF probes) deployed in artificial intelligence tasks to acquire event information can include one or more of the following: data dependency probes, control dependency probes, and communication dependency probes. Target probes can be used to construct one or more of these data dependencies, control dependencies, or communication dependencies. The following sections describe the three types of dependencies—data, control, and communication—and their corresponding probes.

[0079] I. Data Dependencies and Data Dependency Probes.

[0080] This data dependency relationship can be used to describe the "producer + consumer" relationship between data, such as a directed edge representing data dependency from a data producer to a data consumer. This data can be a tensor in an artificial intelligence task. If the output data address addr_A of operator A1 is written at the first moment, and the data address addr_A is read as input data by operator B1 at the second moment after the first moment, that is, operator B1 needs to read the data generated by operator A1, then operator B1 is data dependent on operator A1. If the target dependency relationship data is constructed using a directed graph, then a directed edge representing data dependency from operator A1 to operator B1 can be established.

[0081] This data dependency probe can be deployed within the memory processing function provided by the target system. The event information obtained based on this data dependency probe can include the source address and target address of the memory copy event performed by the memory processing function, and data dependency relationships can be generated based on the source address and target address.

[0082] For example, the memory handling function could be the MemcpyAsync function, which can obtain the linear layer weights and gradient transfers of the linear layer operator based on the data dependency probe and establish data dependencies.

[0083] For example, in artificial intelligence tasks, a convolution operator generates a feature map, which the subsequent ReLU activation operator needs to read and compute. Therefore, the ReLU operator's data depends on the convolution operator. In backpropagation, the weight update operator needs to read the gradient values ​​generated by the gradient computation operator. Therefore, the weight update operator's data depends on the gradient computation operator. The target dependency data can include directed edges from the convolution operator to the ReLU activation operator, and directed edges from the gradient computation operator to the weight update operator.

[0084] For example, during forward propagation, the model receives input data X and weights W. After computation by the Linear Forward Propagation operator, it produces output data Y. This data dependency probe can record that data Y was produced by the Linear Forward Propagation operator, meaning the Linear Forward Propagation operator is the producer of data Y. In the subsequent backpropagation process, the Linear Backpropagation operator reads data Y and calculates its gradient based on Y. During this process, the data dependency probe can record that data Y was used by the Linear Backpropagation operator, meaning the Linear Backpropagation operator is the consumer of data Y. Therefore, a directed edge of data dependency can be established from the Linear Forward Propagation operator to the Linear Backpropagation operator. The subsequent weight update operator needs to use this gradient, so a directed edge of data dependency can be established from the Linear Backpropagation operator to the weight update operator. By gradually tracing the "life trajectory" of the data, a complete and accurate data dependency graph reflecting the direction of data flow can be automatically constructed.

[0085] In some examples, the AI ​​task can be training an AI model, and iterative boundary nodes can be constructed during data dependency construction. These iterative boundary nodes can be the start and end points of the data flow in each iteration. For example, in deep learning training, the input and output data of each iteration can be considered iterative boundary nodes. These nodes serve to separate different iterations in the data dependency graph. For instance, when the optimizer updates weights, it needs the updated weight information. In this case, a directed edge representing the iterative data dependency from the weight update operator to the next iteration can be established; this directed edge represents a type of iterative boundary relationship.

[0086] II. Control Dependencies and Control Dependency Probes.

[0087] This control dependency relationship can be used to describe the control logic of the program execution flow. For example, if whether a certain operator B2 is executed or the execution path is determined by the output of another operator A2, then operator B2 is control dependent on operator A2, and a directed edge of control dependency from operator A2 to operator B2 can be established.

[0088] The control dependency probe can be deployed at the execution entry point and / or execution exit point of an operator. The event information obtained based on the control dependency probe can include the current operator and its predecessor in the operator call stack, and control dependencies can be generated based on the current operator and its predecessor. For example, a directed control dependency edge pointing from the predecessor to the current operator can be generated.

[0089] For example, taking the Linear model as an example, in a Multilayer Perceptron (MLP) model, the Linear layer can be located within the Sequential module. The relevant control dependency establishment process is as follows: When the entry event of the first linear layer (Linear1) arrives at the operator call stack, a directed edge of control dependency is established with the parent stack Sequential::forward. After the exit event of the first linear layer (Linear1), the entry event of the activation layer (ReLU) arrives at the operator call stack, establishing a directed edge of control dependency from the first linear layer (Linear1) to the activation layer (ReLU). After the exit event of the activation layer (ReLU), the entry event of the second linear layer (Linear2) arrives at the operator call stack, establishing a directed edge of control dependency from the activation layer (ReLU) to the second linear layer (Linear2). In this way, the control dependency data (such as the control chain) is finally formed: Linear1→ReLU→Linear2, which can accurately reflect the model structure and the control dependencies of operators within the model.

[0090] In some examples, the AI ​​task can be training an AI model, and iterative boundary nodes can also be constructed during the construction of control dependencies. These iterative boundary nodes in the control dependency data can be loop structures within the control flow. For instance, in deep learning training, each iteration of a training loop (such as a for loop or a while loop) forms a control dependency boundary as an iterative control dependency directed edge, which is a type of iterative boundary relation.

[0091] III. Communication Dependencies and Communication Dependency Probes.

[0092] This communication dependency describes the communication dependence between different computing units (processing units / processing queues), such as communication dependencies arising from synchronizing state. For example, in distributed training, multiple processing units synchronize each other's data or state through communication operations (such as ensemble communication). A communication dependency arises if a local operation must wait for a communication operation to complete before it can continue, or if a communication operation needs to wait for all local operations to complete before it can continue.

[0093] The communication dependency probe can be deployed within the collection communication library functions provided by the target system. The event information obtained based on the communication dependency probe can include a list of communication objects within the collection communication library functions, and communication dependencies can be generated based on the list of communication objects.

[0094] For example, taking the DataParallel training scenario as an example, a user-space probe (uprobe) is implanted in the Collective Communication Library (NCCL) to obtain event information of collective communication-related functions (such as ncclAllReduce). This event information may include reading the user-space communicator structure. Parsing the event information can obtain the processing units participating in this communication synchronization and can mark the operation as a gradient synchronization type. After each processing unit completes the backpropagation of the linear layer, a local gradient (dW_local) is generated. This local gradient is sent by calling the ensemble communication related functions. The communication dependency probe deployed in the ensemble communication library functions captures this information and can establish communication dependencies by creating communication nodes (e.g., AllReduce_Linear_Grad). Cross-node dependency edges can also be established, such as: directed cross-node communication dependencies from the linear backward operation (linear_backward_op) of each computing node to the communication node (AllReduce_Linear_Grad); and directed cross-node communication dependencies from the communication node (AllReduce_Linear_Grad) to the weight update operation (weight_update_op) of each processing unit. This forms a communication object covering all processing units participating in this communication synchronization, accurately reconstructing the communication dependencies in data-parallel training.

[0095] In some examples, the AI ​​task can be training an AI model, and iterative boundary nodes can be constructed during the construction of communication dependencies. These iterative boundary nodes in the communication dependency data can be synchronization points in distributed training. For example, in distributed training, each iteration of the communication operation (such as ncclAllReduce) forms a communication dependency boundary as an iterative communication dependency directed edge, which is a type of iterative boundary relationship.

[0096] In this way, the above three types of probes can build three dependencies: "data + control + communication", thereby generating target dependency data, comprehensively covering the fault modes of artificial intelligence tasks, and improving the comprehensiveness and reliability of fault detection.

[0097] Step S230: Determine whether the target system has a fault that blocks the operation of the artificial intelligence task based on the target dependency data.

[0098] For example, fault detection can be performed based on target dependency data to determine whether the target system has a fault that is blocking the operation of artificial intelligence tasks.

[0099] In some examples, the fault detection method in this step may include one or more of the following: cyclic fault detection, communication fault detection, or timeout fault detection.

[0100] Cycle detection can identify circular dependencies in the target system by checking for loops in the dependency data. For example, an incremental depth-first search (DFS) algorithm can be run to determine if a cycle exists in the dependency data. It's important to note that cycle detection can check all dependencies in the target dependency data, or it can detect only the incremental dependency and its associated data when a new dependency is added.

[0101] Communication fault detection can determine whether there is excessively long communication between processing units or processing queues based on the communication dependencies in the target dependency data, and identify slow-running communication nodes based on the communication dependencies, so as to determine whether a communication dependency fault has occurred in the target system.

[0102] Timeout fault detection can identify the currently running operators based on target dependency data, detect whether the operator's runtime has expired, and thus determine whether a timeout fault has occurred in the target system.

[0103] Thus, by detecting loop failures, communication failures, or timeout failures as described above, if at least one of the loop dependency failures, communication dependency failures, or timeout failures is detected, it can be determined that the target system has a failure that blocks the operation of artificial intelligence tasks.

[0104] In some examples, when a fault is found in the target system that is blocking the operation of artificial intelligence tasks, a fault message can be generated and sent to the user via display devices, voice, light, email or chat software to prompt the user to troubleshoot and handle the fault.

[0105] The fault detection method using steps S210 to S230 above determines data dependencies, control dependencies, and communication dependencies through event information. This can cover various fault modes that block the operation of artificial intelligence tasks, improving the reliability of fault detection. Furthermore, it enables flexible management of dependencies through hierarchical global and local dependency data. By analyzing and processing only event-related local and global dependency data, it also improves fault detection efficiency. Thus, it can improve the efficiency and reliability of fault detection in artificial intelligence task scenarios.

[0106] In some embodiments of this disclosure, the target dependency data is generated or updated periodically. For example, it may be updated based on a 100-millisecond period. The specific method of generating the target dependency data based on event information in step S220 may include steps S221 to S223.

[0107] Step S221: Obtain the target dependency data at the start of the current period.

[0108] For example, the target dependency data is stored in memory. The target dependency data for that period can be retrieved at the beginning of each period; that is, the target dependency data stored in memory at the start of that period.

[0109] Step S222: Generate incremental dependency data for the current period based on the event information of the target processing queue in the current period.

[0110] For example, event information for the current period can be obtained in real time, but instead of being directly added to the target dependency data, incremental dependency data for the current period is generated, which reduces the amount of data processed in the current period.

[0111] Step S223: At the end of the current period, the incremental dependency data is merged into the target dependency data to obtain the updated target dependency data.

[0112] In this embodiment, by using an incremental update strategy, it is not necessary to update the target dependency data in full for every event. Instead, incremental updates are performed periodically, which can reduce the computational overhead of fault detection. While balancing real-time performance and computational overhead, it can reduce the memory and processor usage caused by fault detection.

[0113] In some examples, the target dependency data described above can be stored in the memory of each processing unit. The target dependency data for that period can be retrieved at the beginning of each cycle, i.e., the target dependency data stored in memory at the start of that cycle. There are several possible specific methods for merging the incremental dependency data into the target dependency data in step S223 above.

[0114] For example, each processing unit can synchronize the incremental dependency data of one or more target processing queues within its unit, and update the global dependency data and the local dependency data corresponding to the target processing queues.

[0115] For example, each processing unit can send the incremental dependency data generated by itself in the current cycle to other processing units in the target system so that the other processing units can update the target dependency data.

[0116] In some examples, a bidirectional stream can be established between processing units based on a high-performance remote procedure call framework (such as gRPC). Each processing unit can use this bidirectional stream to send incremental dependency data generated in the current cycle to other processing units in the target system. Optionally, the bidirectional stream between processing units can be a persistent stream, established when the target system starts, so that the bidirectional stream does not need to be established again for each data transmission. Optionally, each processing unit can broadcast the above incremental dependency data using the gossip protocol to reduce bandwidth consumption; for example, the bandwidth consumption can be less than or equal to 10 Mbps per processing unit. It should be noted that each processing unit can also use TCP, UDP, or other protocols.

[0117] In some embodiments of this disclosure, the method for determining whether the target system has a fault that blocks the execution of the artificial intelligence task based on the target dependency data in step S230 above can be to perform fault detection based on the target dependency data to determine whether the target system has a fault that blocks the execution of the artificial intelligence task. This fault detection method may include one or more of loop fault detection, communication fault detection, or timeout fault detection.

[0118] In some examples, the fault detection method in step S230 above may include a cycle fault detection method. For example, when adding a directed dependency edge from the source node to the target node in the target dependency data, a depth-first search algorithm can be executed in the target dependency data to determine whether there is a path from the target node to the source node that causes a directed cycle in the target dependency data; if a directed cycle appears in the target dependency data, it is determined that the target system has a cycle dependency fault that blocks the operation of the artificial intelligence task.

[0119] This allows for the triggering of circular dependency detection when a new directed dependency edge is added to the target dependency data. In other words, as the target dependency data dynamically changes, each addition of a new directed dependency edge or node may introduce a new circular dependency fault. Therefore, the target system can trigger circular dependency detection at this time. This enables more efficient and reliable fault detection.

[0120] In some examples, step S230 above may include a communication fault detection method. For example, the specific method of communication fault detection may be carried out through the following steps S231 to S233.

[0121] Step S231: If the communication dependency meets the communication failure detection start condition, determine the task execution status and task execution duration of multiple communication nodes corresponding to the communication dependency.

[0122] For example, the communication fault detection activation conditions may include one or more of the following conditions one through four:

[0123] Condition 1: The duration of the communication operation corresponding to the communication dependency is greater than or equal to the second duration threshold. This second duration threshold can be a duration set based on engineering experience.

[0124] Condition 2: The time elapsed since the last communication failure detection is greater than or equal to the preset periodic detection time interval. For example, this communication failure detection can be triggered periodically, and the periodic detection time interval can be set by the user based on engineering experience.

[0125] Condition 3: A new communication dependency is added to the target dependency data. For example, if a new communication dependency is added when updating the target dependency data based on the incremental dependency data, the communication failure detection can be activated.

[0126] Condition 4: The obtained event information is a new communication event. For example, a new communication event could be the capture of an entry or exit event of an NCCL communication function.

[0127] Using any one or more of the above conditions one to four can trigger communication fault detection, thereby improving the robustness and timeliness of the system in fault detection.

[0128] Step S232: For each communication node, if the task execution status of the communication node is "in execution" and the task execution duration is greater than the first duration threshold corresponding to the communication dependency, then the communication node is determined to be a node with an abnormal status.

[0129] The first duration threshold can be a duration set by the user based on engineering experience. The first duration threshold and the second duration threshold can be the same or different.

[0130] In some examples, the first duration threshold can be 1.5 times or 2 times the second duration threshold.

[0131] Step S233: If the proportion of abnormal nodes among multiple communication nodes is greater than a preset abnormal proportion threshold, it is determined that the target system has a communication dependency fault that blocks the operation of artificial intelligence tasks.

[0132] For example, the preset anomaly ratio can be a ratio set by the user based on engineering experience, such as 20%, 30%, or 50%.

[0133] Thus, by performing communication fault detection using steps S231 to S233, it is possible to detect whether there is a fault in the communication dependency relationship and identify the faulty communication node, thereby improving the granularity and accuracy of fault detection.

[0134] In some examples, step S230 above may include a timeout fault detection method. For example, the specific method of timeout fault detection may be: determining the target operator in the running state based on target dependency data; if the target operator meets the timeout fault detection condition, then determining that the target system has experienced a timeout fault that blocks the operation of the artificial intelligence task.

[0135] The timeout fault detection condition can be determined based on the runtime of the target operator, the tensor size of the target operator, and the processor utilization of the target processing unit running the target operator. The tensor size of the target operator can be determined based on the matrix dimension; for example, a tensor with 1028 by 4096 dimensions has a tensor size of 4210688 (the product of 1028 x 4096).

[0136] For example, the timeout fault detection condition may include: the runtime of the target operator is greater than a third duration threshold, and the tensor size of the target operator is less than or equal to a preset size threshold.

[0137] For example, the timeout fault detection conditions may include: the runtime of the target operator is greater than a third duration threshold, the tensor size of the target operator is greater than a preset size threshold, and the processor utilization of the target processing unit running the target operator is less than or equal to a first preset utilization threshold.

[0138] The aforementioned third duration threshold, preset size threshold, and first preset utilization rate threshold can all be duration thresholds set by the user based on engineering experience. For example, the third duration threshold could be 100 milliseconds, 1 second, or 3 seconds. The preset size threshold could be 4210688, 1×10 8 Or 1×10 9 The first preset utilization threshold can be 5%, 10%, or 20%.

[0139] Thus, by using the timeout fault detection in this example, and setting timeout fault detection conditions based on three dimensions of "runtime + tensor size + processor utilization", it is possible to distinguish between normal long time consumption introduced by large-scale tensors and abnormal blocking of small-scale tensors, avoid false detection and false alarms, and improve the accuracy of fault detection in artificial intelligence tasks.

[0140] In some examples, step S230 above may include the three methods described above: loop fault detection, communication fault detection, and timeout fault detection. Furthermore, based on these three fault detection methods, comprehensive diagnostic detection can be performed, generating fault indication information based on decision tree rules.

[0141] For example, after determining that the target system has encountered a fault that is blocking the operation of the artificial intelligence task based on the target dependency data, the method may further include: generating fault indication information based on the fault type that is blocking the operation of the artificial intelligence task and the processor utilization of the target system. The fault indication information is used to indicate the cause of the fault that is blocking the operation of the artificial intelligence task. The fault type may include one or more of timeout faults, circular dependency faults, and communication dependency faults.

[0142] The processor utilization of the target system can be the utilization of the processing unit in the target system. For example, if the processing unit is a GPU, then the processor utilization can be the GPU utilization.

[0143] There are several ways to generate fault message information based on the fault type that blocks the execution of the artificial intelligence task and the processor utilization of the target system. For example, the fault type and the current processor utilization can be directly combined to obtain the fault message information. Alternatively, the fault type and processor utilization can be further judged to determine the fault type that should be included in the fault message information. For example, this fault message information can be generated based on the following decision logic:

[0144] First, if the fault type includes a communication dependency fault and the processor utilization is less than or equal to a second preset utilization threshold (e.g., 10% or 20%), then the communication dependency fault can be identified as the primary fault blocking the execution of the artificial intelligence task. The generated fault message in this case can include "Network congestion or presence of abnormal nodes; please check NCCL and network topology." Furthermore, the fault message can also include information related to the abnormal nodes to facilitate troubleshooting and resolution for the user.

[0145] Secondly, if the fault type includes a timeout fault and the processor utilization is greater than or equal to a third preset utilization threshold (e.g., 80% or 90%), it can be determined that the AI ​​task is not blocked and the operator is still being executed by the processor. In this case, the generated fault message may include "Operator takes a long time (not suspended), it is recommended to increase the timeout threshold for this operator." Furthermore, the fault message may also include the aforementioned operator-related information to facilitate user troubleshooting and resolution.

[0146] Furthermore, if the fault type includes a timeout fault and the processor utilization is less than the fourth preset utilization threshold (e.g., 10% or 20%), then the timeout fault can be identified as the primary fault blocking the execution of the artificial intelligence task. In this case, the generated fault message may include "Operator takes a long time, please check the operator." Further, the fault message may also include operator-related information about the timeout fault to facilitate user troubleshooting and resolution.

[0147] Finally, if the fault type includes a circular dependency fault, the generated fault message may include "A circular deadlock has occurred; please check the flow dependencies." Furthermore, the fault message may also include the dependent edges and related nodes where the circular dependency fault occurs, to facilitate troubleshooting and resolution for users.

[0148] In this way, by classifying the failures that block the operation of artificial intelligence tasks into three categories—timeout, circular dependency, and communication dependency—and further combining them with processor utilization for correlation analysis, the system can automatically generate fault prompts with clear root causes without additional manual intervention. These prompts can directly point to the essential reason why the artificial intelligence task is blocked, reducing the difficulty of fault location, narrowing the scope of investigation, and improving the efficiency of fault diagnosis.

[0149] Figure 3 This is a block diagram of a fault detection system implementing a fault detection method according to an embodiment of this disclosure. Figure 3 As shown, the fault detection system may include a kernel-mode probe 310 and a user-space data processing engine 320. The kernel-mode probe 310 can execute... Figure 2 The step S210 shown is used to obtain event information generated by the target system under test when running artificial intelligence tasks; the user space data processing engine 320 can execute... Figure 2 Steps S220 and S230, as shown, generate target dependency data based on event information and determine whether the target system has encountered a fault that blocks the execution of the artificial intelligence task based on the target dependency data. In this way, through a layered architecture design of kernel-mode tracing + user-mode analysis + distributed collaboration, non-intrusive fault detection is achieved for the entire chain of artificial intelligence tasks (such as PyTorch distributed training tasks).

[0150] In some examples, the kernel-mode probe 310 can employ an eBPF probe, dynamically mounted via uprobe / uretprobe into the binary files related to operators, set communication, and memory processing in the AI ​​task, capturing nanosecond-precision timing data at function entry / exit points. This kernel-mode probe may include a symbol resolution module 311, a probe binding module 312, a register access module 313, and an event aggregation module 314. The symbol resolution and probe binding modules deploy the probe within the AI ​​task, the register access module captures event information, and the event aggregation module injects the captured event information into the bpf_perf_output circular buffer. Using an eBPF probe results in a single probe overhead of less than 200 nanoseconds and an impact on training performance of less than 1.5%, avoiding modification of the AI ​​task's software framework source code and achieving a truly zero-intrusion probe. It should be noted that the specific implementation of this kernel-mode probe can also refer to implementation methods in related technologies.

[0151] The symbol resolution module can execute corresponding symbol resolution instructions to obtain the virtual address offset of operators when initiating fault detection for the target system. For example, it can handle C++ name mangling, use C++ filt to restore symbol names, and establish a symbol + address mapping table. The probe binding module can declare probe mount points using the SEC macro in the C code of Ebpf. When running the linear layer process distance for an AI training task, the probe capture process can include the following steps S1 to S13.

[0152] S1, The training process calls linear layer operators (e.g., at::linear).

[0153] S2. The processing unit executes to the function entry point (e.g., the entry address is 0x...d4f0).

[0154] S3 triggers the INT3 breakpoint, causing the user-mode probe (uprobe) handler in the kernel to be trapped.

[0155] S4. The kernel saves the context in the register context structure (pt_regs), such as the Destination Index (RDI) and Source Index (RSI).

[0156] S5. After verifying the security of the eBPF program, execute the probe bytecode.

[0157] S6. The probe reads RDI / RSI to obtain the input / weight pointer, calls the BPF time acquisition function (e.g., bpf_ktime_get_ns) to obtain the kernel time, and records the event information.

[0158] S7. Execute the BPF safe read user memory function (bpf_probe_read_user) to safely copy event information, which may include tensor metadata.

[0159] S8. Call the performance event submission function (perf_submit) to write the event information to the ring buffer (ringbuf).

[0160] S9. The kernel restores the original instructions and registers and returns to user mode to execute the matrix multiplication kernel (GEMM kernel).

[0161] S10, The function has finished executing and is ready to return (ret) instruction.

[0162] S11, triggering the user-mode return probe (uretprobe), and then trapping the kernel exit probe again.

[0163] S12. Execute the exit probe to capture the output tensor address, calculate the operator execution duration, generate event information, and write it to the circular buffer.

[0164] S13. Restore the original return address and return to the caller normally.

[0165] In this way, the kernel-mode probe can obtain event information generated by the target system when running artificial intelligence tasks without affecting the operation of the artificial intelligence tasks through the above steps.

[0166] In some examples, the user-space data processing engine 320 may include an event sharding module 321, a dependency graph construction module 322, and a fault detection module 323, undertaking complex logical calculations. The event sharding module can adopt an architecture of "event sharding + parallel processing + incremental aggregation." For example, input event information can be hashed and distributed to multiple worker threads according to "processing unit, processing queue." Each worker thread independently constructs a local dependency graph and performs fault detection, eliminating lock contention. The dependency graph construction module can merge operator, kernel, and set communication events into a unified directed dependency graph using timestamp alignment and call stack hash matching. This graph serves as the target dependency relationship data. Based on hierarchical global and local dependency relationship data, it can retain the complete causal chain while compressing the node scale. Furthermore, this dependency graph construction module can also employ an incremental update strategy, such as periodically merging incremental dependencies every 100ms, reducing the complexity of dependency updates from O(N) to O(ΔN).

[0167] In some examples, this userspace processing engine can also provide distributed collaboration, achieving cross-node synchronization via bidirectional gRPC streams. Each processing unit asynchronously broadcasts incremental dependency data (including new operator identifiers, dependency edges, communication status, and other incremental information), quickly achieving a global view based on the gossip protocol. Figure 1 This yields the synchronized target dependency data.

[0168] In some examples, the fault detection module can accurately determine whether the problem is local computing blockage or network partitioning based on synchronized target dependency data, improving the accuracy of the fault location. For instance, this fault detection module can model dependency graph anomaly patterns to achieve multi-dimensional fault detection, including loop fault detection, communication fault detection, and timeout fault detection, thus enabling comprehensive fault detection.

[0169] In some examples, the fault detection system can also provide an information output module that can provide interactive D3.js spatiotemporal graphs for visual diagnosis. For example, it can render dependency sequence hierarchically according to processing units / processing queues, and provide visual fault prompts such as highlighted alarms for fault status nodes. Furthermore, the information output module can also generate a JSON structured diagnostic report, integrating three types of detection results: timeout, circular dependency, and communication dependency, and providing fault investigation and repair suggestions, reducing the Mean Time To Repair (MTTR) from hours to minutes.

[0170] Figure 4 This is a schematic diagram of the structure of a computing device provided in an embodiment of this disclosure. Figure 4 As shown, the computing device 400 may include a memory 410 and a processor 420. The memory 410 may be used to store computer instructions, and the processor 420 may be used to retrieve computer instructions from the memory 410 to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The processor may be one or more processors, which may execute instructions individually or jointly. The memory may also be one or more memories, which may store the aforementioned computer instructions individually or jointly. The processor 420 may include... Figure 1 The processing unit in the target system shown.

[0171] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the foregoing embodiments of this disclosure. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited thereto, and may also be a temporary storage medium.

[0172] This disclosure also provides a computer program product that may include a computer program that, when executed by a processor, can implement any of the methods described in the foregoing embodiments of this disclosure.

[0173] This disclosure also provides a chip that may include one or more processing units, which can be used to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure. The chip may be in the form of an Application Specific Integrated Circuit (ASIC), a System on Chip (SOC), a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), etc., and this embodiment is not limited to this. Optionally, the chip may further include a storage unit, which can be used to store computer instructions, and the processing unit can be used to retrieve the computer instructions from the storage unit to execute all or part of the steps of any of the methods in the foregoing embodiments of this disclosure.

[0174] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0175] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0176] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0177] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Smalltalk, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, may execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0178] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0182] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. A fault detection method characterized by, The method includes: The system acquires event information generated by the target system during the execution of an artificial intelligence task; the target system includes one or more processing units, each processing unit includes one or more processing queues, and the event information includes relevant information generated during the execution of the operators of the artificial intelligence task. Target dependency data is generated based on the event information. The target dependency data includes global dependency data and local dependency data corresponding to the target processing queue. The target processing queue includes the processing queue that generates the event information and the processing queue associated with the event information. The local dependency data includes data dependencies. The global dependency data includes communication dependencies between different processing units and / or different processing queues in the target system. The data dependencies are generated based on the source address and target address of the memory copy event obtained by the data dependency probe. The data dependency probe is deployed within the memory processing function provided by the target system. The event information obtained based on the data dependency probe includes the source address and target address of the memory copy event performed by the memory processing function. Based on the target dependency data, determine whether the target system has a fault that blocks the operation of the artificial intelligence task.

2. The method of claim 1, wherein, The target dependency data is generated or updated periodically; the generation of target dependency data based on the event information includes: Retrieve target dependency data at the start of the current period; Incremental dependency data for the current period is generated based on the event information of the target processing queue in the current period; At the end of the current period, the incremental dependency data is merged into the target dependency data to obtain the updated target dependency data.

3. The method according to claim 2, characterized in that, The target dependency data is stored in the memory of each of the processing units; merging the incremental dependency data into the target dependency data to obtain the updated target dependency data includes: Each of the aforementioned processing units synchronizes the incremental dependency data of one or more target processing queues within its unit, and updates the global dependency data and the local dependency data corresponding to the target processing queues; or, Each of the processing units sends the incremental dependency data generated by the processing unit in the current period to other processing units in the target system, so that the other processing units can update the target dependency data.

4. The method according to claim 1, characterized in that, The step of determining whether the target system has a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes: When a directed dependency edge from the source node to the target node is added to the target dependency data, a depth-first search algorithm is executed in the target dependency data to determine whether there is a path from the target node to the source node that would cause a directed cycle in the target dependency data; if a directed cycle appears in the target dependency data, it is determined that the target system has a circular dependency fault that blocks the operation of the artificial intelligence task.

5. The method according to claim 1, characterized in that, The step of determining whether the target system has a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes: If the communication dependency meets the communication failure detection start condition, determine the task execution status and task execution duration of multiple communication nodes corresponding to the communication dependency. For each communication node, if the task execution status of the communication node is "in execution" and the task execution duration is greater than the first duration threshold corresponding to the communication dependency, then the communication node is determined to be an abnormal node. If the proportion of the abnormal nodes among the multiple communication nodes is greater than a preset abnormal proportion threshold, then it is determined that the target system has a communication dependency failure that blocks the operation of the artificial intelligence task.

6. The method according to claim 5, characterized in that, The communication fault detection activation conditions include one or more of the following: The duration of the communication operation corresponding to the communication dependency is greater than or equal to the second duration threshold. The current time is greater than or equal to the preset periodic detection time interval since the last communication failure detection. The target dependency data includes newly added communication dependencies; The obtained event information is a new communication event.

7. The method according to claim 1, characterized in that, The step of determining whether the target system has a fault that blocks the execution of the artificial intelligence task based on the target dependency data includes: Based on the target dependency data, determine the target operator that is in the running state; If the target operator meets the timeout fault detection conditions, it is determined that the target system has experienced a timeout fault that blocks the execution of the artificial intelligence task; the timeout fault detection conditions are determined based on the runtime of the target operator, the tensor size of the target operator, and the processor utilization of the target processing unit running the target operator.

8. The method according to claim 1, characterized in that, After determining, based on the target dependency data, that the target system has encountered a fault that is blocking the execution of the artificial intelligence task, the method further includes: Based on the fault type that blocks the execution of the artificial intelligence task and the processor utilization of the target system, fault prompt information is generated. The fault prompt information is used to indicate the reason for the fault that the artificial intelligence task is blocked from running. The fault type includes one or more of timeout faults, circular dependency faults, and communication dependency faults.

9. The method according to any one of claims 1 to 8, characterized in that, The artificial intelligence task is the task of training an artificial intelligence model; the global dependency data also includes iterative boundary relationships, which include one or more of the data dependencies, control dependencies, or communication dependencies between each training iteration of the artificial intelligence task.

10. The method according to any one of claims 1 to 8, characterized in that, The local dependency data also includes control dependencies; the acquisition of event information generated by the target system during the execution of artificial intelligence tasks includes: The event information is obtained based on the extended Berkeley packet filter probes deployed in the artificial intelligence task; wherein, the extended Berkeley packet filter probes include control dependency probes and / or communication dependency probes; wherein: The control dependency probes are deployed at the execution entry and exit points of the operator. The event information obtained based on the control dependency probes includes the current operator and the preceding operator in the operator call stack. The control dependency relationship is generated based on the current operator and the preceding operator. The communication dependency probe is deployed within the collection communication library function provided by the target system. The event information obtained based on the communication dependency probe includes a list of communication objects within the collection communication library function, and the communication dependency relationship is generated based on the list of communication objects.

11. A computing device, characterized in that, The method includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method of any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images