A model training fault analysis method and device

Abstract

The application discloses a model training fault analysis method and device, relates to the technical field of computers, and can improve the training efficiency of a model and the resource utilization rate of a training cluster and reduce the training cost. The application can acquire fault information and / or abnormal indexes when a training cluster performs model training, determine fault nodes in a cross-stack graph when the model training is performed according to the fault information and / or the abnormal indexes, and determine a fault propagation path corresponding to the fault nodes and used for determining fault reasons and / or abnormal reasons when the model training is performed according to the cross-stack graph. Based on the application, the root cause of a fault or an abnormality can be quickly and accurately determined, the training efficiency of a model and the resource utilization rate of a training cluster can be improved, and the training cost can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for model training fault analysis. Background Technology

[0002] As models iterate and upgrade, the size of their training data and parameters also increases (e.g., large models), requiring enormous computing and storage resources. To meet the massive computing and storage demands of model training, large models can be trained in large-scale training cluster environments. However, as the size of the training cluster expands, the failure of any node or component within the cluster can impact the training task, significantly increasing the probability of training interruptions. This reduces training efficiency and resource utilization within the training cluster, ultimately increasing training costs. Summary of the Invention

[0003] This application provides a model training fault analysis method and apparatus, which can improve the training efficiency of the model and the resource utilization of the training cluster, and reduce the training cost.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] Firstly, a method for analyzing model training faults is provided. This method includes: acquiring fault information and / or anomaly indicators during model training in a training cluster; constructing a cross-stack graph based on software stack information and hardware information, such as software stack information representing the hierarchy of software nodes and the calling relationships between software nodes during model training, and hardware information representing the hierarchy of hardware nodes in the training cluster and the network connection relationships between hardware nodes; determining faulty nodes in the cross-stack graph based on the fault information and / or anomaly indicators; and determining, based on the cross-stack graph, fault propagation paths corresponding to the faulty nodes, used to determine the causes of faults and / or anomalies during model training.

[0006] The solution provided in the first aspect above enables the model training fault analysis device to determine faulty nodes in the cross-stack graph based on fault information and / or abnormal indicators, and to determine the fault propagation path corresponding to the faulty nodes based on the cross-stack graph. Since the cross-stack graph contains software stack information representing the levels of software nodes and the calling relationships between software nodes during model training, as well as hardware information representing the levels of hardware nodes and the network connection relationships between hardware nodes in the training cluster, the model training fault analysis device can determine the root cause of the fault based on the fault propagation path, such as the fundamental cause of the fault or abnormality. This facilitates rapid fault repair, improves training efficiency and resource utilization of the training cluster, and reduces training costs.

[0007] In some examples, software nodes include one or more of the model's neural network layers, acceleration components, framework functions, logic operators, and computation operators, while hardware nodes include one or more of the training cluster's artificial intelligence (AI) accelerator cards, switches, optical modules, and network interfaces.

[0008] In one possible implementation, determining the fault node in the cross-stack graph during model training based on fault information and / or anomaly indicators includes: making a first annotation at the corresponding fault node in the cross-stack graph based on the fault information; and / or, making a second annotation at the corresponding fault node in the cross-stack graph based on the anomaly indicators. Thus, annotating the fault node in the cross-stack graph with fault information and / or anomaly indicators facilitates the rapid and accurate acquisition of the fault propagation path based on the fault node in the cross-stack graph, thereby determining the cause of the fault and / or the cause of the anomaly during model training based on the fault propagation path.

[0009] In one possible implementation, determining the fault propagation path corresponding to the faulty node based on the cross-stack graph includes: determining the neighboring nodes of the faulty node in the cross-stack graph, where neighboring nodes are nodes connected to the faulty node within a preset hop count range; determining a fault subgraph based on the faulty node, neighboring nodes, call relationships, and / or network connection relationships; and determining the fault propagation path based on the fault subgraph. This transforms obtaining the fault propagation path from the entire cross-stack graph into obtaining it from the fault subgraph, narrowing the search range for fault propagation paths, improving the efficiency of fault propagation path acquisition, thereby improving the training efficiency of the model and the resource utilization of the training cluster, and reducing training costs.

[0010] In one possible implementation, the faulty node includes a first faulty node and a second faulty node. The determination of the faulty subgraph based on the faulty node, neighboring nodes, call relationships, and / or network connections includes: determining a first faulty subgraph corresponding to the first faulty node based on the first faulty node, its corresponding neighboring nodes, call relationships, and / or network connections; determining a second faulty subgraph corresponding to the second faulty node based on the second faulty node, its corresponding neighboring nodes, call relationships, and / or network connections; if the first and second faulty subgraphs are connected, a faulty subgraph is determined based on the first and second faulty subgraphs; if the first and second faulty subgraphs are not connected, a faulty subgraph is determined based on the first faulty subgraph, the second faulty subgraph, and auxiliary nodes, where auxiliary nodes are nodes that can connect the first and second faulty subgraphs across the stack graph. Thus, by merging the first and second faulty subgraphs into a single faulty subgraph, the number of nodes in the faulty subgraph can be increased, which is beneficial for uncovering the relationships between multiple faulty nodes and obtaining a more comprehensive and accurate fault propagation path.

[0011] In some examples, the first and second fault subgraphs being connected means that any node in the first fault subgraph can be connected to any node in the second fault subgraph by one or more edges. When the first and second fault subgraphs are connected, the first and second fault subgraphs may have overlapping portions. The first and second fault subgraphs being disconnected means that no node in the first fault subgraph is connected to any node in the second fault subgraph.

[0012] In one possible implementation, the fault subgraph comprises multiple levels. Obtaining the fault propagation path based on the fault subgraph includes: starting from the path's starting point in the fault subgraph, searching the fault subgraph based on the call relationships and / or network connections between nodes until the path's ending point is found. The path's starting point is the highest-level fault node in the fault subgraph. The fault propagation path is then obtained based on the path's starting point, the call relationships and / or network connections between nodes involved in the search, and the path's ending point. This allows for searching from the highest-level fault node in the fault subgraph down to lower levels, resulting in a more complete fault propagation path, which is beneficial for improving the accuracy of fault cause and / or anomaly cause analysis during model training.

[0013] In some examples, the fault subgraph includes software and hardware layers, with paths originating and ending in either the software or hardware layers. This allows for the identification of software-level fault propagation paths, or combined software and hardware-level fault propagation paths, through fault subgraph analysis. Further analysis of these propagation paths can then reveal software faults during model training within the training cluster, or interrelated software and hardware faults, improving the accuracy of fault and / or anomaly analysis during model training.

[0014] In one possible implementation, the above-mentioned process, starting from the path's starting point, searches the fault subgraph based on the call relationships and / or network connections between nodes until the path's ending point is found. This includes: starting from the path's starting point, searching the software layer based on the call relationships between nodes in the fault subgraph until the software layer's ending point is found; if the software layer's ending point is located at a non-lowest level within the software layer, it is taken as the path's ending point. Thus, since nodes at non-lowest levels in the software layer cannot directly call hardware nodes, the model training fault analysis device can determine that a software fault exists during model training in the training cluster, eliminating the need to investigate hardware faults and improving fault analysis efficiency.

[0015] In another possible implementation, the software layer is at a higher level than the hardware layer. Starting from the path's origin, a search is performed in the fault subgraph based on the call relationships and / or network connections between nodes until the path's endpoint is reached. This includes: starting from the path's origin, searching within the software layer based on the call relationships between nodes in the fault subgraph until the software layer endpoint is reached (the software layer endpoint is at the lowest level); and starting from the software layer endpoint, searching within the hardware layer based on the network connections between nodes in the fault subgraph until the hardware layer endpoint is reached and used as the path endpoint. Thus, since the lowest-level nodes in the software layer can directly call hardware nodes, searching from the software layer endpoint to the hardware layer endpoint can determine if there are software-hardware related faults during model training in the training cluster. This eliminates the need to investigate purely software and purely hardware faults separately, improving fault analysis efficiency.

[0016] As an example, the above-described search, starting from the path origin and based on the calling relationships between nodes in the fault subgraph, is performed in the software layer until the software layer endpoint is reached. This includes: starting from the path origin, searching in the software layer based on the calling relationships between nodes in the fault subgraph; if the current search node has a first label and / or a second label, searching for a one-hop neighbor node that has not been searched from the search node until the software layer endpoint is reached; the one-hop neighbor node is at the same level as the search node or the neighbor node is at the next level below the search node; if the current search node does not have a first label and a second label, and the search node is connected to a fault edge, the other node connected to the fault edge is taken as a new search node, and searching for a one-hop neighbor node that has not been searched from the new search node until the software layer endpoint is reached.

[0017] Thus, by searching the software layer hop-by-hop, no one-hop neighbor node corresponding to the search node can be missed, thereby improving the granularity of the search and the accuracy of the fault propagation path in the software layer. Furthermore, even when the current search node has not been labeled with its first and second annotations, confirming the connection between the search node and the fault edge can uncover potential associations between the search node and its one-hop neighbors, such as associations between multiple fault nodes, further improving the granularity of the search and the accuracy of the fault propagation path in the software layer.

[0018] As an example, the above-described method, starting from the endpoint of the software layer, searches the hardware layer based on the network connections between nodes in the fault subgraph until the endpoint of the hardware layer is found. This includes: if the endpoint of the software layer is at the lowest level in the software layer, using the fault node of the hardware layer connected to the endpoint of the software layer as the starting point of the hardware layer; and starting from the starting point of the hardware layer, searching the hardware layer based on the network connections between nodes in the fault subgraph until the endpoint of the hardware layer is found. Thus, stopping the search in the software layer when the current search node is at the lowest level in the software layer allows for a more complete fault propagation path in the software layer, which is beneficial for improving the accuracy of fault cause and / or anomaly cause analysis during model training. Furthermore, since the lowest-level nodes in the software layer can directly call hardware nodes, the model training fault analysis device, starting from the endpoint of the software layer and searching the hardware layer based on the network connections between nodes in the fault subgraph until the endpoint of the path is found, can determine that there are software and hardware-related faults during model training in the training cluster. This eliminates the need to investigate purely software and purely hardware faults, improving fault analysis efficiency.

[0019] As an example, the above method starts from the beginning of the hardware layer and searches the hardware layer based on the network connections between nodes in the fault subgraph until the end of the hardware layer is found. This includes: obtaining the connection path corresponding to the starting point of the hardware layer based on the network connections between nodes in the fault subgraph; if a faulty node and / or faulty edge exists in the connection path, the end point of the connection path is taken as the end point of the hardware layer. In this way, the connection path corresponding to the starting point of the hardware layer is determined based on the network connections between nodes in the fault subgraph, and then the connection path is checked to obtain the fault propagation path in the hardware layer. This eliminates the need to obtain the fault propagation path in the hardware layer by searching hop by hop as in the software layer, saving computing resources and improving the efficiency of obtaining the fault propagation path.

[0020] In some examples, a fault edge is a call relationship and / or network connectivity relationship between at least one end of a faulty node. For instance, a faulty node might be determined based on call relationships between software nodes and / or network connectivity relationships between hardware nodes.

[0021] In one possible implementation, the above-mentioned method starts from the path starting point in the fault subgraph and searches the fault subgraph based on the call relationships and / or network connections between nodes until the path ending point is found. This includes: starting from the path starting point in the fault subgraph, if the current search node has not been labeled with the first and second annotations and is not connected to a fault edge, then the search node is taken as the path ending point. Thus, stopping the search of the software layer when the current search node has not been labeled with the first and second annotations and is not connected to a fault edge allows for a more complete fault propagation path to be obtained in the software layer, which is beneficial for improving the accuracy of fault cause and / or anomaly cause analysis during model training.

[0022] In one possible implementation, the anomaly indicator indicates a runtime failure during model training, and the faulty node is the node exhibiting the runtime failure. Obtaining the anomaly indicator includes: obtaining the memory utilization rate of the AI ​​accelerator cards in the training cluster; if the absolute value of the difference between the current memory utilization rate of the AI ​​accelerator card and its preset memory utilization rate is greater than or equal to a first value, the memory utilization rate of the AI ​​accelerator card is considered an anomaly indicator; and / or, obtaining the computing power utilization rate of the AI ​​accelerator cards in the training cluster; if the absolute value of the difference between the computing power utilization rate of the AI ​​accelerator card and its preset computing power utilization rate is greater than or equal to a second value, the computing power utilization rate of the AI ​​accelerator card is considered an anomaly indicator. The system uses the following metrics: 1) The execution time of a neural network layer in the model is obtained. If the difference between the execution time of this neural network layer and its preset execution time is greater than or equal to a third value, the execution time of this neural network layer is considered an anomaly metric. 2) The call time of a framework function in the model is obtained. If the difference between the call time of this framework function and its preset call time is greater than or equal to a fourth value, the call time of this framework function is considered an anomaly metric. 3) The computation time of a computation operator in the model is obtained. If the difference between the computation time of this computation operator and its preset computation time is greater than or equal to a fifth value, the computation time of this computation operator is considered an anomaly metric. Thus, in model training scenarios, by obtaining anomaly metrics for different nodes, the types of faulty nodes can be enriched, facilitating the subsequent acquisition of more accurate fault propagation paths. This allows for the discovery of potential correlations between different types of faulty nodes, quickly identifying the root cause of operational failures.

[0023] In some examples, model training scenarios include, but are not limited to, scenarios where model training is interrupted. Runtime failures may include, but are not limited to, failures that cause model training to be interrupted.

[0024] In one possible implementation, anomaly indicators represent performance failures in the model during training, with faulty nodes being the nodes exhibiting performance issues. Obtaining these anomaly indicators includes: acquiring the execution time of nodes in the model; if the difference between the node's execution time and its corresponding preset execution time is greater than or equal to a sixth value, the node's execution time is used as an anomaly indicator. Thus, in model performance analysis scenarios, acquiring anomaly indicators for different nodes can enrich the types of faulty nodes, facilitating the subsequent acquisition of more accurate fault propagation paths, thereby uncovering potential correlations between different types of faulty nodes and quickly determining the root cause of performance failures.

[0025] In some examples, model performance analysis scenarios include, but are not limited to, scenarios where the model training process is not interrupted but the training speed decreases. Performance failures include, but are not limited to, failures that cause a decrease in model training speed.

[0026] In one possible implementation, obtaining fault information includes: retrieving error information from the model's training logs; and determining fault information based on the error information, whereby the fault information includes fault node identifiers. Thus, by obtaining fault information including fault node identifiers, the sources of fault nodes can be increased, which is beneficial for identifying more fault nodes and thereby improving the accuracy of the subsequently obtained fault propagation paths.

[0027] In a second aspect, a model training fault analysis apparatus is provided, comprising: a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the model training fault analysis apparatus in implementing the method as described in any possible implementation of the first aspect.

[0028] Thirdly, a model training fault analysis apparatus is provided, comprising corresponding modules, units, or means for implementing the method in the first aspect described above. These modules, units, or means can be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the aforementioned functions.

[0029] As an example, the model training fault analysis device may include: an information acquisition module for acquiring fault information and / or anomaly indicators during model training in the training cluster; and a processing module for constructing a cross-stack graph based on software stack information and hardware information, such as software stack information representing the hierarchy of software nodes and the calling relationships between software nodes during model training, and hardware information representing the hierarchy of hardware nodes and the network connection relationships between hardware nodes in the training cluster; determining faulty nodes in the cross-stack graph during model training based on fault information and / or anomaly indicators; and determining fault propagation paths corresponding to the faulty nodes, used to determine the causes of faults and / or anomalies during model training, based on the cross-stack graph.

[0030] Fourthly, a computer-readable storage medium is provided that stores computer program instructions that, when executed by a processor, implement the method as described in any possible implementation of the first aspect.

[0031] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to implement the method as described in any possible implementation of the first aspect.

[0032] Sixthly, a chip system is provided, comprising processing circuitry and a storage medium storing computer program instructions; when executed by a processor, the computer program instructions implement the method as described in any possible implementation of the first aspect. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0033] Figure 1 A schematic diagram of the hardware structure of a model training fault analysis device provided in this application embodiment;

[0034] Figure 2 A flowchart of a model training fault analysis method provided in this application embodiment;

[0035] Figure 3 A schematic diagram illustrating cross-stack graph construction provided in an embodiment of this application;

[0036] Figure 4 A schematic diagram illustrating the acquisition of software stack information and hardware information as provided in an embodiment of this application;

[0037] Figure 5 A schematic diagram illustrating a cross-stack icon annotation provided in an embodiment of this application;

[0038] Figure 6 A schematic diagram illustrating the fault analysis process of training two models provided in the embodiments of this application;

[0039] Figure 7 A flowchart of another model training fault analysis method provided in the embodiments of this application;

[0040] Figure 8 A schematic diagram illustrating the acquisition of a fault propagation path as provided in an embodiment of this application;

[0041] Figure 9 This application provides a schematic diagram of a model training fault analysis process.

[0042] Figure 10 This is a structural block diagram of a model training fault analysis device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0044] In the following text, the terms "first," "second," etc., are used only to distinguish different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. For example, if the described object is a "label," then the ordinal numbers preceding "label" in "first label" and "second label" do not limit the position or order of the "labels," and "first" and "second" do not restrict the order of "first label" and "second label." Similarly, if the described object is a "fault subgraph," then the ordinal numbers preceding "fault subgraph" in "first fault subgraph" and "second fault subgraph" do not limit the priority of the "fault subgraphs." Furthermore, the number of described objects is not limited by ordinal numbers and can be one or more; for example, in "first fault node," the number of "fault nodes" can be one or more. In addition, objects modified by different prefixes can be the same or different; for example, if the described object is a "value," then "first value," "second value," "third value," "fourth value," "fifth value," "sixth value," and "seventh value" can be the same value or different values. In summary, the use of ordinal numbers and other prefixes used to distinguish the described objects in the embodiments of this application does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0045] Furthermore, in the embodiments of this application, "connection" can be a direct connection or an indirect connection; in addition, it can refer to an electrical connection or a communication connection; for example, the connection of two electrical components A and B can refer to A and B being directly connected, or it can refer to A and B being indirectly connected through other electrical components or connection media, or it can refer to A and B being indirectly connected through other communication devices or communication media, as long as it enables communication between A and B.

[0046] Models (such as machine learning models with large-scale parameters and complex computational structures, i.e., large models) can be trained in large-scale training cluster environments. As the scale of the training cluster expands, the probability of various faults increases, significantly increasing the probability of training interruptions. For example, hardware-level faults may include, but are not limited to, one or more of the following: ensemble communication failure, memory failure, ECC failure; software-level faults may include, but are not limited to, one or more of the following: operator version mismatch, operator precision mismatch; single-task-level faults may include, but are not limited to, one or more of the following: process deadlock, communication interruption, etc. In some examples, a single task can represent that the model has completed a specified number of iterations of training. For example, if the total number of iterations for the model is 5000, and the specified number of iterations can be 1000, then every 1000 iterations completed by the model can be considered as one completed single task.

[0047] To resume training, troubleshooting is typically performed. However, due to the diversity of faults and the potential correlation between different faults, the accuracy of fault node localization during troubleshooting can be low. This can lead to excessively long time required to resolve faulty nodes, reducing training efficiency and increasing training costs. For example, in the MagaScale (Scaling Large Language Model Training to More Than 10,000 GPUs) system, the occurrence of training faults can be determined by detecting abnormal heartbeat information or heartbeat timeouts. When a fault occurs, network diagnostic tests are performed to locate the faulty node, and then redundant nodes in the cluster are used to replace the faulty node to resume training. However, this method can only locate faulty nodes related to the network; it cannot accurately locate other types of faulty nodes, making it difficult to quickly resume training.

[0048] To address the issues of low training efficiency, low resource utilization in the training cluster, and high training costs, this application provides a model training fault analysis method. This method acquires fault information and / or anomaly indicators during model training in the training cluster. It constructs a cross-stack graph based on the hierarchy of software nodes and the calling relationships between them, as well as the hierarchy of hardware nodes and the network connections between them. It then identifies faulty nodes in the cross-stack graph based on the fault information and / or anomaly indicators. Finally, it determines the fault propagation path corresponding to the faulty node, used for analyzing the causes of faults and / or anomalies during model training, based on the cross-stack graph. This method can identify faulty nodes in the cross-stack graph based on fault information and / or abnormal indicators, and determine the fault propagation path corresponding to the faulty node based on the cross-stack graph. Since the cross-stack graph contains software stack information representing the level of software nodes and the calling relationship between software nodes during model training, as well as hardware information representing the level of hardware nodes and the network connection relationship between hardware nodes in the training cluster, it is possible to determine the root cause of the fault based on the fault propagation path. This is beneficial for quickly repairing faults, improving training efficiency and resource utilization of the training cluster, and reducing training costs.

[0049] The method provided in this application can be applied to devices or apparatuses capable of model training fault analysis, hereinafter referred to as "model training fault analysis apparatus". The model training fault analysis apparatus can perform model training fault analysis by means of hardware, by means of software, or by means of a combination of hardware and software.

[0050] In some embodiments, the model training fault analysis device may also have the ability to acquire fault information and / or abnormal indicators when the training cluster is training the model.

[0051] For example, the model training failure analysis device may include, but is not limited to, electronic devices such as smartphones, netbooks, tablets, personal computers (PCs), PDAs, and smart screen devices. This application does not limit the specific functions and structure of the model training failure analysis device.

[0052] The hardware structure diagram of the model training fault analysis device 100 can be shown as follows: Figure 1 As shown. In Figure 1 The mid-model training fault analysis device 100 may include a processor 110 and a memory 120.

[0053] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the model training fault analysis device 100. In other embodiments of this application, the model training fault analysis device 100 may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0054] The processor 110 may include one or more processing units, such as an application processor (AP), a microcontroller unit (MCU), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, the application processor may include a graphics processor and a digital signal processor, and the microcontroller unit may include a graphics processor.

[0055] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0056] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0057] In this embodiment of the application, the processor 110 is capable of determining the fault node in the cross-stack graph during model training based on fault information and / or abnormal indicators, determining the fault propagation path corresponding to the fault node based on the cross-stack graph, and determining the fault cause and / or abnormal cause during model training based on the fault propagation path.

[0058] In some examples, processor 110 can also be used to make a first annotation at the corresponding fault node in the cross-stack graph based on fault information; and / or, to make a second annotation at the corresponding fault node in the cross-stack graph based on anomaly indicators.

[0059] In some examples, processor 110 can also be used to determine neighboring nodes of a faulty node across a stack graph, where neighboring nodes are nodes connected to the faulty node within a preset hop count range; determine a fault subgraph based on the faulty node, neighboring nodes, call relationships, and / or network connection relationships; and determine a fault propagation path based on the fault subgraph.

[0060] In some examples, the fault subgraph includes multiple levels, and the processor 110 can also be used to start from the path start point in the fault subgraph, and search in the fault subgraph according to the calling relationship and / or network connection relationship between nodes in the fault subgraph until the path end point is found. The path start point is the fault node with the highest level in the fault subgraph; and obtain the fault propagation path according to the path start point, the calling relationship and / or network connection relationship between nodes involved in the search process, and the path end point.

[0061] Memory 120 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct read and write operations by the processor 110.

[0062] As an example, the program code stored in memory 120 includes, but is not limited to, one or more of the following: program code corresponding to the model (such as a large model) (denoted as code 1) and training fault analysis code (denoted as code 2). The processor 110 can perform fault analysis on code 1 by calling code 2 to obtain the fault propagation path in code 1.

[0063] In some embodiments, the model training fault analysis device 100 may be connected to a display screen. The display screen may be used to display an interface including program code corresponding to the model (e.g., a large model), an interface including a cross-stack graph, an interface including fault propagation paths, an interface including fault causes and / or anomaly causes, etc. In some examples, the model training fault analysis device 100 may include a display screen (…). Figure 1(Not shown), the model training fault analysis device 100 implements display functions through a GPU, display screen, application processor, microcontroller unit, etc. The GPU is a microprocessor for image processing, connected to the display screen, application processor, and microcontroller unit. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0064] In this embodiment, the model training fault analysis device 100 may be equipped with software that has one or more functions such as model (e.g., large model) training and editing of program code corresponding to the model (e.g., large model), as well as software or hardware capable of connecting to a training cluster. The model training fault analysis device 100 can display functions such as fault propagation path, fault cause, and / or abnormal cause through GPU, display screen, and application processor.

[0065] In some examples, the model training fault analysis device 100 can be connected to the display screen of other electronic devices, so that when the model training fault analysis device 100 acquires the cross-stack graph, fault propagation path, fault cause and / or abnormal cause, it can transmit the cross-stack graph, fault propagation path, fault cause and / or abnormal cause to the display screen of other electronic devices for display.

[0066] The following will describe in detail, with reference to the accompanying drawings, a model training fault analysis method provided in the embodiments of this application.

[0067] Please refer to Figure 2 , Figure 2 A flowchart of a model training fault analysis method provided in an embodiment of this application is shown. This method can be applied to a model training fault analysis device with model training fault analysis capabilities. Figure 2 As shown, the model training fault analysis method may include S201-S203:

[0068] S201: The model training fault analysis device acquires fault information and / or abnormal indicators when the training cluster is training the model.

[0069] As one possible approach, when a model (such as a large model) is trained on a training cluster, the model training failure analysis device can acquire and record the model's running information and / or acquire the model's training logs, thereby confirming whether there are any anomalies based on historical and current running information. If an anomaly is found, the device can acquire the corresponding anomaly indicators; and obtain error information from the model's training logs; and determine the failure information based on the error information.

[0070] In some examples, fault information may include, but is not limited to, fault node identifiers and fault details. Fault node identifiers can be used to indicate the faulty node. Fault details can be used to indicate the cause of the error for the corresponding faulty node. For example, the fault details for an AI accelerator card could be that the video memory is exhausted.

[0071] As an example, a model training failure analysis device can parse the acquired training logs and, upon parsing error information, extract the fault node identifier and fault content associated with that error information. In this way, by acquiring fault information including fault node identifiers, the model training failure analysis device can increase the sources of fault nodes, facilitating the discovery of more fault nodes and thus improving the accuracy of subsequently obtained fault propagation paths.

[0072] In some examples, historical operation information can refer to the model operation information collected by the model training fault analysis device within a preset time period prior to the current time. For example, assuming the preset time period is 30 minutes and the current time is 14:00, then historical operation information can refer to the model operation information collected by the model training fault analysis device between 13:30 and 14:00.

[0073] In some examples, the model training fault analysis method provided in this application can be applied to model training scenarios, such as scenarios where model training is interrupted. In such scenarios, abnormal indicators can indicate that there is a running fault in the model during training. The running fault can include, but is not limited to, faults that cause the model training to be interrupted, such as the network connection between hardware nodes being disconnected. The faulty node can be a node with a running fault.

[0074] In model training scenarios, model runtime information may include, but is not limited to, one or more of the following: AI accelerator card memory utilization rate, such as the proportion of GPU memory occupied in the AI ​​accelerator card; AI accelerator card computing power utilization rate, such as the ratio of the effective computing power of the GPU to its peak computing power; execution time of neural network layers, such as the time it takes for a neural network layer to perform a single data processing operation; frame function call time, such as the time taken from the start to the end of a frame function call; computation time of computation operators, such as the time it takes for a computation operator to complete a single computation; and training time for a single task. Abnormal indicators may include, but are not limited to, one or more of the following: abnormal AI accelerator card memory utilization rate, abnormal AI accelerator card computing power utilization rate, abnormal single-layer execution time of neural network layers, abnormal frame function call time, abnormal computation time of computation operators, and abnormal training time for a single task.

[0075] In model training scenarios, the model training fault analysis device can confirm the existence of corresponding abnormal indicators in different ways. For the same abnormal indicator, the device can also confirm its existence in multiple ways, which is not limited in this application. Thus, by acquiring abnormal indicators from different nodes, the model training fault analysis device can enrich the types of fault nodes, facilitating the acquisition of more accurate fault propagation paths and uncovering potential correlations between different types of fault nodes, thereby quickly determining the root cause of operational failures.

[0076] As an example, the model training fault analysis device can obtain the memory usage rate of the AI ​​accelerator card in the training cluster. If the absolute value of the difference between the memory usage rate of the AI ​​accelerator card and the preset memory usage rate of the AI ​​accelerator card is greater than or equal to a first value, the memory usage rate of the AI ​​accelerator card is taken as an abnormal indicator.

[0077] In some examples, the preset memory utilization rate of the AI ​​accelerator card can be the average of its historical memory utilization rates. Historical memory utilization rate can refer to the memory utilization rate of the AI ​​accelerator card obtained by the model training failure analysis device within a preset time period prior to the current moment. The preset memory utilization rate of the AI ​​accelerator card can also be set based on other rules. For example, the preset memory utilization rate of the AI ​​accelerator card can be a predetermined fixed value. The same applies to the following aspects: the computing power utilization rate of the AI ​​accelerator card, the execution time of neural network layers, the call time of framework functions, the computation time of computation operators, and the training time of a single task. This application does not impose any limitations on these aspects.

[0078] In some examples, the first value can be set by the developer based on experience.

[0079] For example, assuming the first value is 20%, the memory usage of the AI ​​accelerator card identified as H065 at the current moment is 35%, and the historical memory usage of the AI ​​accelerator card includes 50%, 60%, and 70%, then the preset memory usage of the AI ​​accelerator card is (50% + 60% + 70%) / 3 = 60%. Since the absolute value of the difference between the memory usage of the AI ​​accelerator card and the preset memory usage (|35% - 60%| = 25%) is greater than the first value, it is determined that the memory usage of the AI ​​accelerator card is abnormal.

[0080] As another example, the model training fault analysis device can obtain the computing power utilization rate of the AI ​​accelerator card in the training cluster. If the absolute value of the difference between the computing power utilization rate of the AI ​​accelerator card and the preset computing power utilization rate of the AI ​​accelerator card is greater than or equal to a second value, the computing power utilization rate of the AI ​​accelerator card is taken as an abnormal indicator.

[0081] In some examples, the preset computing power utilization rate of the AI ​​accelerator card can be the average of the historical computing power utilization rate of the AI ​​accelerator card. The historical computing power utilization rate can refer to the computing power utilization rate of the AI ​​accelerator card obtained by the model training fault analysis device within a preset time period before the current moment.

[0082] In some examples, the second value can be set by the developer based on experience.

[0083] For example, assuming the second value is 10%, the computing power utilization rate of the AI ​​accelerator card with the identifier H065 obtained at the current moment is 30%. The computing power utilization rate of the AI ​​accelerator card includes 50%, 60%, and 70%. Then the preset computing power utilization rate of the AI ​​accelerator card is (50% + 60% + 70%) / 3 = 60%. Since the absolute value of the difference between the computing power utilization rate of the AI ​​accelerator card and the preset computing power utilization rate (|30% - 60%| = 30%) is greater than the second value, it is determined that the computing power utilization rate of the AI ​​accelerator card is abnormal.

[0084] As another example, the model training failure analysis device can obtain the execution time of the neural network layer of the model. If the difference between the execution time of the neural network layer and the preset execution time of the neural network layer is greater than or equal to a third value, the single-layer execution time of the neural network layer is used as an abnormal indicator.

[0085] In some examples, the preset execution time of a neural network layer can be the average of its historical execution times. Historical execution time can refer to the execution time of the neural network layer obtained by the model training fault analysis device within a preset time period prior to the current moment.

[0086] In some examples, the third value can be set by the developer based on experience.

[0087] As another example, the model training failure analysis device can obtain the call duration of the model's frame function. If the difference between the call duration of the frame function and the preset call duration of the frame function is greater than or equal to a fourth value, the call duration of the frame function is used as an abnormal indicator.

[0088] In some examples, the preset call duration of the frame function can be the average of the historical call durations of the frame function. The historical call duration can refer to the call duration of the frame function obtained by the model training fault analysis device within a preset time period before the current moment.

[0089] In some examples, the fourth value can be set by the developer based on experience.

[0090] As another example, the model training fault analysis device can obtain the computation time of the model's computation operator. If the difference between the computation time of the computation operator and the preset computation time of the computation operator is greater than or equal to a fifth value, the computation time of the computation operator is used as an abnormal indicator.

[0091] In some examples, the preset computation time of the computation operator can be the average of the historical computation times of the computation operator. The historical computation time can refer to the computation time of the computation operator obtained by the model training fault analysis device within a preset time period before the current moment.

[0092] In some examples, the fifth value can be set by the developers based on experience. The aforementioned acquisition of anomaly indicators is for a single hardware node or a single software node. In this application, in order to determine the overall training status of the model, the model training failure analysis device can confirm the existence of anomaly indicators representing overall training anomalies in different ways, and this application does not limit them here.

[0093] As an example, the model training failure analysis device can obtain the training time of a single task of the model. If the difference between the training time of a single task and the preset training time of a single task is greater than or equal to the seventh value, the training time of the single task will be used as an abnormal indicator.

[0094] In some examples, the preset training duration for a single task can be the average of the historical training durations for that single task. The historical training duration can refer to the training duration acquired by the model training the fault analysis device within a preset time period prior to the current moment.

[0095] In some examples, the seventh value can be set by the developers based on experience.

[0096] In some examples, the model training failure analysis device can periodically acquire and record the model's running information, as well as the model's training logs.

[0097] In some examples, the model training fault analysis method provided in this application can be applied to model performance analysis scenarios, such as scenarios where the model training process is not interrupted but the training speed decreases. In this scenario, abnormal indicators can indicate that the model has performance faults during training. Performance faults can include, but are not limited to, faults that cause the model training speed to decrease, and fault nodes can be nodes with performance faults.

[0098] In model performance analysis scenarios, model runtime information may include, but is not limited to, the execution time of nodes within the model. Abnormal metrics may include, but are not limited to, abnormal node execution times.

[0099] As an example, a model training failure analysis device can obtain the execution time of nodes in the model. If the difference between the execution time of a node and the preset execution time corresponding to that node is greater than or equal to a sixth value, the execution time of that node is taken as an anomaly indicator. Thus, in the scenario of model performance analysis, by obtaining anomaly indicators of different nodes, the model training failure analysis device can enrich the types of faulty nodes, so as to obtain more accurate fault propagation paths in the future, thereby uncovering the potential correlation between different types of faulty nodes and quickly determining the root cause of performance failures.

[0100] In some examples, the execution time of a node can refer to the time required for the node to perform one operation, or the time required for the node to perform one single task. The preset execution time for a node can be the average of the node's historical execution times. Historical execution time can refer to the execution time of the node obtained by the model training fault analysis device during the model training process before the current moment.

[0101] In some examples, the sixth value can be set by the developers based on experience.

[0102] In some examples, when there are multiple nodes, there can also be multiple sixth values. Each node can correspond to a sixth value, and the sixth values ​​corresponding to different nodes can be the same or different. The model training fault analysis device can periodically obtain the execution time of each node in the model. If the difference between the execution time of a node and the preset execution time corresponding to that node is greater than or equal to the fourth value corresponding to that node, the execution time of that node is taken as an abnormal indicator.

[0103] In some examples, the model training failure analysis device can periodically acquire and record the model's operational information.

[0104] S202: The model training fault analysis device determines the fault node in the cross-stack graph during model training based on fault information and / or abnormal indicators.

[0105] As an example, such as Figure 3 As shown, during model training, the model training fault analysis device can collect software stack information and hardware information, and construct a cross-stack graph based on the collected software stack information and hardware information. The cross-stack graph can include software layers and hardware layers.

[0106] In some examples, software stack information represents the hierarchy of software nodes and the calling relationships between them during model training. For example, software stack information may include, but is not limited to, the model call stack, computation graph structure, and operator data format. Hardware information represents the hierarchy of hardware nodes in the training cluster and the network connections between them. For example, hardware information may include, but is not limited to, the distribution of AI accelerator card clusters, AI accelerator card models, and cluster network structure.

[0107] In some examples, software nodes include, but are not limited to, one or more of the following: neural network layers of the model, acceleration components, frame functions, logical operators, and computation operators. Neural network layers may include, but are not limited to, convolutional layers and fully connected layers; acceleration components may include, but are not limited to, pipelined acceleration components; frame functions may include, but are not limited to, matrix multiplication, vector multiplication, and activation functions; logical operators may include, but are not limited to, operators that perform AND, OR, NOT, and other operations; and computation operators may include, but are not limited to, operators used to perform addition, subtraction, multiplication, division, and other operations.

[0108] In some examples, a software layer includes the hierarchy corresponding to a software node. This hierarchy includes, but is not limited to, one or more of the following: neural network structure layer, acceleration framework layer, basic AI framework layer, computation layer, and operator layer. Software nodes in the network structure layer may include, but are not limited to, neural network layers; software nodes in the acceleration framework layer may include, but are not limited to, acceleration components; software nodes in the basic AI framework layer may include, but are not limited to, framework functions; software nodes in the computation layer may include, but are not limited to, logical operators; and software nodes in the operator layer may include, but are not limited to, computation operators.

[0109] In some examples, the hardware layer includes a hierarchy corresponding to hardware nodes, which include, but are not limited to, one or more of the following: AI accelerator cards for the training cluster, switches, optical modules, and network interfaces. AI accelerator cards, for example, are hardware components specifically designed to accelerate artificial intelligence tasks, typically equipped with high-performance GPUs, NPUs, or other types of processors optimized for AI computing.

[0110] In some examples, the hardware nodes correspond to layers including, but not limited to, one or more of the following: AI accelerator card runtime layer and network topology layer. Hardware nodes in the AI ​​accelerator card runtime layer may include, but are not limited to, AI accelerator cards, and hardware nodes in the network topology layer may include, but are not limited to, switches, optical modules, and network interfaces.

[0111] In some examples, such as Figure 4 As shown, the model training fault analysis device can set data probes in the model (such as a large model) to collect software stack information and hardware information during model training.

[0112] In some examples, data probes can be used to detect and analyze data. By setting up data probes, the model training failure analysis device can delve into the dataset, data storage system, running data processing flow, etc., to obtain software stack information and hardware information.

[0113] As one possible approach, the model training fault analysis device can perform a first annotation at the corresponding fault node in the cross-stack graph based on fault information; and / or, perform a second annotation at the corresponding fault node in the cross-stack graph based on anomaly indicators. In this way, the model training fault analysis device annotates fault nodes in the cross-stack graph using fault information and / or anomaly indicators, which facilitates the rapid and accurate acquisition of the fault propagation path based on the fault nodes in the cross-stack graph, thereby determining the cause of the fault and / or the cause of the anomaly during model training based on the fault propagation path.

[0114] In some examples, the first annotation can distinguish a node with fault information from other nodes in the cross-stack graph that do not exhibit faults or anomalies, for example, by changing the color, shape, or size of the node with fault information; the second annotation can distinguish a node with anomaly indicators from other nodes in the cross-stack graph that do not exhibit faults or anomalies, for example, by changing the color, shape, or size of the node with anomaly indicators. The first and second annotations can be the same or different, and this is not limited here.

[0115] In some examples, each node in the cross-stack graph can have corresponding attribute information. The model training fault analysis device can add fault information to the attribute information of the corresponding fault node, as well as add abnormal indicators to the attribute information of the corresponding fault node.

[0116] For example, a cross-stack graph can be like this: Figure 5 As shown in the left figure, the fault information indicates that the logic operator marked as 22 and the computation operator marked as 3 are faulty nodes. The abnormal indicators indicate that the AI ​​accelerator card marked as 23 is a faulty node. The model training fault analysis device performs the first annotation on the logic operator marked as 22 and the computation operator marked as 3 in the cross-stack graph, such as changing the node shape of logic operator 22 and computation operator 3 to a square. It performs the second annotation on the AI ​​accelerator card marked as 23 in the cross-stack graph, such as changing the node shape of AI accelerator card 23 to a triangle. Figure 5 The right figure shows a cross-stack diagram.

[0117] S203: The model training fault analysis device determines the fault propagation path corresponding to the fault node based on the cross-stack graph. The fault propagation path is used to determine the fault cause and / or abnormal cause during model training.

[0118] One possible approach is for the model training failure analysis device to identify the neighboring nodes of a faulty node across the stack graph. Neighboring nodes are those connected to the faulty node within a preset hop count range. Based on the faulty node, neighboring nodes, call relationships, and / or network connectivity, a fault subgraph is determined. Finally, the failure propagation path is determined from the fault subgraph. In this way, the model training failure analysis device can change the process of obtaining the failure propagation path from the entire stack graph to obtaining it from the fault subgraph, narrowing the search range for failure propagation paths, improving the efficiency of failure propagation path acquisition, and consequently improving the model training efficiency and the resource utilization of the training cluster, while reducing training costs.

[0119] In some examples, the preset hop count range can be determined based on the number of nodes in the stack graph and a preset value X. For example, when the number of nodes is less than or equal to X, the preset hop count range can be A1 hops or less (e.g., A1 could be 3, etc., without limitation); when the number of nodes is greater than X, the preset hop count range can be A2 hops or less (e.g., A1 could be 5, etc., without limitation). For example, A2 is greater than A1.

[0120] In some examples, neighboring nodes may include one or more of the following: one-hop neighboring nodes, two-hop neighboring nodes, ..., N-hop neighboring nodes, where N can represent the maximum number of hops included in a preset range. For example, N may be A1 or A2, etc., without limitation.

[0121] In some examples, a one-hop neighbor node can refer to a neighbor node directly connected to the faulty node via an edge, a two-hop neighbor node can refer to a neighbor node connected to the faulty node via two edges, and so on. For example, if faulty node A is connected to node B via edge 1, then node B is a one-hop neighbor node of faulty node A; if faulty node A is connected to node B via edge 1, and node B is connected to node C via edge 2, then node C is a two-hop neighbor node of faulty node A.

[0122] As an example, the model training fault analysis device can start from the faulty node and search across the stack graph according to the call relationship and / or network connection relationship corresponding to the faulty node to obtain the neighboring nodes of the faulty node.

[0123] As an example, the model training fault analysis device can start from the fault node at the highest level in the fault subgraph, search according to the call relationship and / or network connection relationship corresponding to the fault node, and determine the fault propagation path based on the search results.

[0124] As one possible approach, after obtaining the fault propagation path, the model training fault analysis device can determine the cause of the fault and / or the cause of the anomaly during model training based on the fault propagation path and preset analysis rules. For example, such as... Figure 6 As shown in (a), when the model is trained on the training cluster, the model training fault analysis device can collect software stack information and hardware information, and then construct a cross-stack graph based on the software stack information and hardware information; the model training fault analysis device can also obtain fault information and / or abnormal indicators, and then search the fault subgraph based on the fault information and / or abnormal indicators to obtain the fault propagation path; after obtaining the cross-stack graph and the fault propagation path, the model training fault analysis device can determine the fault cause and / or abnormal cause during model training based on the fault propagation path and preset analysis rules.

[0125] In some examples, preset analysis rules can be set by developers based on their own experience.

[0126] As another possible implementation, after obtaining the fault propagation path, the model training fault analysis device can display the cross-stack graph and the fault propagation path, allowing developers to determine the causes of faults and / or anomalies during model training based on the cross-stack graph and the fault propagation path. For example, such as... Figure 6 As shown in (b), when the model is trained on the training cluster, the model training fault analysis device can collect software stack information and hardware information, and then construct a cross-stack graph based on the software stack information and hardware information; the model training fault analysis device can also obtain fault information and / or abnormal indicators, and then search the fault subgraph based on the fault information and / or abnormal indicators to obtain the fault propagation path; after obtaining the cross-stack graph and the fault propagation path, the model training fault analysis device can display the cross-stack graph and the fault propagation path so that developers can determine the fault cause and / or abnormal cause during model training based on the cross-stack graph and the fault propagation path.

[0127] In some embodiments, based on Figure 6 (a) or Figure 6 After determining the cause of the fault and / or the cause of the anomaly using the method shown in (b), developers can optimize the analysis rules based on the cause of the fault and / or the cause of the anomaly, and store the optimized analysis rules in the model training fault analysis device. Optionally, developers can also store the cause of the fault and / or the cause of the anomaly in the model training fault analysis device for reference when automatically performing fault cause and / or anomaly cause analysis later, or for reference when developers analyze fault cause and / or anomaly cause analysis later.

[0128] In some embodiments, based on Figure 6After determining the fault cause and / or anomaly cause using the method shown in (a), the model training fault analysis device can optimize the analysis rules in the model training fault analysis device based on the fault cause and / or anomaly cause. For example, assuming the model training fault analysis device includes a pre-trained rule reasoning model, the model training fault analysis device can input one or more of the fault propagation path, fault subgraph, fault cause, and anomaly cause into the pre-trained rule reasoning model to obtain the analysis rules corresponding to the fault cause and / or anomaly cause. Optionally, the model training fault analysis device can also store the fault cause and / or anomaly cause in the model training fault analysis device for reference during subsequent automatic fault cause and / or anomaly cause analysis.

[0129] In this embodiment, the model training fault analysis device can determine the faulty node in the cross-stack graph based on fault information and / or abnormal indicators, and determine the fault propagation path corresponding to the faulty node based on the cross-stack graph. Since the cross-stack graph contains software stack information representing the level of software nodes and the calling relationship between software nodes during model training, and hardware information representing the level of hardware nodes and the network connection relationship between hardware nodes in the training cluster, the model training fault analysis device or developers can analyze the potential correlation between multiple faulty nodes based on the fault propagation path (e.g., a fault in node A causes a fault in node B), thereby determining the root cause of the fault, which is beneficial for quickly repairing the fault, improving training efficiency and resource utilization in the training cluster, and reducing training costs.

[0130] The following will, with reference to the accompanying drawings and taking a model training scenario as an example, provide a detailed description of the aspects related to determining the fault propagation path in a model training fault analysis method provided in this application embodiment.

[0131] like Figure 7 As shown, an embodiment of this application provides a model training fault analysis method that may include steps S701-S706:

[0132] S701: The model training fault analysis device acquires fault information and / or abnormal indicators when the training cluster is training the model.

[0133] For information on obtaining fault information and / or abnormal indicators, please refer to the above text. Figure 2 The introduction of S201 will not be repeated.

[0134] S702: The model training fault analysis device determines the fault node in the cross-stack graph during model training based on fault information and / or abnormal indicators.

[0135] For information on identifying faulty nodes across stack graphs, please refer to the section above. Figure 2The introduction of S202 will not be repeated.

[0136] S703: The model training fault analysis device determines the neighboring nodes of the faulty node in the cross-stack graph. The neighboring nodes are the nodes connected to the faulty node within a preset number of hops.

[0137] In some examples, the faulty node may include, but is not limited to, a first faulty node and a second faulty node. The first faulty node may include one or more faulty nodes, and the second faulty node may include one or more nodes; this application does not impose any limitations on this.

[0138] As one possible implementation, the model training fault analysis device can start from a first fault node and search in the cross-stack graph according to the call relationship and / or network connection relationship corresponding to the first fault node to obtain the neighbor nodes of the first fault node; and start from a second fault node and search in the cross-stack graph according to the call relationship and / or network connection relationship corresponding to the second fault node to obtain the neighbor nodes of the second fault node.

[0139] S704: The model training fault analysis device determines the fault subgraph based on the fault node, neighbor nodes, call relationships and / or network connection relationships.

[0140] As one possible implementation, the model training fault analysis device determines a first fault subgraph corresponding to the first fault node based on the first fault node, the neighboring nodes corresponding to the first fault node, the call relationship, and / or the network connection relationship; and determines a second fault subgraph corresponding to the second fault node based on the second fault node, the neighboring nodes corresponding to the second fault node, the call relationship, and / or the network connection relationship; if the first fault subgraph and the second fault subgraph are connected, the model training fault analysis device can determine a fault subgraph based on the first fault subgraph and the second fault subgraph; if the first fault subgraph and the second fault subgraph are not connected, the model training fault analysis device can determine a fault subgraph based on the first fault subgraph, the second fault subgraph, and auxiliary nodes, where the auxiliary nodes are nodes in the cross-stack graph that can connect the first fault subgraph and the second fault subgraph.

[0141] In some examples, the first and second fault subgraphs being connected means that any node in the first fault subgraph can be connected to any node in the second fault subgraph by one or more edges. When the first and second fault subgraphs are connected, the first and second fault subgraphs may have overlapping portions. The first and second fault subgraphs being disconnected means that no node in the first fault subgraph is connected to any node in the second fault subgraph.

[0142] In the embodiments of this application, the model training fault analysis device can increase the number of nodes in the fault subgraph by merging the first fault subgraph and the second fault subgraph into a single fault subgraph, thereby facilitating the discovery of the correlation between multiple fault nodes and obtaining a more comprehensive and accurate fault propagation path.

[0143] S705: The model training fault analysis device starts from the path starting point in the fault subgraph and searches the fault subgraph according to the calling relationship and / or network connection relationship between nodes in the fault subgraph until the path ending point is found. The path starting point is the fault node with the highest level in the fault subgraph.

[0144] In some examples, the fault subgraph may include, but is not limited to, software and hardware layers. The path start point may be in a software layer and the path end point may also be in a software layer, or the path start point may be in a software layer and the path end point may be in a hardware layer. In this way, the model training fault analysis device can use the fault subgraph to find fault propagation paths at the software level, or fault propagation paths combining the software and hardware levels. Furthermore, it can analyze these fault propagation paths to identify software faults during model training in the training cluster, or faults related to the interrelation of software and hardware, thereby improving the accuracy of fault and / or anomaly cause analysis during model training.

[0145] In some examples, the starting point of the path can be the fault node at the highest level in the fault subgraph. In this way, the model training fault analysis device can start from the highest level fault node in the fault subgraph and search down to lower levels, thereby obtaining a more complete fault propagation path, which helps to improve the accuracy of fault cause and / or anomaly cause analysis during model training.

[0146] As one possible approach, the model training fault analysis device can start from the path origin and search in the software layer according to the calling relationship between nodes in the fault subgraph until the software layer endpoint is found; if the software layer endpoint is located at a non-lowest level in the software layer, the software layer endpoint is taken as the path endpoint.

[0147] In this embodiment, since nodes that are not at the lowest level in the software layer cannot directly call hardware nodes, the model training fault analysis device can determine that there is a software fault when the training cluster performs model training, so that there is no need to troubleshoot hardware faults, thus improving the efficiency of fault analysis.

[0148] As an example, when searching for the endpoint of a software layer, the model training fault analysis device can start from the path start point and search in the software layer according to the calling relationship between nodes in the fault subgraph. If the current search node has a first label and / or a second label, the model training fault analysis device can start searching for a one-hop neighbor node that has not been searched from the search node until the endpoint of the software layer is found. The one-hop neighbor node is at the same level as the search node or the neighbor node is at the next level below the level of the search node. If the current search node does not have a first label and a second label, and the search node is connected to a fault edge, the model training fault analysis device can take another node connected to the fault edge as a new search node, and start searching for a one-hop neighbor node that has not been searched from the new search node until the endpoint of the software layer is found.

[0149] In this embodiment, the model training fault analysis device searches the software layer using a hop-by-hop search method. This ensures that no one-hop neighbor node corresponding to a search node is missed, thereby improving the granularity of the search and ultimately enhancing the accuracy of fault propagation paths in the software layer. Furthermore, when the current search node has not been labeled with its first and second annotations, the model training fault analysis device confirms the connection between the search node and the fault edge, uncovering potential associations between the search node and its one-hop neighbor nodes, such as associations between multiple fault nodes. This further enhances the granularity of the search and improves the accuracy of fault propagation paths in the software layer.

[0150] As an example, when searching for the endpoint of a software layer, the model training fault analysis device can start from the path start point in the fault subgraph. If the current search node has not been labeled with the first and second labels and is not connected to the fault edge, the search node is taken as the path endpoint.

[0151] In this embodiment, the model training fault analysis device stops searching the software layer when the current search node has not been labeled with the first and second labels and the search node is not connected to the fault edge. This can obtain a more complete fault propagation path in the software layer, which is beneficial to improving the accuracy of fault cause and / or abnormal cause analysis during model training.

[0152] In some examples, a fault edge is a call relationship and / or network connectivity relationship between at least one end of a faulty node. For instance, a faulty node might be determined based on call relationships between software nodes and / or network connectivity relationships between hardware nodes.

[0153] As one possible approach, when a fault or anomaly occurs in the call relationship and / or network connection relationship between nodes, the model training fault analysis device can collect the corresponding fault information and / or anomaly indicators, obtain the corresponding fault node based on the fault information and / or anomaly indicators, and then treat the call relationship and / or network connection relationship involved in the corresponding fault node as a fault edge in the cross-stack graph.

[0154] In some examples, when a call relationship and / or network connection relationship fails / abnormalizes, the model training failure analysis device may designate the node at one end of the call relationship and / or network connection relationship that fails / abnormalizes as the fault node, or it may designate the nodes at both ends of the call relationship and / or network connection relationship that fails / abnormalizes as the fault node.

[0155] For example, assuming the fault information indicates a fault / abnormality in the network connection relationship between Switch_1 and Switch_2, the model training fault analysis device can collect fault information and / or abnormal indicators related to the network connection relationship between Switch_1 and Switch_2. Based on the collected fault information and / or abnormal indicators, it determines the fault node in the cross-stack graph as the node corresponding to Switch_1, or the node corresponding to Switch_2, or the node corresponding to both Switch_1 and Switch_2. Then, it takes the edge connecting Switch_1 and Switch_2 as the fault edge.

[0156] As another possible implementation, the model training fault analysis device can start from the path start point and search in the software layer according to the calling relationship between nodes in the fault subgraph until the software layer end point is found. If the software layer end point is located at the lowest level in the software layer, the model training fault analysis device can start from the software layer end point and search in the hardware layer according to the network connection relationship between nodes in the fault subgraph until the hardware layer end point is found, and use the hardware layer end point as the path end point.

[0157] In this embodiment, since the lowest-level node in the software layer can directly call the hardware node, the model training fault analysis device starts from the end point of the software layer and searches to the end point of the path in the hardware layer. It can determine that there are software and hardware related faults when the training cluster is training the model, so that there is no need to investigate pure software faults and pure hardware faults, thus improving the efficiency of fault analysis.

[0158] As an example, when searching for the endpoint of a software layer, the model training fault analysis device can start from the path start point and search in the software layer according to the calling relationship between nodes in the fault subgraph. If the current search node has a first label and / or a second label, the model training fault analysis device can start searching for a one-hop neighbor node that has not been searched from the search node until the endpoint of the software layer is found. If the current search node does not have a first label and a second label, and the search node is connected to a fault edge, the model training fault analysis device can take another node connected to the fault edge as a new search node, and start searching for a one-hop neighbor node that has not been searched from the new search node until the endpoint of the software layer is found.

[0159] As an example, when searching for the endpoint of a software layer, if the current search node is located at the lowest level in the software layer, the model training fault analysis device can use the search node as the endpoint of the software layer.

[0160] In this embodiment, the model training fault analysis device stops searching the software layer when the current search node is located at the lowest level in the software layer. This allows for a more complete fault propagation path in the software layer, which is beneficial for improving the accuracy of fault cause and / or anomaly cause analysis during model training.

[0161] In some examples, the lowest level in a software layer can be an operator layer.

[0162] As an example, the model training fault analysis device starts from the endpoint of the software layer and searches in the hardware layer according to the network connection relationship between nodes in the fault subgraph until the endpoint of the hardware layer is found. This includes: if the model training fault analysis device determines that the endpoint of the software layer is located at the lowest level in the software layer, taking the fault node of the hardware layer connected to the endpoint of the software layer as the starting point of the hardware layer; starting from the starting point of the hardware layer, searching in the hardware layer according to the network connection relationship between nodes in the fault subgraph until the endpoint of the hardware layer is found.

[0163] In some examples, the model training fault analysis device can obtain the starting point of the hardware layer based on the cross-layer call relationship corresponding to the endpoint of the software layer.

[0164] In this embodiment, since the lowest-level node in the software layer can directly call the hardware node, the model training fault analysis device starts from the end point of the software layer and searches in the hardware layer according to the network connection relationship between nodes in the fault subgraph until the end point of the path is found. This can determine that there is a software and hardware related fault when the training cluster is training the model, so that there is no need to investigate pure software faults and pure hardware faults, thus improving the efficiency of fault analysis.

[0165] As an example, the model training fault analysis device starts from the starting point of the hardware layer and searches the hardware layer according to the network connection relationship between nodes in the fault subgraph until the end point of the hardware layer is found. This includes: the model training fault analysis device obtains the connection path corresponding to the starting point of the hardware layer according to the network connection relationship between nodes in the fault subgraph; if there are fault nodes and / or fault edges in the connection path, the model training fault analysis device takes the end point of the connection path as the end point of the hardware layer.

[0166] In some examples, the deployment of hardware nodes in the training cluster is relatively fixed compared to the deployment of software nodes in the model. This results in relatively fixed network connections between hardware nodes, and the connection paths determined based on these connections are also relatively fixed. This allows the model training fault analysis device to determine the connection path corresponding to the starting point of the hardware layer based on the network connections between nodes in the fault subgraph. Then, by checking the connection paths, the fault propagation path in the hardware layer can be obtained. This eliminates the need to obtain the fault propagation path in the hardware layer by searching hop-by-hop in the software layer, saving the computing resources of the model training fault analysis device and improving the efficiency of fault propagation path acquisition.

[0167] For example, assuming the starting point of the hardware layer is node A, based on the network connection relationship between node A and other hardware nodes, the connection path corresponding to node A is determined to include path 1: node A-node B-node C, and node B is a fault node. Then the model training fault analysis device can take the end point of path 1 (node ​​3) as the end point of the hardware layer, which is equivalent to path 1 being the fault propagation path in the hardware layer.

[0168] In some examples, when the starting point of the hardware layer is a fault node, the model training fault analysis device can use the endpoints of each connection path corresponding to the starting point of the hardware layer as the endpoint of the hardware layer.

[0169] In some examples, when the starting point of the hardware layer is a non-fault node, the endpoint of the connection path corresponding to the starting point of the hardware layer, which contains a fault node and / or a faulty edge, can be used as the endpoint of the hardware layer.

[0170] In some examples, where there are multiple connection paths corresponding to the starting point of the hardware layer, the model training fault analysis device can determine whether there are fault nodes and / or fault edges in each connection path. If there are fault nodes and / or fault edges in the connection path, the model training fault analysis device takes the end point of the connection path as the end point of the hardware layer.

[0171] In some examples, where there are no fault edges or other fault nodes besides the starting point of the hardware layer in the connection path from the starting point of the hardware layer, the model training fault analysis device can take the starting point of the hardware layer as the ending point of the hardware layer.

[0172] S706: The model training fault analysis device obtains the fault propagation path based on the path start point, the calling relationship and / or network connection relationship between nodes involved in the search process, and the path end point.

[0173] As one possible approach, the model-trained fault analysis device can generate a fault propagation path by concatenating the path start point, the nodes involved in the search process and / or the network connection relationship between the nodes involved in the search process, and the path end point.

[0174] For example, such as Figure 8 As shown in the left figure, assuming there are multiple fault nodes in the fault subgraph, such as node 22 of the computation layer, node 3 of the operator layer, and node 23 of the AI ​​accelerator card runtime layer, the path starting point can be node 22 of the computation layer. The model training fault analysis device searches the fault subgraph starting from the path starting point, for example, by searching using the method in step S705, and obtains... Figure 8 The fault propagation path shown in the right figure includes, for example, node 22 of the computation layer, node 2 of the operator layer, node 3 of the operator layer, node 14 of the AI ​​accelerator card runtime layer, node 23 of the AI ​​accelerator card runtime layer, node 14 of the network topology layer, node 24 of the network topology layer, and node 32 of the network topology layer.

[0175] In some examples, when there are multiple fault nodes at the highest level (i.e., multiple path start points) in the fault subgraph, the model training fault analysis device can obtain the path end points corresponding to each of the multiple path start points to obtain multiple fault propagation paths.

[0176] As one possible implementation, the model training fault analysis device can start from each path origin and search in the software layer according to the calling relationship between nodes in the fault subgraph until the software layer endpoint corresponding to each path origin is found. If the software layer endpoint corresponding to a path origin is located at a non-lowest level in the software layer, the model training fault analysis device takes the software layer endpoint corresponding to the path origin as the path endpoint. If the software layer endpoint corresponding to a path origin is located at the lowest level in the software layer, the fault node of the hardware layer connected to the software layer endpoint is taken as the hardware layer origin. Starting from the hardware layer origin, the connection path corresponding to the hardware layer origin is obtained according to the network connection relationship between nodes in the fault subgraph. If there are fault nodes and / or fault edges in the connection path, the model training fault analysis device takes the endpoint of the connection path as the hardware layer endpoint.

[0177] In some examples, when there are fault nodes and / or fault edges in multiple connection paths, the model training fault analysis device can use the endpoints of the above multiple connection paths as the endpoints of the hardware layer, so that there are multiple path endpoints in the fault subgraph. In the subsequent process, the model training fault analysis device can concatenate the fault propagation paths in the software layer with the above multiple connection paths to obtain multiple fault propagation paths.

[0178] For example, suppose the fault propagation path in the software layer is: fault node A - non-fault node B - fault node C - fault node D at the lowest level in the software layer (hereinafter referred to as the software layer endpoint D). Based on the software layer endpoint D, the starting point of the hardware layer is determined to be non-fault node E (hereinafter referred to as the hardware layer starting point E). The hardware layer starting point E corresponds to connection path 1 and connection path 2. For example, connection path 1 is: hardware layer starting point E - fault node F, and connection path 2 is: hardware layer starting point E - fault node G. Then the model training fault analysis device can take the endpoints of connection path 1 and connection path 2 as path endpoints, such as fault node F and fault node G, and concatenate the fault propagation path in the software layer with connection path 1 and connection path 2 respectively to obtain two fault propagation paths. For example, fault propagation path 1 is: fault node A - non-fault node B - fault node C - software layer endpoint D - hardware layer starting point E - fault node F, and fault propagation path 2 is: fault node A - non-fault node B - fault node C - software layer endpoint D - hardware layer starting point E - fault node G.

[0179] In this embodiment, the model training fault analysis device searches in the software layer in a hop-by-hop manner, for example, searching for the one-hop neighbor nodes of the current search node, and searches in the hardware layer in a path-by-path manner, for example, searching for the connection path of the starting point of the hardware layer. This can improve the precision of the software layer search and save the computing resources of the model training fault analysis device without affecting the accuracy of the hardware layer search, thereby improving the search efficiency of the fault propagation path.

[0180] To better understand the solutions in the embodiments of this application, the following uses a model training scenario as an example to introduce the solutions involved in the embodiments of this application.

[0181] Please see Figure 9 ,when Figure 9When the model is trained in the training cluster, the model training fault analysis device can collect full-stack information of software and hardware, obtaining software stack information and hardware information. For example, software stack information includes, but is not limited to, model call stack, computation graph structure, operator data format, etc., and hardware information includes, but is not limited to, AI accelerator card cluster distribution, AI accelerator card model, cluster network structure, etc.; and the model training fault analysis device can obtain fault information and / or abnormal indicators during model training in the training cluster. Then, the model training fault analysis device can construct a cross-stack graph based on the software stack information and hardware information, and obtain the fault propagation path in the cross-stack graph based on the fault information and / or abnormal indicators. For example, the model training fault analysis device can determine the fault node in the cross-stack graph based on the fault information and / or abnormal indicators, further determine the fault subgraph corresponding to the fault node, and obtain the fault propagation path based on the fault subgraph; afterwards, the model training fault analysis device can display the cross-stack graph and fault propagation path, or determine the fault cause and / or abnormal cause during model training based on the fault propagation path and preset analysis rules.

[0182] Figure 10 This is a schematic diagram of the structure of a possible model training fault analysis device provided in an embodiment of this application. Figure 10 As shown, the model training fault analysis device 1000 includes: a processing module 1001, an information acquisition module 1002, and a display module 1003.

[0183] The processing module 1001 is used to execute the processing functions of the model training fault analysis device in the above method embodiments. For example, it determines the fault node in the cross-stack graph during model training based on fault information and / or abnormal indicators; it determines the fault propagation path corresponding to the fault node based on the cross-stack graph, and determines the fault cause and / or abnormal cause during model training based on the fault propagation path. Another example is determining the neighbor nodes of the fault node in the cross-stack graph, where neighbor nodes are nodes connected to the fault node within a preset hop count range; it determines the fault subgraph based on the fault node, neighbor nodes, call relationships, and / or network connection relationships; and it determines the fault propagation path based on the fault subgraph. Yet another example is starting from the path starting point in the fault subgraph, searching the fault subgraph based on the call relationships and / or network connection relationships between nodes until the path ending point is found, where the path starting point is the highest-level fault node in the fault subgraph; and obtaining the fault propagation path based on the path starting point, the call relationships and / or network connection relationships between nodes involved in the search process, and the path ending point.

[0184] The information acquisition module 1002 is used to perform the information acquisition function of the model training fault analysis device in the above method embodiment. For example, it acquires fault information and / or abnormal indicators when the training cluster is training the model. For example, it acquires software stack information and hardware information.

[0185] Display module 1003 is used to perform the display function of the model training fault analysis device in the above method embodiment. For example, it displays the cross-stack diagram and the fault propagation path.

[0186] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0187] Since the model training fault analysis device 1000 provided in this embodiment can execute the above method, the technical effects it can achieve can be referred to the above method embodiment, and will not be repeated here.

[0188] In one possible design, the model training fault analysis device 1000 may further include a communication module. Figure 10 (Not shown in the image), this communication module can transmit the cross-stack graph and fault propagation path to other electronic devices to display the cross-stack graph and fault propagation path in other electronic devices.

[0189] In one possible design, the model training fault analysis device 1000 may further include a storage module. Figure 10 (Not shown in the image), this storage module stores programs or instructions. When the processing module 1001 executes the program or instructions, it causes the model training fault analysis device 1000 to perform the above-mentioned functions. Figure 2 and Figure 7 The method embodiment shown illustrates the function of the model training fault analysis device. It should be understood that the processing module 1001 involved in the model training fault analysis device 1000 can be implemented by a processor or processor-related circuit components, and can be a processor or processing unit; the information acquisition module 1002 can be implemented by information acquisition unit-related circuit components; and the display module 1003 can be implemented by display-related circuit components, and can be a display screen. It should be understood that the various solutions in the embodiments of this application can be reasonably combined and used, and the explanations or descriptions of the various terms appearing in the embodiments can be mutually referenced or explained in the various embodiments, without limitation.

[0190] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0191] The methods provided in the embodiments of this application above are described from the perspective of the model training fault analysis device as the execution subject. To implement the functions of the methods provided in the embodiments of this application above, the model training fault analysis device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution. The embodiments of this application can divide the model training fault analysis device into functional modules. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in the embodiments of this application is illustrative and only represents a logical functional division; other division methods may be used in actual implementation.

[0192] It should be understood that the various modules in the model training fault analysis device can be implemented in software and / or hardware, without specific limitations. In other words, the model training fault analysis device is presented in the form of functional modules. Here, "module" can refer to application-specific integrated circuits (ASICs), circuits, processors and memories that execute one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above functions.

[0193] In the above embodiments, depending on the context, the terms "when / at..." or "after..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrases "when determining..." or "if (the stated condition or event) is detected" can be interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)". Furthermore, in the above embodiments, relational terms such as "first" and "second" are used to distinguish one entity from another, without limiting any actual relationship or order between these entities.

[0194] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0195] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this embodiment is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)). Where there is no conflict, the solutions in the above embodiments can be combined.

[0196] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a model training fault analysis device. Of course, the processor and storage medium can also exist as discrete components in the model training fault analysis device.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A model training fault analysis method, characterized in that, The method includes: Obtain fault information and / or abnormal indicators during model training in the training cluster; The faulty node in the cross-stack graph during model training is determined based on the fault information and / or the abnormal indicators. The cross-stack graph is constructed based on software stack information and hardware information. The software stack information represents the level corresponding to the software node and the calling relationship between the software node during model training. The hardware information represents the level corresponding to the hardware node in the training cluster and the network connection relationship between the hardware node. The fault propagation path corresponding to the fault node is determined based on the cross-stack graph, and the fault propagation path is used to determine the fault cause and / or abnormal cause during model training.

2. The method according to claim 1, characterized in that, The step of determining the fault node in the cross-stack graph during model training based on the fault information and / or the abnormal indicators includes: Based on the fault information, a first annotation is made at the corresponding fault node in the cross-stack graph; and / or, A second annotation is made at the corresponding fault node in the cross-stack graph based on the aforementioned anomaly index.

3. The method according to claim 1 or 2, characterized in that, Determining the fault propagation path corresponding to the faulty node based on the cross-stack graph includes: Determine the neighboring nodes of the faulty node in the cross-stack graph, wherein the neighboring nodes are nodes connected to the faulty node within a preset hop count range; The fault subgraph is determined based on the faulty node, the neighboring nodes, the call relationship, and / or the network connection relationship; The fault propagation path is determined based on the fault subgraph.

4. The method according to claim 3, characterized in that, The faulty nodes include a first faulty node and a second faulty node. Determining the faulty subgraph based on the faulty nodes, neighboring nodes, the call relationships, and / or the network connection relationships includes: The first fault subgraph corresponding to the first fault node is determined based on the first fault node, the neighbor nodes corresponding to the first fault node, the call relationship and / or the network connection relationship. The second fault subgraph corresponding to the second fault node is determined based on the second fault node, the neighbor nodes corresponding to the second fault node, the call relationship and / or the network connection relationship; If the first fault subgraph and the second fault subgraph are connected, the fault subgraph is determined based on the first fault subgraph and the second fault subgraph. If the first fault subgraph and the second fault subgraph are not connected, the fault subgraph is determined based on the first fault subgraph, the second fault subgraph, and the auxiliary node. The auxiliary node is a node in the cross-stack graph that enables the first fault subgraph and the second fault subgraph to be connected.

5. The method according to claim 3 or 4, characterized in that, The fault subgraph includes multiple levels, and obtaining the fault propagation path based on the fault subgraph includes: Starting from the path start point in the fault subgraph, a search is performed in the fault subgraph based on the calling relationship and / or network connection relationship between nodes in the fault subgraph until the path end point is found. The path start point is the fault node with the highest level in the fault subgraph. The fault propagation path is obtained based on the starting point of the path, the calling relationships and / or network connection relationships between nodes involved in the search process, and the ending point of the path.

6. The method according to claim 5, characterized in that, The fault subgraph includes a software layer and a hardware layer. The path starts in the software layer and ends in the software layer, or the path starts in the software layer and ends in the hardware layer.

7. The method according to claim 5 or 6, characterized in that, The process of starting from the path origin in the fault subgraph and searching the fault subgraph based on the call relationships and / or network connection relationships between nodes until the path origin is found includes: Starting from the path origin in the fault subgraph, if the current search node is not labeled with the first and second labels, and the search node is not connected to the fault edge, the search node is taken as the path endpoint. The fault edge is a call relationship and / or network connection relationship with at least one end being the fault node.

8. The method according to any one of claims 1-7, characterized in that, The software node includes one or more of the neural network layer, acceleration component, framework function, logic operator, and computation operator of the model, and the hardware node includes one or more of the artificial intelligence AI acceleration card, switch, optical module, and network interface of the training cluster.

9. The method according to any one of claims 1-8, characterized in that, The abnormal indicators indicate that the model has a runtime failure during training, and the failure node is a node that has a runtime failure. Obtaining the abnormal indicators includes: The memory usage rate of the AI ​​accelerator card in the training cluster is obtained. If the difference between the current memory usage rate of the AI ​​accelerator card and the preset memory usage rate of the AI ​​accelerator card is greater than or equal to a first value, the memory usage rate of the AI ​​accelerator card is used as the abnormal indicator. And / or, Obtain the computing power utilization rate of the AI ​​accelerator card in the training cluster. If the difference between the computing power utilization rate of the AI ​​accelerator card and the preset computing power utilization rate of the AI ​​accelerator card is less than or equal to a second value, the computing power utilization rate of the AI ​​accelerator card is used as the abnormal indicator. And / or, The execution time of the neural network layer in the model is obtained. If the difference between the execution time of the neural network layer and the preset execution time of the neural network layer is greater than or equal to a third value, the execution time of the neural network layer is used as the abnormal indicator. And / or, Obtain the call duration of the framework function in the model. If the difference between the call duration of the framework function and the preset call duration of the framework function is greater than or equal to a fourth value, the call duration of the framework function is used as the abnormal indicator. And / or, Obtain the computation time of the computation operator in the model. If the difference between the computation time of the computation operator and the preset computation time of the computation operator is greater than or equal to the fifth value, the computation time of the computation operator is used as the abnormal indicator.

10. The method according to any one of claims 1-8, characterized in that, The abnormal indicators indicate that the model has a performance failure during training, and the failure node is a node with a performance failure. Obtaining the abnormal indicators includes: The execution time of a node in the model is obtained. If the difference between the execution time of the node and the preset execution time corresponding to the node is greater than or equal to the sixth value, the execution time of the node is used as the abnormal indicator.

11. The method according to any one of claims 1-10, characterized in that, The acquisition of fault information includes: Obtain error information from the training logs of the model; The fault information is determined based on the error information, and the fault information includes the fault node identifier.

12. A model training fault analysis device, characterized in that, The model training fault analysis device includes: Memory is used to store computer program instructions; A processor is configured to execute the computer program instructions to support the model training fault analysis apparatus in implementing the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processing circuit, implement the method as described in any one of claims 1-11.

14. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-11.

Images