Fault Node Localization Method, Device and Medium for Distributed Model Training

Abstract

This invention relates to the field of artificial intelligence technology. It provides a method, apparatus, device, and medium for fault node localization in distributed model training. The method includes: acquiring loss value change information and log output information during the distributed model training process; determining, based on the loss value change information and log output information, when an abnormal GPU node is identified, determining a set of potentially affected neighboring GPU nodes based on the abnormal GPU node and a heterogeneous communication topology graph; the heterogeneous communication topology graph is a communication topology graph reflecting the actual physical connection relationships, with each GPU as a node; the edges between nodes in the heterogeneous communication topology graph are configured with propagation fault weights. This invention achieves an improvement from only processing error-reporting nodes to "identifying the source based on the propagation path," realizing a leap in fault localization granularity from the whole-machine level to the GPU level, significantly improving resource utilization.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and medium for fault node localization in distributed model training. Background Technology

[0002] In large-scale distributed deep learning training systems, a job scheduling framework based on Slurm (Simple Linux Utility for Resource Management) is used to manage computing clusters consisting of hundreds of GPUs (Graphics Processing Units, the computing power carriers). Training tasks run in data-parallel or model-parallel modes, and the GPU nodes communicate frequently with each other via NVLink (within a single node) and InfiniBand (across nodes). In such systems: 1. Log output mechanism: Standard output and error messages during training are redirected to a log file in shared storage (e.g., `log-`). .out), used for subsequent investigation; 2. Common exception types: including runtime errors thrown by Python or the NCCL library such as `RuntimeError`, `AssertionError`, and `ncclRemoteError`; 3. Loss monitoring: The loss function value serves as the core indicator for measuring whether training is normal, and is extracted from the logs through text parsing; 4. Hardware Topology: AI servers are generally configured with multiple GPU nodes interconnected via NVSwitch. Cross-node communication relies on the InfiniBand network to complete aggregated communication operations such as AllReduce. NVSwitch is a dedicated switching chip launched by NVIDIA. Its core function is to connect multiple GPUs within the same physical node (compute server). It builds a fully interconnected, non-blocking topology through NVLink, enabling full-speed direct communication between any two GPUs. It also supports cross-node expansion and is a key hardware component for ultra-large-scale deep learning training clusters. It can solve the bandwidth bottleneck of PCIe Switch and maximize the communication efficiency of GPUs within a node.

[0003] 5. Node identification mechanism: Each computing host has a unique hostname (e.g., SHLG-ZS-07), and each GPU can be distinguished by its device number.

[0004] Currently, in most AI training clusters, anomaly detection mainly relies on simple rule-based judgments implemented manually or via scripts. Anomaly monitoring uses keyword retrieval, matching preset error patterns such as "AssertionError," "RuntimeError," or "ncclRemoteError" in logs using tools like grep, triggering alarms upon detection. For loss monitoring, fixed thresholds are typically set for judgment, or engineers visually observe trends on a visualization platform. When anomalies are identified, fault localization depends on whether the hostname is explicitly output in the logs, lacking the ability to understand contextual semantics and failing to generalize the identification of unseen error types. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method, apparatus, device, and medium for fault node localization in distributed model training.

[0006] This invention provides a method for fault node localization in distributed model training, wherein the distributed model is trained by a computing cluster composed of GPU nodes, and the method includes: Obtain information on loss value changes and log output during the training process of a distributed model; Based on the loss value change information and the log output information, when it is determined that there is an abnormal GPU node, the set of potentially affected neighboring GPU nodes is determined based on the abnormal GPU node and the heterogeneous communication topology. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0007] According to the present invention, a method for locating faulty nodes in distributed model training, wherein determining the existence of abnormal GPU nodes based on the loss value change information and the log output information includes: Anomaly scores are determined based on the loss value change information; When an anomaly is determined to exist during the training of the distributed model based on the anomaly score, a context fragment based on the key information of the anomaly is generated according to the log output information. Based on the context fragment, an anomaly diagnosis result is generated, which includes the identifier of the abnormal GPU node; The abnormal GPU node is identified based on the identifier.

[0008] According to the present invention, a method for locating fault nodes in distributed model training includes determining anomaly scores based on the loss value change information, comprising: Based on the loss value change information, determine the effective loss value sequence; Based on the effective loss value sequence, determine the mean and standard value; The relative offset and the rate of change of trend are determined based on the mean and the standard value. Anomaly scores are determined based on the relative offset and the rate of change of the trend.

[0009] According to the present invention, a method for locating fault nodes in distributed model training includes generating a context fragment based on key anomaly information according to the log output information, comprising: Based on the log output information, determine the key information of the anomaly; Select consecutive text lines that precede and follow the aforementioned critical abnormal information, up to a predetermined number of lines. Based on the continuous text, a context fragment is generated; the context fragment includes key exception information, stack trace, process ID, timestamp, and context of previous and subsequent operations.

[0010] According to the present invention, a fault node localization method for distributed model training is provided, wherein the anomaly diagnosis result includes: anomaly type, root cause description, identification of the abnormal GPU node, severity level, and action strategy.

[0011] According to the present invention, a method for locating faulty nodes in distributed model training, wherein determining a set of potentially affected neighboring GPU nodes based on the abnormal GPU node and a heterogeneous communication topology includes: Using the abnormal GPU node as the starting point for propagation simulation, multiple rounds of iterative simulation are performed on the heterogeneous communication topology graph. Based on the propagation fault weights configured on the edges between each node in the heterogeneous communication topology graph, the potentially affected neighboring GPU nodes are determined.

[0012] According to the present invention, a method for locating fault nodes in distributed model training further includes: Configure risk scores for each GUP node in the heterogeneous communication topology diagram; Based on the risk scores, a risk level distribution map is determined; Based on the risk level distribution map, a tiered response strategy is determined.

[0013] According to the present invention, a method for locating fault nodes in distributed model training is provided, the method further comprising constructing a heterogeneous communication topology graph, including: With each GPU node as a node, a communication topology graph G=(V,E) reflecting the actual physical connection relationship is established, where the node set V represents all GPU nodes participating in training, and the edge set E represents the high-speed communication link between GPU nodes. Based on the physical distance between each GPU node, the historical error frequency of the link, and the communication bandwidth, the propagation fault weights of the edges configured between each node are determined.

[0014] According to the present invention, a fault node localization method for distributed model training is provided, and the edge types are divided into edges connecting GPU node pairs within the same computing server node, edges connecting GPU node pairs between different computing server nodes, and edges connecting the computing server node to which each GPU belongs and the master server node in the computing cluster.

[0015] According to the present invention, a method for locating fault nodes in distributed model training includes determining the propagation fault weights of the edges configured between nodes based on the physical distance between GPU nodes, the historical error frequency of the links, and the communication bandwidth. Based on the physical distance between each GPU node, the historical error frequency of the link, and the communication bandwidth, the propagation fault weights of the edges configured between each node are determined using the following calculation formula; ; in, To propagate fault weights, The normalization coefficient is... For nodes and The physical distance between them It is the attenuation constant; The historical error frequency of the link within a preset time period; ; For nodes and The edge between; Communication bandwidth includes the number of error reports on the link and the number of communication rounds.

[0016] According to the present invention, a method for locating fault nodes in distributed model training further includes: Based on the context fragment, it is encoded into a semantic vector; The semantic vectors and the anomaly diagnosis results are stored in a vector database to construct a fault knowledge base.

[0017] According to the present invention, a method for locating fault nodes in distributed model training further includes: Obtain the semantic vector corresponding to the new context fragment; Determine the semantic vector corresponding to the new context fragment, and the semantic vector with the best similarity in the fault knowledge base; The fault diagnosis result corresponding to the semantic vector with the best similarity is matched to the new context fragment.

[0018] The present invention also provides a fault node localization device for distributed model training, wherein the distributed model is trained by a computing cluster composed of GPU nodes, and the device includes: The acquisition module is used to acquire information on loss value changes and log output during the training process of the distributed model; The positioning module is used to determine the set of potentially affected neighboring GPU nodes based on the abnormal GPU node and the heterogeneous communication topology map when the abnormal GPU node is determined based on the loss value change information and the log output information. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described methods for fault node localization in distributed model training.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for fault node localization in distributed model training.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for fault node localization in distributed model training.

[0022] This invention provides a method, apparatus, device, and medium for fault node localization in distributed model training. It acquires loss value change information and log output information during the distributed model training process. Based on these information, when an abnormal GPU node is identified, a set of potentially affected neighboring GPU nodes is determined using the abnormal GPU node and a communication topology map reflecting the actual physical connections, with each GPU as a node. This approach breaks through the traditional method of isolating at the whole-machine level, achieving an improvement from only handling error-reporting nodes to "identifying the source based on the propagation path." It realizes a leap in fault localization granularity from the whole-machine level to the GPU level, significantly improving resource utilization. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the fault node localization method for distributed model training provided by the present invention.

[0025] Figure 2 This is a schematic diagram of the fault node localization device for distributed model training provided by the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Figure 1 This diagram illustrates a flowchart of a fault node localization method for distributed model training provided by the present invention. (See attached diagram.) Figure 1 The distributed model is trained by a computing cluster consisting of GPU nodes. This method includes the following steps: Step 11: Obtain information on the changes in loss values ​​and log output during the training process of the distributed model.

[0029] Step 12: Based on the loss value change information and log output information, when it is determined that there is an abnormal GPU node, determine the set of potentially affected neighboring GPU nodes based on the abnormal GPU node and the heterogeneous communication topology graph; wherein, the heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0030] Regarding steps 11 and 12, it should be noted that in ultra-large-scale distributed deep learning training systems, a job scheduling framework based on Slurm (Simple Linux Utility for Resource Management) is used to manage a computing cluster consisting of hundreds of GPU (Graphics Processing Unit) nodes. Training tasks run in data-parallel or model-parallel manner, and the GPU nodes communicate frequently with each other via NVLink (within a single node) and InfiniBand (across nodes). In such systems: 1. Log output mechanism: Standard output and error messages during training are redirected to a log file in shared storage (e.g., `log-`). .out), used for subsequent investigation; 2. Common exception types: including runtime errors thrown by Python or the NCCL library such as `RuntimeError`, `AssertionError`, and `ncclRemoteError`; 3. Loss monitoring: The loss function value serves as the core indicator for measuring whether training is normal, and is extracted from the logs through text parsing; 4. Hardware Topology: AI servers are generally configured with multiple GPU nodes interconnected via NVSwitch. Cross-node communication relies on the InfiniBand network to complete aggregated communication operations such as AllReduce. NVSwitch is a dedicated switching chip launched by NVIDIA. Its core function is to connect multiple GPUs within the same physical node (compute server). It builds a fully interconnected, non-blocking topology through NVLink, enabling full-speed direct communication between any two GPUs. It also supports cross-node expansion and is a key hardware component for ultra-large-scale deep learning training clusters. It can solve the bandwidth bottleneck of PCIe Switch and maximize the communication efficiency of GPUs within a node.

[0031] 5. Node identification mechanism: Each computing host has a unique hostname (e.g., SHLG-ZS-07), and each GPU can be distinguished by its device number.

[0032] Therefore, the present invention can first continuously monitor the trend of loss value changes and log output content during the model training process.

[0033] During the training of large models, the changing trend of the loss function is a core indicator for measuring whether the training is normal. However, simple threshold judgments (such as "loss>10") are prone to false positives in the early stages of training and cannot adapt to the fluctuating characteristics at different stages. To address this, this invention designs a dynamic scoring mechanism based on a combination of statistical analysis and trend changes.

[0034] Within each monitoring period, the effective loss values ​​from the most recent few steps are extracted from the latest log output to form a time series, which is defined as the loss value series. If we take the nearest N non-NaN loss values ​​(e.g., N=50), we can calculate the mean of this loss value sequence. with standard deviation This serves as the baseline statistic for the current training state.

[0035] To assess the deviation of the latest loss value from the baseline, we introduce the relative deviation and the rate of change of trend: in It represents the loss value of the latest step (i.e., the loss value of the most recent effective training step). It represents the sequence of the N most recent non-NaN loss values. of the median. The adjustment factor is a very small positive number (e.g., 10⁻⁸) to prevent the denominator from being zero (or close to zero) and causing numerical instability.

[0036] This is addressed by the deviation score. A higher score indicates that the current loss value deviates more from the historical average. The trend change rate score reflects the rate of increase of the current loss value relative to the typical value during training. Since losses are typically high and volatile in the early stages of training, and tend to level off after entering a stable phase, a potential collapse signal should only be considered when the current value is not only absolutely high but also exhibits an abnormal upward trend.

[0037] The deviation score and the trend change rate score are weighted and combined to obtain the final anomaly score: when τ (adaptive threshold, initially set to 3, adjustable during training) triggers a "potential training collapse" signal. In other words, it determines anomalies in the distributed model training process based on abnormal scores; specifically, it initiates a fault diagnosis process when significant degradation in training performance or serious errors are detected.

[0038] The advantage of this method is that it does not rely on a fixed threshold, but adaptively adjusts the sensitivity according to the dynamic characteristics of the training process itself, thereby effectively avoiding misjudgments in the early stages of training or during normal fluctuations and improving the robustness of anomaly detection.

[0039] At this point, after initiating the fault diagnosis process, the abnormal GPU node is identified. Then, based on the abnormal GPU node and the heterogeneous communication topology graph, a set of potentially affected neighboring GPU nodes is determined. The heterogeneous communication topology graph is a communication topology graph reflecting the actual physical connections, built with each GPU as a node. In other words, semantic understanding is used to analyze the root cause of the error, and the potential range of fault propagation is predicted by combining the communication topology structure between GPUs. The edges between nodes in this heterogeneous communication topology graph are configured with fault propagation weights. This means that by propagating the fault weights, the abnormal GPU node can be expanded outwards to determine potentially affected nodes, thus constructing a node set.

[0040] In this invention, further explanation is provided: based on context fragments, an anomaly diagnosis result is generated, which includes an identifier of the abnormal GPU node; the abnormal GPU node is then identified based on the identifier. During the generation of the context fragments, key anomaly information is determined based on log output information; a preset number of consecutive lines of text before and after the key anomaly information are selected. Generate context fragments from continuous text; context fragments include critical exception information, stack trace, process ID, timestamp, and context of previous and subsequent operations.

[0041] Specifically, during the training task, the output log file (such as log-) is read periodically. The system uses a .out file to identify text segments containing error information. When a preset key error pattern (such as "RuntimeError", "AssertionError", "ncclRemoteError", etc.) is detected, the system extracts a certain number of consecutive lines of text before and after the keyword to form a complete context segment. This segment includes not only the error message itself, but also the stack trace, process ID, timestamp, and the preceding and following operational context, ensuring semantic integrity.

[0042] Let the context fragment extracted in the i-th detection be... It takes the form of a text sequence that mixes natural language and code: Where L is the number of lines of text, such as 10–30 lines, to cover the complete error occurrence scenario.

[0043] The purpose of this step is to provide sufficient context for subsequent semantic analysis and avoid misjudgments due to missing information.

[0044] In this invention, the extracted context fragments The input is fed into a lightweight large language model M with approximately 1 billion parameters (such as the open-source model qwen-1B, or jiutian-1B for China Mobile). This model has been fine-tuned on a large number of labeled AI training log datasets and has the ability to understand the underlying errors of deep learning frameworks (such as PyTorch and Megatron-LM).

[0045] After receiving the input, the model generates a structured diagnostic output. It takes the form of a JSON object containing multiple semantic fields: The meanings of each field are as follows: Error type, such as NCCL_TIMEOUT, IB_LINK_FAILURE; Root cause description, such as "InfiniBand subnet manager timeout due to HCA reset"; : Affected component, in the format "NodeName:GPUx", represents the specific faulty hardware, i.e., the identifier of the abnormal GPU node; Severity level, with values ​​of FATAL, ERROR, or WARNING; Suggested actions, such as "Run IB test and drain node", are suggested action strategies.

[0046] This application introduces a lightweight large language model fine-tuned on training logs. By understanding the semantic context, this model can infer the root cause of unseen errors, completing classification and localization without pre-configured rules, thus improving anomaly detection coverage to over 95%.

[0047] It should also be noted that, in order to enable the system to have "memory" capabilities, this invention transforms the log context upon which each diagnosis is based into a measurable semantic vector. Specifically, a pre-trained sentence embedding model (such as a Sentence-BERT-like model) is used to embed the context fragments... Encode as a d-dimensional real vector ; Where d is usually taken as 768 or 1024, representing the dimension of the semantic space.

[0048] This vector captures the deep semantic features of the log text, making semantically similar errors (such as the same type of IB timeout occurring at different times) closer together in the vector space.

[0049] The system then uses this vector along with the corresponding diagnostic results. Metadata such as timestamps and processing status are stored together in a vector database, forming a continuously growing fault knowledge base K. .

[0050] When a new exception context is detected At that time, the system first calculates its corresponding semantic vector. Then, a nearest neighbor search is performed in the knowledge base K to find the most similar entry in history.

[0051] Similarity is measured using cosine similarity: If the maximum similarity exceeds a preset threshold (e.g., 0.9), the current error is considered highly similar to a historical fault, and the system directly reuses its diagnostic results. With this processing strategy, there is no need to call the large model again for inference.

[0052] This mechanism significantly reduces the computational overhead of repeated diagnoses and enables the automatic transfer of processing experience, giving the system a self-evolving characteristic of "getting smarter with use".

[0053] The system regularly maintains the knowledge base. Fault entries that have been verified as resolved are marked as "resolved"; cases that are triggered multiple times and successfully handled are given increased priority. Additionally, a manual annotation interface is supported, allowing operations personnel to correct model misjudgments and use the corrected data for subsequent incremental model training.

[0054] In large-scale distributed model training systems, computational tasks typically involve hundreds or even thousands of GPUs working collaboratively, with frequent communication between these devices via high-speed interconnect networks. When a GPU or communication link fails, its impact can spread to neighboring nodes through the physical connection structure, triggering cascading communication timeouts or training interruptions. To prevent such fault propagation, this invention proposes a fault propagation prediction algorithm based on a multi-level communication topology graph. By constructing a fine-grained hardware connection relationship model and combining it with historical link stability data, the algorithm predicts the potential propagation path of faults in the system, thereby enabling proactive isolation measures and improving the overall stability of the cluster.

[0055] The following mainly explains the process of identifying the set of potentially affected neighboring GPU nodes based on abnormal GPU nodes and heterogeneous communication topology, including: Using the abnormal GPU node as the starting point for propagation simulation, multiple rounds of iterative simulation are performed on the heterogeneous communication topology graph. Based on the propagation fault weights configured on the edges between each node in the heterogeneous communication topology graph, the potentially affected neighboring GPU nodes are identified.

[0056] In this regard, it should be noted that the construction process of the heterogeneous communication topology graph is explained first. This invention uses each GPU as a node to establish a communication topology graph G=(V,E) that reflects the actual physical connection relationship, where the node set V represents all GPU instances participating in training, and the edge set E represents the high-speed communication links between GPUs. Its type is divided into the following three categories according to the physical medium and connection layer: NVLink edges (Intra-node Edges) connect pairs of GPUs within the same compute node; InfiniBand (Inter-node Edge) connects GPU pairs between different compute nodes; The control-plane edge connects each GPU's node to the Slurm master node; The resulting topology graph G is a heterogeneous graph whose structure truly reflects the actual data exchange paths during the training process.

[0057] Not all links have the same risk of failure propagation. To quantify the vulnerability of different connections, this invention quantifies the vulnerability of each edge. Define a propagation weight , representing the probability strength of a fault propagating from node i to node j.

[0058] This weight is determined by three factors: physical distance, link history error frequency, and communication bandwidth. Specifically, the propagation probability function is defined as follows: ; in: The physical distance between nodes i and j is set to 1 within the same rack and 2 across racks. λ is the attenuation constant, which controls the rate at which distance affects the propagation probability, and is usually taken as 1.5; The error frequency of this link over a past period (e.g., 24 hours) is calculated as follows: α is the normalization coefficient used to adjust the overall propagation intensity, and its value ranges from 0.6 to 0.9.

[0059] When the system identifies a GPU as the source of the fault through log analysis or loss anomaly detection, it marks it as "infected" and uses it as the starting point for the propagation simulation.

[0060] Let the fault source be node The system assigns an initial infection strength to it. All other nodes are initialized to a "susceptible" state with an infection strength of 0. Simultaneously, the system records the communication mode of the current training task (e.g., whether it is in the all-reduce phase) to determine which links are under high load and further adjust their propagation weights.

[0061] This invention employs a SIR-like (Susceptible-Infected-Recovered) propagation model and performs multiple rounds of iterative simulation on the topology graph to predict the spread range of the fault signal.

[0062] Each round of propagation proceeds as follows: For each node u that is currently in the "infected" state, traverse all its direct neighbors. If v is currently in a "susceptible" state, then according to the edge Probability of propagation Determine if you are infected: ; Where ξ is a random number used to simulate the uncertainty of propagation.

[0063] This process is repeated several rounds (usually 2–3 rounds), each round representing a potential spread within a communication cycle. As the number of rounds increases, the infection range gradually expands until no new nodes are infected.

[0064] Finally, the system outputs a set of potentially affected nodes R={v∈V|v is simulated to be infected}, which is the predicted high-risk area.

[0065] This application constructs a propagation simulation mechanism based on physical distance and the frequency of historical link errors. After confirming the initial fault node, the system can run multiple rounds of propagation simulation on the topology graph to identify potentially high-risk GPUs and guide a hierarchical isolation strategy, thereby reducing the cascading failure rate by 80%.

[0066] Based on the propagation simulation results, the system assigns a risk score to each node: ; in It can be defined as the product of the maximum path propagation probabilities from the source node to v.

[0067] The system generates a risk level distribution map based on this score and formulates a tiered response strategy accordingly. Risk score ≥ 0.8: Mark the node as "under maintenance" and have maintenance personnel troubleshoot the fault; Risk score ≥ 0.5: Mark as "Under Observation" and suspend the assignment of new tasks; Risk score <0.5: Maintain normal status.

[0068] This mechanism represents an upgrade from "point-based faults" to "area-based prevention and control," effectively curbing the horizontal spread of faults within the cluster.

[0069] In this invention, it should also be noted that, after completing anomaly detection, fault diagnosis, and impact range prediction, a closed-loop adaptive training recovery controller is designed to achieve complete closed-loop management from perception to execution. This controller can dynamically decide and execute a series of recovery actions based on fault severity, historical recovery results, and system status, including node isolation, task cleanup, and resubmission. It also possesses state memory and strategy adjustment capabilities, making the recovery behavior intelligent and adaptive, avoiding resource waste caused by blind retries.

[0070] S3.1: Initiate the fault handling process: When the system confirms a blocking fault using the Loss anomaly scoring model or a lightweight large model, the controller is triggered to enter the working state. At this time, the system records the current timestamp, fault type, involved nodes, and diagnostic basis as context information for this recovery operation, and then proceeds to the next stage.

[0071] S3.2: Perform diagnostic verification and information supplementation: To improve decision-making accuracy, the controller first calls underlying diagnostic tools to verify suspected faulty nodes. For example, if the error is related to InfiniBand communication, the IB network test script is automatically executed to check link connectivity, bandwidth stability, and peer status; if it involves memory anomalies, the memory usage trend and ECC error count of the GPU on that node are checked.

[0072] S3.4: Clean up failed tasks and prepare to restart the environment: After node isolation is complete, the controller queries all running jobs under the current user that are related to this training. For these jobs, it performs a cancellation operation one by one to terminate their continued execution, preventing old tasks from consuming resources or conflicting with new tasks.

[0073] S3.5: Submit a new training task: After the environment cleanup is complete, the controller calls the task submission interface to resubmit the job using the original training script. The new task inherits the original resource configuration, but can automatically adjust the startup parameters based on historical failure scenarios, such as enabling a more frequent checkpoint saving mechanism or increasing the number of fault-tolerant retries.

[0074] The fault node localization method for distributed model training provided by this invention obtains information on loss value changes and log output during the distributed model training process. Based on the loss value change information and log output information, when an abnormal GPU node is determined, a set of potentially affected neighboring GPU nodes is identified based on the abnormal GPU node and a communication topology map reflecting the actual physical connection relationship, with each GPU as a node. This method breaks through the traditional approach of isolating at the whole machine level, and achieves an improvement from only handling error-reporting nodes to "identifying the source based on the propagation path". It realizes a leap in fault localization granularity from the whole machine level to the GPU level, significantly improving resource utilization.

[0075] The fault node localization device for distributed model training provided by the present invention will be described below. The fault node localization device for distributed model training described below can be referred to in correspondence with the fault node localization method for distributed model training described above.

[0076] Figure 2 This diagram illustrates the structure of a fault node localization device for distributed model training provided by the present invention. (See attached diagram.) Figure 2 The distributed model is trained by a computing cluster consisting of GPU nodes. The device includes an acquisition module 21 and a localization module 22, wherein: The acquisition module is used to acquire information on loss value changes and log output during the training process of the distributed model; The localization module is used to determine the set of potentially affected neighboring GPU nodes based on the abnormal GPU node and the heterogeneous communication topology map when an abnormal GPU node is identified, according to the loss value change information and log output information. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0077] Since the apparatus of this embodiment is based on the same principle as the method of the above embodiment, more detailed explanations will not be repeated here.

[0078] It should be noted that, in the embodiments of the present invention, the relevant functional modules can be implemented by a hardware processor.

[0079] The fault node localization device for distributed model training provided by this invention acquires information on loss value changes and log output during the distributed model training process. Based on the loss value change information and log output information, when an abnormal GPU node is determined, a set of potentially affected neighboring GPU nodes is identified based on the abnormal GPU node and a communication topology map reflecting the actual physical connection relationship, with each GPU as a node. This breaks through the traditional approach of isolating at the whole machine level, and achieves an improvement from only processing error-reporting nodes to "identifying the source based on the propagation path". It realizes a leap from the whole machine level to the GPU level in fault localization granularity, significantly improving resource utilization.

[0080] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 31, a communication interface 32, a memory 33, and a communication bus 34, wherein the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34. The processor 31 can call logical instructions in the memory 33 to execute a fault node localization method for distributed model training, the method including: Obtain information on loss value changes and log output during the training process of a distributed model; When an abnormal GPU node is identified based on the loss value change information and log output information, the set of potentially affected neighboring GPU nodes is determined based on the abnormal GPU node and the heterogeneous communication topology. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0081] Furthermore, the logical instructions in the aforementioned memory 33 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of providing a fault node localization method for distributed model training, the method comprising: Obtain information on loss value changes and log output during the training process of a distributed model; When an abnormal GPU node is identified based on the loss value change information and log output information, the set of potentially affected neighboring GPU nodes is determined based on the abnormal GPU node and the heterogeneous communication topology. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a fault node localization method for distributed model training, the method comprising: Obtain information on loss value changes and log output during the training process of a distributed model; When an abnormal GPU node is identified based on the loss value change information and log output information, the set of potentially affected neighboring GPU nodes is determined based on the abnormal GPU node and the heterogeneous communication topology. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for locating fault nodes in distributed model training, characterized in that, The distributed model is trained by a computing cluster consisting of GPU nodes, and the method includes: Obtain information on loss value changes and log output during the training process of a distributed model; Based on the loss value change information and the log output information, when it is determined that there is an abnormal GPU node, the set of potentially affected neighboring GPU nodes is determined based on the abnormal GPU node and the heterogeneous communication topology. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

2. The fault node localization method for distributed model training according to claim 1, characterized in that, The step of determining the existence of an abnormal GPU node based on the loss value change information and the log output information includes: Anomaly scores are determined based on the loss value change information; When an anomaly is determined to exist during the training of the distributed model based on the anomaly score, a context fragment based on the key information of the anomaly is generated according to the log output information. Based on the context fragment, an anomaly diagnosis result is generated, which includes the identifier of the abnormal GPU node; The abnormal GPU node is identified based on the identifier.

3. The fault node localization method for distributed model training according to claim 2, characterized in that, The step of determining the anomaly score based on the loss value change information includes: Based on the loss value change information, determine the effective loss value sequence; Based on the effective loss value sequence, determine the mean and standard value; The relative offset and the rate of change of trend are determined based on the mean and the standard value. Anomaly scores are determined based on the relative offset and the rate of change of the trend.

4. The fault node localization method for distributed model training according to claim 2 or 3, characterized in that, The step of generating a context fragment based on key anomaly information according to the log output information includes: Based on the log output information, determine the key information of the anomaly; Select consecutive text lines that precede and follow the aforementioned critical abnormal information, up to a predetermined number of lines. Based on the continuous text, a context fragment is generated; the context fragment includes key exception information, stack trace, process ID, timestamp, and context of previous and subsequent operations.

5. The fault node localization method for distributed model training according to claim 4, characterized in that, The anomaly diagnosis results include: anomaly type, root cause description, identifier of the abnormal GPU node, severity level, and action strategy.

6. The fault node localization method for distributed model training according to claim 1, characterized in that, The step of determining the set of potentially affected neighboring GPU nodes based on the abnormal GPU nodes and the heterogeneous communication topology includes: Using the abnormal GPU node as the starting point for propagation simulation, multiple rounds of iterative simulation are performed on the heterogeneous communication topology graph. Based on the propagation fault weights configured on the edges between each node in the heterogeneous communication topology graph, the potentially affected neighboring GPU nodes are determined.

7. The fault node localization method for distributed model training according to claim 6, characterized in that, The method further includes: Configure risk scores for each GUP node in the heterogeneous communication topology diagram; Based on the risk scores, a risk level distribution map is determined; Based on the risk level distribution map, a tiered response strategy is determined.

8. The fault node localization method for distributed model training according to claim 1, characterized in that, The method further includes constructing a heterogeneous communication topology graph, including: Using each GPU node as a node, a communication topology graph G=(V,E) reflecting the actual physical connection relationship is established, where the node set V represents all GPU nodes participating in training, and the edge set E represents the high-speed communication link between GPU nodes; the edge types are divided into edges connecting GPU node pairs within the same compute server node, edges connecting GPU node pairs between different compute server nodes, and edges connecting the compute server node to which each GPU belongs to the master server node in the compute cluster. Based on the physical distance between each GPU node, the historical error frequency of the link, and the communication bandwidth, the propagation fault weights of the edges configured between each node are determined.

9. The fault node localization method for distributed model training according to claim 8, characterized in that, Edge types are categorized based on physical medium and connection level into edges connecting GPU node pairs within the same compute server node, edges connecting GPU node pairs between different compute server nodes, and edges connecting the compute server node to which each GPU belongs to the master server node in the compute cluster.

10. The fault node localization method for distributed model training according to claim 8 or 9, characterized in that, The process of determining the propagation fault weights of the edges configured between each node based on the physical distance between each GPU node, the historical error frequency of the links, and the communication bandwidth includes: Based on the physical distance between each GPU node, the historical error frequency of the link, and the communication bandwidth, the propagation fault weights of the edges configured between each node are determined using the following calculation formula; ； in, To propagate fault weights, The normalization coefficient is... For nodes and The physical distance between them It is the attenuation constant; The historical error frequency of the link within a preset time period; ; For nodes and The edge between; Communication bandwidth includes the number of error reports on the link and the number of communication rounds.

11. The fault node localization method for distributed model training according to claim 2, characterized in that, The method further includes: Based on the context fragment, it is encoded into a semantic vector; The semantic vectors and the anomaly diagnosis results are stored in a vector database to construct a fault knowledge base.

12. The fault node localization method for distributed model training according to claim 11, characterized in that, The method further includes: Obtain the semantic vector corresponding to the new context fragment; Determine the semantic vector corresponding to the new context fragment, and the semantic vector with the best similarity in the fault knowledge base; The fault diagnosis result corresponding to the semantic vector with the best similarity is matched to the new context fragment.

13. A fault node localization device for distributed model training, characterized in that, The distributed model is trained by a computing cluster consisting of GPU nodes, and the device includes: The acquisition module is used to acquire information on loss value changes and log output during the training process of the distributed model; The positioning module is used to determine the set of potentially affected neighboring GPU nodes based on the abnormal GPU node and the heterogeneous communication topology map when the abnormal GPU node is determined based on the loss value change information and the log output information. The heterogeneous communication topology graph is a communication topology graph that reflects the actual physical connection relationship, with each GPU as a node; the edges between each node in the heterogeneous communication topology graph are configured with propagation fault weights.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the fault node localization method for distributed model training as described in any one of claims 1-12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fault node localization method for distributed model training as described in any one of claims 1-12.

Images