A hang monitoring analysis method and system for distributed model training

Abstract

The application discloses a HANG monitoring analysis method and system for distributed model training, and belongs to the technical field of distributed training monitoring. The method periodically collects the call stacks of a batch of nodes in the training process, dynamically replaces the nodes, merges the stacks of multiple ranks in the same node into a set, and judges the change degree between adjacent samples through Jaccard similarity. When the similarity is higher than the threshold value for consecutive times and does not hit the whitelist mode, it is determined that the training task enters the HANG state. The stacks of all nodes and all ranks are automatically collected, a StackTrie is constructed, and the abnormal ranks deviating from the majority execution path are identified according to the branch coverage rate. The alarm result and the analysis result are output. The application can identify training stagnation without relying on single abnormal indicators, reduce false positives, support drilling from the cluster level alarm to the rank level positioning, and is suitable for distributed training monitoring platforms.

Description

Technical Field

[0001] This invention relates to the field of distributed training monitoring technology, and in particular to a training HANG monitoring, detection and analysis method based on periodic stack sampling, similarity determination, whitelist filtering and global fork analysis. Specifically, it relates to a HANG monitoring and analysis method and system for distributed model training. Background Technology

[0002] In large-scale distributed model training, training tasks are typically executed collaboratively by multiple computing nodes. Nodes maintain training consistency through mechanisms such as parameter synchronization, ensemble communication, gradient aggregation, broadcasting, and barriers. As model size, data size, and parallelism continue to increase, the training system's dependence on communication, storage, scheduling, and data pipelines significantly increases, making it more susceptible to problems such as partial blocking, synchronization waiting, data stalling, and thread stagnation.

[0003] The term "HANG" typically doesn't refer to a traditional process exit or crash. Instead, it indicates that while the training task appears to be running, the steps stop advancing, the loss stops changing, the call path remains stable over a long period, GPU resources are continuously consumed, and model parameters are no longer effectively updated. This type of problem is highly insidious: from the operating system's perspective, the process still exists; from the service's perspective, the training task seems "alive"; but from a business perspective, training has stopped.

[0004] Existing monitoring methods mostly rely on GPU utilization, Step Time, task heartbeats, log keyword alerts, and job status polling. While these methods are effective for explicit errors or complete resource outages, their ability to identify implicit hangs (processes not dead but training stopped) is limited. This is because: firstly, a decrease in GPU utilization does not necessarily mean a complete freeze; it may simply be a periodic synchronization or data reading; secondly, error messages may not appear in the logs, and heartbeats may continue to be reported normally. Therefore, a single indicator is insufficient to accurately distinguish between normal long processing times and true freezes.

[0005] Furthermore, HANGs in distributed training often do not occur simultaneously across the entire system, but may first appear on a few nodes or individual ranks. For example, some ranks may be stuck on the data loading path, some may be blocked on communication calls, and some may be stuck in specific synchronization functions. Relying solely on the state of local nodes for judgment is not only prone to false positives, but also makes it difficult to answer key questions such as "which rank became abnormal first," "which execution branch deviated from the majority path," and "has the problem spread globally?"

[0006] Existing solutions also face a common challenge: there are many legitimate long-term blocking scenarios during training, such as checkpoint saving, model write to disk, data reloading, preprocessing waits, and startup initialization. These scenarios often manifest as a persistently unchanged stack at the call stack level. Without whitelist filtering, normal long-term executions can easily be misjudged as HANG (Hang). Furthermore, a single sample only reflects a fleeting slice, failing to demonstrate continuous stability or determine whether a stack interruption is merely temporary. Therefore, a comprehensive solution is lacking in large-scale model training scenarios that balances continuity determination, whitelist filtering, global drill-down, and anomaly rank localization.

[0007] Based on the above problems, there is an urgent need for a HANG monitoring, detection and analysis method that can unify periodic sampling, stack structure comparison, continuous stability judgment, global call path analysis and Rank positioning, so that training monitoring can be upgraded from "discovering anomalies" to "identifying anomalies, locating anomalies, and interpreting anomalies". Summary of the Invention

[0008] The purpose of this invention is to provide a HANG monitoring and analysis method and system for distributed model training.

[0009] The objective of this invention is achieved through the following technical solution: a HANG monitoring and analysis method for distributed model training, comprising the following steps:

[0010] Within a single continuous sampling window, the call stack information of the same batch of nodes in the same training task of the training cluster is periodically collected;

[0011] The stack information of all ranks under the same node within a single sampling window is merged into a stack set;

[0012] For the same node, calculate the Jaccard similarity between the stack set obtained from the current sample and the previous sample;

[0013] For a node in the same batch of nodes, if the Jaccard similarity is higher than the preset threshold for a preset number of consecutive preset number of times and does not match the whitelist pattern, the training task is determined to have HANG.

[0014] After determining that a HANG has occurred, collect stack information of all nodes and all ranks globally, construct a StackTrie, identify abnormal ranks based on the coverage of the forked branches, and output alarm results and analysis results.

[0015] If it is determined that no HANG has occurred, the above steps are repeated in the next sampling window with a different batch of nodes.

[0016] Furthermore, the number of nodes in the same batch is 4; when the number of nodes in the training cluster is less than 4, all nodes are sampled.

[0017] Furthermore, the preset number of times is 3, and the preset threshold is 0.95.

[0018] Furthermore, the whitelist mode includes at least one of the checkpoint, save model, load data, or DataLoader modes.

[0019] Furthermore, the method for identifying abnormal Ranks includes: calculating the Rank coverage of each branch in StackTrie, and identifying the Ranks in branches with coverage below the minority threshold as abnormal Ranks.

[0020] Furthermore, the method also includes: when nodes exceeding a preset proportion (e.g., 3 / 4) are determined to be HANG, the global state of the training task is determined to be HANG, and a red alarm is displayed on the monitoring interface.

[0021] This invention also provides a HANG monitoring and analysis system for distributed model training, comprising:

[0022] The sampling module is used to periodically collect the call stack information of the same batch of nodes in the same training task of the training cluster within a single continuous sampling window, and to merge the stack information of multiple ranks under the same node into a stack set within a single sampling window.

[0023] The similarity analysis module is used to calculate the Jaccard similarity between the current sample and the stack set obtained from the previous sample of the same node;

[0024] The HANG determination module is used to determine whether the training task has been HANGed. Specifically, for a certain node in the same batch of nodes, if the Jaccard similarity is higher than a preset threshold for a consecutive preset number of times and does not match the whitelist pattern, the training task is determined to have been HANGed.

[0025] The global analysis module is used to collect stack information of all nodes and all ranks globally after determining that a HANG has occurred, construct a StackTrie, and identify abnormal ranks based on the coverage of the forked branches.

[0026] The alarm module is used to output HANG alarms and analysis results.

[0027] Furthermore, the system also includes a visualization module for displaying global status, node status, Rank details, flame graph, and alarm red lights on the monitoring interface.

[0028] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program that, when executed by the processor, implements the method described herein.

[0029] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method described thereon.

[0030] The beneficial effects of this invention are as follows: This invention achieves accurate identification and rapid location of hidden HANGs in distributed training through a combined mechanism of "periodic sampling + continuous similarity determination + whitelist filtering + global branching analysis." Compared with methods relying solely on GPU utilization, heartbeats, or log keywords, this invention can trigger alarms promptly even when the process has not exited but training has essentially stalled, significantly reducing false negatives. Simultaneously, by identifying whitelist scenarios, it avoids misjudging normal long-term blocking events such as checkpoints and data loading as faults, reducing false positives. Furthermore, by combining StackTrie and Rank branch coverage analysis, alarm results can be drilled down to the abnormal Rank level, helping users quickly locate the source of the problem, shorten troubleshooting time, and improve the stability, accuracy, and practicality of large-scale distributed training monitoring.

[0031] From an engineering perspective, the monitoring results of this invention have three levels: first, node-level early warning, indicating that the call paths of certain nodes remain unchanged for a long time; second, global-level alarm, indicating that the training task has entered the HANG state; and third, rank-level localization, indicating the execution branch most likely to cause an anomaly. With these three levels interconnected, operations and maintenance personnel can complete the closed loop of "discovery—confirmation—localization—handling" within a single monitoring interface, without having to switch back and forth between log platforms, monitoring platforms, and scheduling platforms.

[0032] From a scalability perspective, this invention parameterizes the number of sampling nodes, similarity threshold, number of consecutive judgments, minority threshold, and whitelist rules. Therefore, it can adapt to both small-scale single-machine multi-GPU training in R&D environments and distributed training at the thousand-GPU level in production environments. In large-scale training scenarios, the system can also control additional overhead through representative sampling and state machine hierarchies, keeping monitoring costs within an acceptable range.

[0033] From a robustness perspective, this invention does not rely on any specific framework, communication library, or deep learning engine. Regardless of the model structure used in the training task, as long as the process call stack can be collected, this invention can be used for HANG monitoring and analysis. Therefore, this invention has good versatility and portability, and can be reused across different training platforms. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a diagram of the overall system architecture described in this invention;

[0036] Figure 2 This is a flowchart of the HANG sampling and detection process described in this invention;

[0037] Figure 3 This is a schematic diagram of the problem Rank analysis described in this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0040] like Figure 1 As shown, this embodiment of the invention provides a HANG monitoring and analysis system for distributed model training, comprising:

[0041] The sampling module is used to periodically collect the call stack information of the same batch of nodes in the same training task of the training cluster within a single continuous sampling window, and to merge the stack information of multiple ranks under the same node into a stack set within a single sampling window.

[0042] The similarity analysis module is used to calculate the Jaccard similarity between the current sample and the stack set obtained from the previous sample of the same node.

[0043] The HANG determination module is used to determine whether a training task has been HANGed. Specifically, for a node in the same batch of nodes, if the Jaccard similarity is higher than a preset threshold for a consecutive preset number of times and does not match the whitelist pattern, the training task is determined to have been HANGed.

[0044] The global analysis module is used to collect stack information of all nodes and all ranks globally after determining that a HANG has occurred, construct a StackTrie, and identify abnormal ranks based on the coverage of the forked branches.

[0045] The alarm module is used to output HANG alarms and analysis results.

[0046] like Figure 2 As shown, corresponding to the system described above, this embodiment of the invention provides a HANG monitoring and analysis method for distributed model training, comprising the following steps:

[0047] S1. Within a single continuous sampling window, periodically collect the call stack information of the same batch of nodes in the same training task of the training cluster.

[0048] S2. Merge the stack information of multiple Ranks under the same node within a single sampling window into a stack set.

[0049] S3. For the same node, calculate the Jaccard similarity between the stack set obtained from the current sample and the previous sample.

[0050] S4. For a node in the same batch of nodes, if the Jaccard similarity is higher than the preset threshold for a preset number of consecutive preset times and does not match the whitelist pattern, it is determined that the training task has HANG.

[0051] S5. After determining that a HANG has occurred, collect stack information of all nodes and all ranks globally, construct a StackTrie, identify abnormal ranks based on the coverage of the forked branches, and output alarm results and analysis results.

[0052] S6. If it is determined that no HANG has occurred, the above steps are repeated in the next sampling window with a different batch of nodes.

[0053] Example 1

[0054] After the training task starts, the monitoring system activates a periodic sampling thread in the background. This sampling thread selects multiple nodes from the training cluster to collect stack information according to a pre-configured time interval. During sampling, the system records the node identifier, Rank identifier, sampling timestamp, and stack content. To avoid short-term fluctuations affecting the judgment, the system does not immediately draw conclusions based on a single sample, but instead organizes multiple sampling results into a continuous window for comparison.

[0055] Within a continuous sampling window, priority is given to ensuring the consistency of the collected node set. If the total number of nodes is small and does not reach the preset number, all nodes are collected; if the total number of nodes is large and reaches the preset number, a subset of nodes is selected according to a fixed strategy and remains unchanged in consecutive rounds. This sampling strategy ensures the comparability of adjacent samples and avoids misjudgments caused by the inherent differences between different nodes.

[0056] The sampled stack data enters the normalization process. The system merges the stack structures of multiple ranks within the same node, removing noise information unrelated to the deadlock determination, such as line number changes in different call frames, differences in temporary variables, or additional fields unrelated to the execution path. After merging, each node forms a more stable stack set, facilitating subsequent similarity determination.

[0057] For example, during the sampling process, 4 nodes are dynamically selected, and large model training typically uses 8 GPUs per node, so the total number of ranks collected is 32.

[0058] Example 2

[0059] To determine whether a training task has encountered a hang, a node-level determination is used to address the question of whether training is truly stuck. Since some stages in distributed training inevitably involve long periods of synchronization or waiting, the system does not rely on a single sample for direct determination. Instead, it requires that samples from several consecutive rounds meet a high similarity condition, with the Jaccard similarity exceeding a preset threshold for a predetermined number of consecutive rounds.

[0060] For example, the Jaccard similarity between the current sample and the stack set obtained from the previous sample of the same node is calculated. If the Jaccard similarity of three consecutive samples is higher than 0.95, the system determines that the stack of the node has minimal changes.

[0061] If the current stack of a node does not belong to the whitelist mode, the system marks it as a suspected HANG node. If multiple nodes are suspected of being HANG simultaneously, the system further determines the global training state as HANG. By adopting the above method of "node-level screening first, and global-level confirmation later", the system can avoid single-point anomalies from directly causing global misjudgments.

[0062] In some implementations, the system can also introduce a voting strategy, which involves counting the number of nodes that meet the high similarity condition in each round of sampling. When the number exceeds a certain proportion of the total number of nodes, a global HANG decision is triggered. This proportion can be configured according to the cluster size to adapt to different training environments.

[0063] Example 3

[0064] For node stacks that the system determines to have undergone minimal changes, a whitelist mechanism is crucial for reducing false positives. During training phases such as checkpoint saving, data preprocessing, model recovery, weight loading, and initial communication, stacks often remain unchanged for extended periods, but this is considered normal for long processing times. Without a whitelist mechanism, the system might misclassify these phases as HANG (Hardware).

[0065] Therefore, the system allows the maintenance of a whitelist set. Whitelist entries can be derived from operational experience, historical alarm statistics, manual verification results, or online learning results. The whitelist can be matched using exact function name matching, stack fragment matching, call chain matching, or regular expression matching. As long as the current stack meets the whitelist criteria, even if its similarity is high, a HANG will not be triggered immediately.

[0066] In an optional embodiment, the system also records the context of each whitelist hit, including sampling time, Rank, node, and call chain, for subsequent optimization of whitelist rules. This allows the whitelist to evolve from a static configuration into a maintainable knowledge base.

[0067] Example 4

[0068] like Figure 3 As shown, once the system confirms HANG, it continues to collect complete stack traces for all nodes and all ranks, and constructs a StackTrie. A StackTrie is a tree structure used to represent the branching relationships between different call paths. Each level of the tree represents a call frame or a path node, and each branch of the tree represents a common execution path for a set of ranks.

[0069] After the system is built, it calculates the branch coverage rate for each fork point, which is the number of Ranks contained on a particular branch and the proportion of those Ranks to the total number of Ranks. Branches with high coverage typically represent the common path of the majority of Ranks, while branches with significantly lower coverage represent minority paths. These minority paths exhibit behavior inconsistent with the majority of Ranks, potentially corresponding to abnormal waiting, abnormal communication, or local code path offsets.

[0070] The system uses the ranks in the minority branch as candidate anomaly ranks and can further count the number of times these ranks appear repeatedly at multiple forks. If a particular rank appears multiple times on the minority path, its anomaly suspicion is higher. Through this method, the system can not only detect "anomalies" but also narrow it down to "which rank is most suspicious".

[0071] Example 5

[0072] During a training task, the system found that multiple node stacks were almost unchanged during the continuous sampling phase, and the whitelist was not matched, so it determined that the training had entered the HANG stage. Subsequently, the system collected the global stack and built a StackTrie, and found that the vast majority of ranks were stuck on the same communication waiting path, but a few ranks entered different data loading paths.

[0073] In the fork coverage statistics, the communication waiting path covers most of the ranks, while the data loading path covers only a small number of ranks. The system marks the ranks located on the data loading path as abnormal ranks and further displays the node information, stack snapshot, and flame graph entry for that rank. Users can use this information to determine whether there are data read anomalies, file system blocking, or local scheduling issues with that rank.

[0074] The value of the analysis method in this embodiment lies in the fact that it does not attribute all anomalies to "training deadlock" in a general way, but rather links global deadlock with local deviations, helping users to identify the Rank that truly deviates from the majority of execution paths.

[0075] Example 6

[0076] Once a HANG node is identified, a red warning light appears on the monitoring interface homepage, indicating that the current training is in an abnormal stagnation state. Simultaneously, the system displays a similarity change curve, a list of HANG nodes, a list of abnormal Ranks, a flame graph entry point, and branch coverage information. Clicking on a node allows users to further view its stack snapshots across multiple sampling epochs; clicking on a Rank allows users to view its path location within the StackTrie.

[0077] The above alert displays not only serve as reminders but also provide an entry point for subsequent analysis. Ideally, the system should place information including "whether the current HANG is global," "which nodes were the first to become abnormal," "which ranks are more suspicious," and "whether the whitelist has been hit" on the same page, reducing the cost for users to switch between multiple modules.

[0078] Furthermore, the system supports configuring sampling interval, number of sampling nodes, number of consecutive judgments, similarity threshold, whitelist rules, and minority threshold as adjustable parameters. It can adapt to both small-scale testing environments and large-scale training environments at the kilocalorie level.

[0079] In the default implementation, the sampling interval is 30 seconds, the number of consecutive judgments is 3, the number of sampling nodes is 4, and the similarity threshold is 0.95. If the training task is small, the number of sampling nodes can be appropriately reduced; if the training task fluctuates greatly, the number of consecutive judgments can be appropriately increased to further reduce false positives. The minority threshold can also be dynamically adjusted according to the number of Ranks to avoid false positives due to insufficient branch numbers in small-scale clusters.

[0080] Example 7

[0081] In a further implementation, the system can also perform unified modeling of the time series before and after a HANG event. Specifically, the system maintains a state machine for each node, with states including normal sampling, suspected stagnation, whitelist hit, suspected HANG, confirmed HANG, and resumption of observation. Each sampling update updates the state machine, and based on the latest round of similarity, whitelist matching results, and branch coverage results, it determines whether to proceed to the next state. Through this setup, the system does not rely on judgments at a single moment but can combine multiple sampling rounds into a continuous chain of evidence.

[0082] When a node is marked as a suspected HANG, the system does not immediately remove it from the training process but continues to observe stack changes in the next round. If the node recovers its changes in subsequent sampling, the system can automatically revert it from the suspected HANG state to the normal sampling state and record this process as "short-term blocking recovery". If the node maintains the exact same stack for multiple consecutive rounds and does not match the whitelist, then the system upgrades it to a confirmed HANG. This state machine design can significantly reduce false alarms caused by transient jitter, network fluctuations, task switching, or brief resource contention.

[0083] In an optional implementation, the system also maintains a historical stack fingerprint for each Rank. The stack fingerprint is a stable identifier generated by normalizing the call stack by frame sequence, key function names, and hierarchical position. If the same Rank maintains the same fingerprint across multiple consecutive time windows, it indicates that the execution path of that Rank is highly stable; if the fingerprint of a certain Rank deviates from that of most Ranks at multiple forks, that Rank is more likely to be the root cause. By introducing stack fingerprints, the system can further reduce stack noise caused by parameter differences, log line number changes, or different environmental variables.

[0084] Example 8

[0085] In terms of system deployment, the HANG monitoring module can be deployed on the control plane where the training scheduling service resides, or it can be deployed as a bypass data acquisition service on the monitoring nodes of the training cluster. The control plane deployment method is suitable for scenarios where the status of training tasks is uniformly orchestrated, while the bypass deployment method is suitable for scenarios where it is not desired to affect the main training process. Regardless of the deployment method used, the monitoring module only reads stack snapshots and scheduling information and does not intervene in the training computation process, thus it does not change the execution semantics of the training itself.

[0086] When the training scale is small and the number of ranks is limited, the system can directly sample all nodes. When the training scale is large, the system can adopt a tiered sampling strategy, that is, first select a small number of representative nodes for coarse-grained judgment, and then perform encrypted sampling on suspected abnormal nodes. Tiered sampling can significantly reduce sampling overhead while ensuring good detection sensitivity in large-scale scenarios. For long-running training tasks, the system can also perform adaptive sampling in conjunction with the training phase, such as appropriately reducing the sampling frequency during the initialization phase, appropriately increasing the sampling frequency during the communication-intensive phase, and temporarily relaxing the HANG judgment condition during the checkpoint phase.

[0087] Furthermore, the system can correlate alarm results with a historical fault database. Each time a HANG (Hazard Anomaly) is confirmed, the system writes the corresponding node information, Rank information, stack snapshot, whitelist hit status, and abnormal branch path into the historical database, forming a continuously accumulating diagnostic sample. Subsequently, when similar symptoms reappear in similar tasks, the system can prioritize hitting historical patterns, thereby shortening the localization time. This historical database not only aids in operations and maintenance but also helps in the continuous optimization of whitelist and threshold configurations.

[0088] Example 9

[0089] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a program that, when executed by the processor, implements the method described above.

[0090] Example 10

[0091] This invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method described thereon.

[0092] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0093] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0094] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A HANG monitoring and analysis method for distributed model training, characterized in that, Includes the following steps: A periodic sampling thread is enabled in the background to periodically collect the call stack information of the same batch of nodes in the same training task of the training cluster within a single continuous sampling window. The stack information of all Ranks under the same node within a single sampling window is merged into a stack set; wherein, a historical stack fingerprint is maintained for each Rank, and the stack fingerprint is a stable identifier generated by normalizing the call stack by frame sequence, key function name and hierarchical position; For the same node, calculate the Jaccard similarity between the stack set obtained from the current sample and the previous sample; For a node in the same batch of nodes, if the Jaccard similarity is higher than a preset threshold for a preset number of consecutive preset times and does not match the whitelist pattern, the node is judged as a suspected HANG. After the node is marked as a suspected HANG, the stack changes are observed in the next round. If the node recovers in subsequent sampling, it is rolled back from the suspected HANG state to the normal sampling state. When more than a preset proportion of nodes are judged as suspected HANG, the training task is judged to have HANG. After determining that a HANG has occurred, the stack information of all nodes and all ranks is collected globally to construct a StackTrie. Abnormal ranks are identified based on the coverage of the branching branches, and alarm results and analysis results are output. The method of identifying abnormal ranks includes: calculating the rank coverage of each branch in the StackTrie, and taking the ranks in the branches with coverage below the minority threshold as abnormal ranks. If it is determined that no HANG has occurred, the above steps are repeated in the next sampling window with a different batch of nodes.

2. The method according to claim 1, characterized in that, The number of nodes in the same batch is 4; when the number of nodes in the training cluster is less than 4, all nodes are sampled.

3. The method according to claim 1, characterized in that, The preset number of times is 3, and the preset threshold is 0.

95.

4. The method according to claim 1, characterized in that, The whitelist mode includes at least one of the checkpoint, save model, load data, or DataLoader modes.

5. The method according to claim 1, characterized in that, Also includes: When the global status of the training task is determined to be HANG, a red alarm will be displayed on the monitoring interface.

6. A HANG monitoring and analysis system for distributed model training, used to implement the method described in any one of claims 1-5, characterized in that, include: The sampling module is used to periodically collect the call stack information of the same batch of nodes in the same training task of the training cluster within a single continuous sampling window, and to merge the stack information of multiple ranks under the same node into a stack set within a single sampling window. The similarity analysis module is used to calculate the Jaccard similarity between the current sample and the stack set obtained from the previous sample of the same node; The HANG determination module is used to determine whether the training task has been HANGed. Specifically, for a certain node in the same batch of nodes, if the Jaccard similarity is higher than a preset threshold for a consecutive preset number of times and does not match the whitelist pattern, the training task is determined to have been HANGed. The global analysis module is used to collect stack information of all nodes and all ranks globally after determining that a HANG has occurred, construct a StackTrie, and identify abnormal ranks based on the coverage of the forked branches. The alarm module is used to output HANG alarms and analysis results.

7. The system according to claim 6, characterized in that, It also includes a visualization module, which displays the global status, node status, Rank details, flame graph, and alarm red lights on the monitoring interface.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a program that, when executed by the processor, implements the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 5.

Images