Communication fault early warning and positioning method and system, center node and training node

Abstract

The invention relates to the technical field of artificial intelligence, and provides a communication fault early warning and positioning method and system, a center node and training nodes, and the method comprises the steps: receiving communication metadata which is transmitted by at least one training node and is associated with each communication operation, the communication metadata is acquired after the training node intercepts a communication interface of the distributed training framework; based on the communication metadata, performing real-time analysis on the execution state of each communication operation to determine whether each communication operation is abnormal; and when it is determined that any communication operation is abnormal, determining a fault node and generating early warning information based on the communication metadata of any communication operation. According to the method, the communication metadata of each communication operation can be accurately captured by intercepting the communication interface level of the distributed training framework, so that real-time analysis can be performed according to the metadata to identify the communication hovering state, and early warning and accurate positioning of a fault are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and distributed systems, in particular to communication fault diagnosis in model distributed training, and more particularly to a communication fault early warning and positioning method and system, a center node and a training node. BACKGROUND

[0002] Currently, large model training represented by large language models (LLM) has entered the era of cluster with ten thousand nodes. These training tasks highly depend on distributed strategies such as data parallelism and model parallelism. The core of these strategies is efficient and frequent distributed communication. However, in such a complex environment of super large scale and high concurrency, communication faults become a key bottleneck that restricts the stability and efficiency of training. Communication delays, non-responses, and even deadlocks, once they occur, are difficult to quickly locate, resulting in a waste of huge computing resources and a long fault recovery time.

[0003] To solve the problem of communication fault positioning in distributed training, traditional methods rely on massive log analysis or monitoring performance indicators. The main defects of these solutions are that the information granularity is too coarse, lack of real-time, it is difficult to directly associate the problem to specific communication operations, and it is unable to effectively perceive whether the communication operation is in a pending state, which is often an early sign of communication deadlock. In addition, some more advanced tools, such as general distributed tracing systems or built-in performance analysis tools of frameworks (such as PyTorch Profiler), also have limitations. They either are not suitable for high-frequency communication scenarios due to strong invasiveness and huge performance overhead; or are difficult to extend to the scale of ten thousand nodes, and lack the ability to capture key metadata such as communication groups and participating nodes, and also cannot effectively warn the pending state of communication operations. SUMMARY

[0004] The present application provides a communication fault early warning and positioning method and system, a center node and a training node, to solve the defects of related technologies in dealing with ten thousand node large model training scenarios, such as not timely positioning, not accurate positioning, high performance overhead, and inability to capture core fault information (especially the pending state of communication).

[0005] The present application provides a communication fault early warning and positioning method, which is applied to a center node, and the method comprises: receiving communication metadata associated with each communication operation sent by at least one training node, the communication metadata being collected after the at least one training node intercepts the communication interface of a distributed training framework; based on the communication metadata, performing real-time analysis on execution states of the communication operations, and determining whether there is an abnormality in the communication operations according to an analysis result, wherein the real-time analysis at least comprises detecting whether the communication operations start or end within a preset time length, so as to determine whether the communication operations are in a suspended state; In a case where it is determined that any communication operation has an abnormality, a fault node is determined based on communication metadata of the any communication operation, and early warning information is generated.

[0006] According to the communication fault early warning and positioning method provided by the application, the detection of whether the communication operations start or end within a preset time length, so as to determine whether the communication operations are in a suspended state, comprises: For any communication operation, if a start time is not detected in communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

[0007] According to the communication fault early warning and positioning method provided by the application, the detection of whether the communication operations start or end within a preset time length, so as to determine whether the communication operations are in a suspended state, comprises: For any communication operation, if a start time is not detected in communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

[0008] According to the communication fault early warning and positioning method provided by the application, the determination of a fault node based on communication metadata of the any communication operation and the generation of early warning information, comprises: For any communication operation in a suspended state, among all training nodes participating in the any communication operation, a training node in which a start time or an end time is not detected within a preset time length is determined as the fault node. Based on communication metadata associated with the any communication operation sent by the fault node, early warning information is generated and an alarm is triggered.

[0009] According to the communication fault early warning and positioning method provided by the application, the real-time analysis on execution states of the communication operations based on the communication metadata, and the determination of whether there is an abnormality in the communication operations according to an analysis result, comprises: Based on a start time and an end time in communication metadata of a completed communication operation, a duration of the completed communication operation is determined. In a case where the execution duration exceeds a baseline duration, it is determined that the completed communication operation has a timeout exception, and the baseline duration is preset or dynamically learned.

[0010] According to the communication fault early warning and positioning method provided by the application, the method further comprises: The life cycle, delay distribution and abnormal trend of the communication operation with the exception are displayed through a visual interface.

[0011] According to the communication fault early warning and positioning method provided by the application, the communication metadata comprises at least one of an operation type, a start time, a communication process identifier, a communication group identifier, a process list participating in communication, a shape and data type of a communication tensor, an end time and an operation result.

[0012] The application further provides a communication fault early warning and positioning method, which is applied to a training node, and the method comprises: The communication interface of a distributed training framework is intercepted to capture each communication operation, and communication metadata associated with the communication operation is collected during execution of the communication operation; The communication metadata of each communication operation is sent to a central node, so that the central node performs real-time analysis on the execution state of each communication operation based on the communication metadata of each communication operation, and determines whether each communication operation has an exception according to an analysis result; The real-time analysis at least comprises detecting whether each communication operation starts or ends within a preset duration to determine whether each communication operation is in a hovering state; and the central node is configured to determine a fault node and generate early warning information based on the communication metadata of any communication operation in a case where it is determined that the any communication operation has an exception.

[0013] According to the communication fault early warning and positioning method provided by the application, the communication metadata of each communication operation is sent to a central node, and the method comprises: The communication metadata of each communication operation is temporarily stored in a local cache; The communication metadata in the local cache is uploaded to the central node asynchronously and in batches.

[0014] According to the communication fault early warning and positioning method provided by the application, the local cache is a ring buffer or a non-blocking queue.

[0015] The application further provides a communication fault early warning and positioning device, which is applied to a central node, and the device comprises: a data receiving unit configured to receive communication metadata associated with each communication operation sent by at least one training node, the communication metadata being collected after the at least one training node intercepts a communication interface of a distributed training framework; an abnormality analyzing unit configured to analyze execution states of the communication operations in real time based on the communication metadata, and determine whether any of the communication operations is abnormal according to an analysis result, wherein the real-time analysis at least includes detecting whether the communication operations start or end within a preset time length to determine whether the communication operations are in a suspended state; a positioning and early warning unit configured to determine a faulty node and generate early warning information based on communication metadata of any communication operation that is determined to be abnormal.

[0016] The application further provides a communication fault early warning and positioning device, which is applied to a training node and includes: an intercepting and capturing unit configured to intercept a communication interface of a distributed training framework to capture each communication operation, and collect communication metadata associated with the communication operation during execution of the communication operation; a data sending unit configured to send the communication metadata of each communication operation to a center node, so that the center node analyzes execution states of the communication operations in real time based on the communication metadata of the communication operations, and determines whether any of the communication operations is abnormal according to an analysis result; wherein the real-time analysis at least includes detecting whether the communication operations start or end within a preset time length to determine whether the communication operations are in a suspended state; and the center node is configured to determine a faulty node and generate early warning information based on communication metadata of any communication operation that is determined to be abnormal.

[0017] The application further provides a center node including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the communication fault early warning and positioning method applied to the center node as described above.

[0018] The application further provides a training node including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the communication fault early warning and positioning method applied to the training node as described above.

[0019] The application further provides a communication fault early warning and positioning system including the center node and at least one training node as described above.

[0020] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the communication fault early warning and positioning method.

[0021] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the communication fault early warning and positioning method.

[0022] The application provides a communication fault early warning and positioning method and system, a center node and a training node, which can capture communication metadata associated with communication operations that cannot be reached by a traditional scheme by precisely intercepting at a communication interface level of a distributed training framework of each training node, and lays a foundation for accurate analysis. After receiving the communication metadata associated with each communication operation sent by each training node, the center node can perform real-time analysis according to the metadata to detect a communication suspension state, so that the communication operation in the suspension state of not being initiated or being initiated but not completed can be captured in real time before the fault actually occurs, thereby realizing early warning. In addition, when an abnormal communication operation is detected, the communication metadata can be combined for correlation analysis, so that the source of the fault can be accurately located, and early warning information containing detailed abnormal conditions can be generated, greatly shortening the fault troubleshooting time and ensuring the stability and efficiency of large-scale training tasks. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0024] Figure 1 is a structural schematic diagram of the communication fault early warning and positioning system provided by the application; Figure 2 is one of the flowcharts of the communication fault early warning and positioning method provided by the application; Figure 3 is another flowchart of the communication fault early warning and positioning method provided by the application; Figure 4 is one of the structural schematic diagrams of the communication fault early warning and positioning device provided by the application; Figure 5 is another structural schematic diagram of the communication fault early warning and positioning device provided by the application; Figure 6 is a structural schematic diagram of the center node provided by the application; Figure 7 is a structural schematic diagram of a training node provided by the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0026] In recent years, with the rapid development of large model technologies represented by large language models (LLM), the training scale has entered the era of super-large-scale clusters with tens of thousands of GPUs. In order to efficiently utilize such a huge amount of computing resources, training tasks highly depend on complex distributed parallel strategies, such as data parallelism, model parallelism, and expert parallelism. The core of the implementation of these strategies is efficient and frequent distributed communication between nodes, which is usually supported by underlying communication libraries and distributed training frameworks (such as PyTorchDistributed).

[0027] However, in such a super-large-scale, highly asynchronous, and communication-computing mixed complex environment with tens of thousands of GPUs, communication failures have become a prominent challenge affecting the stability and efficiency of training. Once communication delays, message losses, node unresponsiveness, and even communication deadlocks occur, the root cause is often difficult to be quickly located. This directly leads to a high cost of computing resources being wasted for a long time, and the average detection time and average recovery time of faults are high, which seriously restricts the research and development iteration efficiency of large models.

[0028] In order to cope with the above challenges, the industry has proposed a variety of solutions for distributed training troubleshooting, but they all have significant limitations in the tens of thousands of GPU training scenario: (1) Log analysis: This method relies on collecting and analyzing the huge amount of log files generated by each node and component in the cluster. Its limitations are that the log volume is huge, the effective information is easily overwhelmed, and manual troubleshooting is time-consuming and labor-intensive; and it is often a post-mortem analysis after the fault occurs, lacking real-time and proactive warning capabilities. In addition, the log usually does not contain semantic information that can directly reflect the Pending state of the communication operation "initiated but not completed" (such a state is often an early sign of communication deadlock or node death).

[0029] (2) Index monitoring: This scheme collects and displays performance indicators such as CPU utilization, GPU utilization, network bandwidth, and video memory occupation in real time. Although it can reflect the overall resource health of the cluster, its core defect is that the monitoring granularity is coarse. These macro indicators are difficult to directly link to a specific distributed communication operation such as broadcast, send, receive, all reduce, all gather, etc., so it is also impossible to perceive whether a communication operation has fallen into the Pending state.

[0030] (3) General distributed tracing scheme: This type of scheme (such as systems based on the OpenTelemetry standard) aims to trace the complete call chain of a request in a distributed system. When applied to this special scenario of large model training, its pain points are very obvious: first, it is highly invasive, introducing a large number of hooks or modifications at the training framework and code level, increasing maintenance complexity; second, it has a huge performance overhead, and the high-frequency communication in training generates a large amount of tracing data, and the generation, transmission, and storage overhead of full tracing may seriously slow down the training main process; finally, the tracing granularity cannot go deep into the communication library bottom layer, making it difficult to capture training-specific metadata such as communication groups, participating nodes, and tensor shapes, which are crucial for diagnosing communication problems.

[0031] (4) Performance analysis tools built into training frameworks: Tools such as PyTorch Profiler focus mainly on performance bottleneck analysis, providing a timeline view of operations to help developers optimize code. However, these tools generally perform well in single-node or a few-node environments, but are difficult to scale to large-scale real-time monitoring of thousands of cards, and cannot provide a global, real-time communication status view. At the same time, their design goal is performance optimization rather than fault warning, so they also generally lack a warning mechanism for the Pending state.

[0032] To this end, the present application provides a low-overhead, high-precision, and real-time communication fault warning and positioning method and system to overcome the above-mentioned defects. The technical solutions provided by the present application will be described in detail below.

[0033] Figure 1 is a structural schematic diagram of the communication fault warning and positioning system provided by the present application, as Figure 1As shown, the system aims to solve the problem of untimely and inaccurate communication fault positioning in the distributed training process of large-scale deep learning models. Specifically, the communication fault early warning and positioning system is a monitoring and diagnosis system applied to large model distributed training scenarios, which adopts a client-server (Client-Server) architecture. In the embodiment of the present application, the system includes a center node 110 and at least one training node 120, wherein the training node 120 serves as a client and is responsible for collecting underlying communication behavior data during the execution of the training task; the center node 110 serves as a server and is responsible for gathering, storing, and real-time analyzing all the data reported by the clients, thereby realizing early warning of communication anomalies and accurate positioning of the fault source.

[0034] The center node 110 is the core analysis and decision unit of the system. It can be a single physical server or a cluster composed of multiple servers with high availability and horizontal expansion capability. The main function of the center node 110 is to receive all the communication metadata uploaded by the training nodes 120, persistently store these data (for example, store them into a time series database), and run a real-time analysis engine. The engine continuously processes data streams to detect abnormal states in communication operations, such as pending, timeout, etc., and generates warning information when an abnormality is found.

[0035] The training node 120 is a specific computing unit that executes distributed training tasks. Usually, a training node 120 can be a physical machine or a virtual machine equipped with multiple high-performance computing accelerator cards. These accelerator cards can be graphics processing units (GPU), general-purpose computing on graphics processing units (GPGPU), tensor processing units (TPU), etc., and in the field, GPU is usually taken as a typical representative. In a training cluster with ten thousand cards, there will be hundreds or thousands of such training nodes 120.

[0036] It should be noted that the training node 120 can be an independent physical or virtual server, which has an independent network identity such as an Internet Protocol (IP) address. A training node 120 is a physical resource carrying unit, which can install one or more GPUs. The GPU is the core computing hardware installed on the training node 120, and each GPU is usually assigned a local device identifier (gpu_id, for example, on a server with 8 GPU cards, gpu_id can be from 0 to 7) within a training node 120.

[0037] In order to fully utilize all GPU resources, a separate control process is usually started for each piece of GPU in the cluster. A distributed training framework (such as PyTorch Distributed) assigns a globally unique integer identifier to each such process, which is the communication process identifier (denoted as rank_id). It should be understood that the rank is the basic unit of communication, which is a logical concept at the level of the distributed training framework. For example, a system consisting of 2 training nodes, each node having 8 GPUs, a total of 16 processes will be run, and their rank_id will be globally uniquely numbered from 0 to 15. The present application can locate the fault to the globally unique communication process by capturing the rank_id associated with each communication operation, and in combination with the node IP (denoted as node_ip) and local gpu_id of the process, the physical fault source can be accurately traced.

[0038] Through the establishment of the above system architecture, the communication events scattered on each training node can be gathered to the center node for global and real-time correlation analysis, providing a basic platform for early warning and accurate positioning of communication faults, thereby effectively shortening the troubleshooting time and improving the stability and resource utilization of large-scale training clusters.

[0039] The structure and function of the center node and the training node of the communication fault early warning and positioning method provided by the present application will be further described below through specific embodiments.

[0040] Figure 2 is one of the flowcharts of the communication fault early warning and positioning method provided by the present application, as shown in Figure 2 The method can be applied to the center node of the communication fault early warning and positioning system described in the above embodiments. By executing the method, the center node can monitor the communication behavior in the large-scale distributed training process in real time, and realize early warning and accurate positioning of potential faults. The method comprises the following steps: Step 210, receiving communication metadata associated with each communication operation sent by at least one training node, the communication metadata being collected by the at least one training node after intercepting the communication interface of the distributed training framework.

[0041] Specifically, in large model distributed training, multiple processes (i.e. ranks) on each training node need to work together through high-frequency communication operations, such as all_reduce, all_gather, send, recv, etc. These communication operations are usually provided by the underlying distributed training framework, such as the torch.distributed module in the PyTorch framework. This module provides a series of functions or methods as communication interfaces for the upper layer training code to call.

[0042] In order to capture each communication without invading the user's training code, the training node will perform an interception (i.e. Hook) operation on the above communication interface. This is a non-intrusive programming technique, for example, the original communication interface function can be dynamically replaced or wrapped at runtime through monkey patching or decorator technology. In this way, whenever the upper layer code calls a communication operation, the injected interception code will be executed first, and then the original communication function will be executed. It should be understood that monkey patching is a technique for dynamically modifying the behavior of a module or object, which allows functions or methods in the module to be replaced or extended at runtime. The decorator technique can be used to add functionality to a function without modifying the original function code.

[0043] In the interception operation, the training node collects key information that is strongly related to the current communication operation, which is the communication metadata. These metadata are crucial for subsequent fault analysis. For example, the communication metadata can include operation type, start time, communication process identifier, communication group identifier, list of processes participating in the communication, shape and data type of the communication tensor, end time, operation result, etc. Here, the operation type (denoted as op_name) is used to identify the specific type of communication operation, such as all_reduce, all_gather, send, etc.; the start time (denoted as start_timestamp) refers to the time when the communication operation starts to execute; the communication process identifier (i.e., rank_id) refers to the globally unique ID of the process that executes the communication operation, which can be directly associated with a specific GPU; the communication group identifier (denoted as process_group_id) is used to identify the communication group to which the current operation belongs, which is used to distinguish different parallel communication tasks; the list of processes participating in the communication (denoted as ranks_in_group) is a list containing the rank_id of all participants in this communication operation; the shape (denoted as tensor_shape) and data type (denoted as tensor_dtype) of the communication tensor refer to the size and type of the data transmitted in this communication, which helps to analyze the communication load; the end time (denoted as end_timestamp) is used to record the timestamp when the communication operation is completed (i.e., execution ends); and the operation result (denoted as status) is used to record whether the operation is successfully completed, whether it is timed out or has an error, etc.

[0044] After the training node collects the communication metadata, it does not immediately send each piece of data to avoid network storm and performance impact on the training main process caused by high-frequency sending. A preferred way is that the training node can maintain a local cache (such as a ring buffer) internally, and first write the collected metadata to the memory cache quickly. Then, through an independent background thread or process, the data is read from the cache in batches in an asynchronous manner, and after efficient serialization (such as Protocol Buffers) and compression, it is sent to the center node through HTTP / 2 (i.e., Hypertext Transfer Protocol 2.0, such as gRPC) or TCP (Transmission Control Protocol) long connection, etc. The data collection service on the center node is responsible for receiving and parsing these batch data.

[0045] At step 220, based on the communication metadata, the execution status of each communication operation is analyzed in real time, and whether there is an abnormality in each communication operation is determined according to the analysis result, wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a suspended state.

[0046] Specifically, after receiving the batch communication metadata uploaded from all training nodes, the center node can decompress, deserialize and preliminarily check these metadata, and store the data that passes the check in a database for storing communication metadata. The database here is preferably a time series database (such as TimescaleDB, InfluxDB, ClickHouse, etc.) to efficiently store and query massive time series data.

[0047] It can be understood that the database needs to be constructed in advance before storing the communication metadata. When constructing the database, the table structure can be carefully designed, for example, the data table can include comm_op_id, op_name, rank_id, node_ip, gpu_id, process_group_id, ranks_in_group, start_timestamp, end_timestamp, status, tensor_shape, tensor_dtype and other key fields, and efficient indexes are established. Here, comm_op_id refers to a unique identifier generated for each communication operation; node_ip refers to the IP address of the training node, and the center node can obtain the IP address of the training node when receiving the communication metadata sent by the training node.

[0048] Further, the center node can use a stream processing framework (such as Apache Flink) or a customized background service to build a real-time analysis engine. The engine continuously consumes the communication metadata stream received from each training node to analyze the execution status of the communication operation. It should be understood that the life cycle of a communication operation can have multiple states. For example, when the center node receives a start_timestamp record about a communication operation for a certain rank_id but has not received its end_timestamp, the operation is in the "executing" state. When the end_timestamp is also received, the state becomes "completed".

[0049] In the embodiments of the present application, the execution state of each communication operation is analyzed in real time to determine whether there is an abnormality in each communication operation. Here, the focus is on a special abnormal state, namely, the pending state. In distributed communication, the pending state refers to the state in which one or more processes have initiated or are ready to participate in a communication operation, but due to the failure of some participants (i.e., some participating processes) to join or complete as expected, other processes are trapped in an infinite wait and cannot continue to perform subsequent tasks. This state is a typical early sign of communication deadlock or node death.

[0050] Specifically, in order to determine whether a certain communication operation is in a pending state, the center node can be implemented by detecting whether the communication operation starts within a preset time length or ends within a preset time length. Whether it is for collective communication (such as broadcast, all_reduce, all_gather, etc.) or point-to-point communication (such as send, recv), the analysis engine of the center node can know which ranks participate in this communication operation by reading the participating communication process list (namely, ranks_in_group) in the metadata. If the engine does not receive the start_timestamp record of all participating ranks within the set preset time length (for example, 30ms, 100ms or 1s, which can be dynamically adjusted according to the network environment and operation type), it can be determined that the communication operation is in a pending state. This usually means that those processes (ranks) that do not report the start time may be stuck in the previous calculation, process crash or network failure, which is the source of the entire communication group stagnation.

[0051] In addition, if the analysis engine has received the start_timestamp of all participating ranks, it means that all participants have entered the communication operation. However, within the preset time length thereafter, the engine fails to receive the end_timestamp record of all participating ranks, which means that a problem has occurred during the execution of the communication, such as the crash of a process, GPU or node during the communication, network partition causing messages to be unable to be delivered, or a logical deadlock at the algorithm level. Therefore, at this time, it can also be determined that the operation is in a pending state.

[0052] Through the above two detection methods, the embodiments of the present application can perceive the pending state of the communication in real time, thereby greatly migrating the fault discovery time point from post-analysis to early warning.

[0053] Step 230, in the case where it is determined that any communication operation has an abnormality, determining a fault node and generating warning information based on the communication metadata of the any communication operation.

[0054] It should be noted that when step 220 detects that a certain communication operation is abnormal (such as in a suspended state), the center node can further trace the fault and issue an alarm. The fault node refers to the specific training node or nodes that are most likely to cause the communication abnormality. In the embodiments of the present application, it can be accurately located to the IP address of the node (i.e. node_ip), the GPU identification on the node (i.e. gpu_id), and the corresponding communication process identification (i.e. rank_id).

[0055] Specifically, in determining the fault node that causes the communication operation abnormality, it can be achieved by analyzing the communication metadata associated with the communication operation. For example, taking the abnormal situation that the communication operation is in a suspended state as an example, for the suspension caused by a long time without starting the operation, the determination of the fault node is very direct, that is, the training node corresponding to the rank that has not reported start_timestamp within the preset time length. The center node can easily query the node_ip and gpu_id information associated with the rank_id of the missing start_timestamp in its database, and thus determine the fault node.

[0056] Once the fault node is determined, the center node will immediately generate a pre-warning information. The information should contain detailed context, such as: fault type, occurrence time, suspension duration, affected communication operation type, communication group identification, and most importantly, the node_ip, gpu_id and rank_id of the fault node. Subsequently, the pre-warning information can be notified to the operation and maintenance personnel through the alarm manager (such as Alertmanager) in the form of email, SMS or instant message, etc.

[0057] In the embodiments of the present application, by determining the fault node and generating the pre-warning information, the transition from passive response to active pre-warning is realized. The operation and maintenance personnel do not need to face a large amount of logs for needle-in-a-haystack troubleshooting after the fault occurs, but can receive the alarm containing accurate positioning clues at the first time. This greatly shortens the average detection time and average recovery time of the fault, avoids the huge waste of valuable computing resources caused by long-time idling, and guarantees the stability and efficiency of the training task.

[0058] The method provided by the embodiment of the application can capture communication metadata associated with communication operations that cannot be reached by a conventional solution by precisely intercepting at the communication interface level of the distributed training framework of each training node, thereby laying a foundation for accurate analysis. After receiving the communication metadata associated with each communication operation sent by each training node, the center node can perform real-time analysis according to the metadata to detect the communication suspension state, so that the communication operation in the suspension state of not being initiated or being initiated but not completed can be captured in real time before the fault actually occurs, thereby realizing early warning. In addition, when an abnormal communication operation is detected, the communication metadata can be associated and analyzed to accurately locate the source of the fault and generate warning information containing detailed abnormal conditions, thereby greatly shortening the fault troubleshooting time and ensuring the stability and efficiency of large-scale training tasks.

[0059] According to any of the above embodiments, the communication metadata includes at least one of an operation type, a start time, a communication process identifier, a communication group identifier, a process list participating in communication, a shape and data type of a communication tensor, an end time, and an operation result.

[0060] Specifically, the communication metadata collected by the training node when intercepting the communication interface and sent to the center node can include at least one of the above fields, but in the preferred embodiment, these fields will be as comprehensive as possible to form an information-rich event record. Each field is described in detail below.

[0061] The operation type (op_name) is a string or enumeration type field used to explicitly identify the specific type of intercepted communication operation, such as all_reduce, all_gather, broadcast, send, recv, barrier, etc. This field is the basis for analysis, and different types of operations have different communication patterns and performance expectations, for example, the default duration for suspension state detection will be distinguished according to op_name.

[0062] The start time (start_timestamp) is a high-precision timestamp (e.g., microsecond or millisecond level) that records the exact time when the communication operation is called on the training node. In addition, the start time is the starting point for calculating the execution duration and is also a key indicator for determining whether the process is in the suspension state.

[0063] The communication process identifier (rank_id) is a globally unique integer that identifies the process that initiates the communication operation, and is also a key for fault node positioning to the process level.

[0064] The communication group identifier (i.e., process_group_id) is an identifier used to distinguish multiple parallel communication groups that may exist in distributed training. For example, in a hybrid parallel strategy, data parallelism and model parallelism may use different communication groups. This field enables the center node to correctly associate the metadata of each rank belonging to the same communication operation and is a prerequisite for correct hover detection.

[0065] The list of processes participating in the communication (i.e., ranks_in_group) is a list containing multiple rank_ids, explicitly listing all members participating in this communication operation. For example, when performing hover detection to determine whether a process has started, the center node needs to refer to this list to determine whether all expected participants have reported the start time.

[0066] The shape of the communication tensor (i.e., tensor_shape) and the data type (i.e., tensor_dtype) record the dimension information (such as [4096, 4096]) of the data tensor transmitted in this communication and the data type (such as float16) of its elements, respectively. These two fields are mainly used for more detailed performance analysis and timeout detection. The time consumption of communication is closely related to the amount of data transmitted (determined by shape and data type). For example, when setting the baseline duration, more accurate matching can be performed based on these two fields, thereby improving the accuracy of timeout judgment and avoiding false positives due to different data volumes. At the same time, these information also helps to analyze the communication pattern and load bottleneck in the entire training process.

[0067] The end time (i.e., end_timestamp) is a high-precision timestamp that records the exact time when the communication operation is completed on the training node. For asynchronous operations, this is usually the time when the associated wait() method returns. In addition, the end time is the endpoint for calculating the execution duration and is also a key indicator for determining whether a process is in the hover state.

[0068] The operation result (i.e., status) is a field indicating the final state of the operation, which can take values such as Success, Timeout, Pending, etc. (filled in by the center node after analysis). This field is used to record the final result of the operation. The training node side may only record whether it is successful, while the center node can backfill or associate the timeout or hover state to this record after analysis, facilitating subsequent queries and statistics.

[0069] It can be understood that, different from the coarse-grained information acquisition of the general distributed tracking or the performance analysis tool of the framework, the embodiment of the application directly hooks the torch.distributed and other underlying communication API (Application Programming Interface, application programming interface) layer, so that the training node can capture extremely fine-grained and communication-intensive metadata, thereby ensuring that the analysis engine of the center node can perform in-depth and accurate fault diagnosis according to the data.

[0070] Based on the above embodiment, in step 220, the detection of whether the communication operation starts or ends within the preset time length is to determine whether the communication operation is in the hovering state, comprising: For any communication operation, if the start time is not detected in the communication metadata of at least one training node participating in the any communication operation within the preset time length, it is determined that the any communication operation is in the hovering state.

[0071] In addition, in step 220, the detection of whether the communication operation starts or ends within the preset time length is to determine whether the communication operation is in the hovering state, further comprising: For any communication operation, if the start time exists in the communication metadata of all training nodes participating in the any communication operation, but the end time is not detected in the communication metadata of at least one training node participating in the any communication operation within the preset time length, it is determined that the any communication operation is in the hovering state.

[0072] It should be noted that the embodiment of the application is a further refinement and description of the detection by the center node of whether the communication operation is in the hovering state. The real-time analysis engine of the center node implements two parallel detection logics, which can comprehensively cover the main scenarios leading to communication hovering, thereby improving the accuracy and timeliness of the early warning.

[0073] Specifically, when the center node performs hovering detection, the two detection logics correspond to the following two cases respectively: The first case is the hang caused by partial participants not starting. The detection logic of this case is mainly used to find the waiting of the whole communication group caused by individual processes "falling behind" (i.e., partial processes are late to enter the communication operation). The specific implementation principle is as follows: when the real-time analysis engine of the center node first receives the metadata about a specific communication operation (for example, all reduce operation) from the data stream, it obtains the rank id set of all ranks that should participate in this communication from the "communication participant process list (i.e., ranks_in_group)" field in the metadata. At the same time, the engine starts a timer for this communication operation, and the duration of the timer is the preset duration. This duration is a configurable threshold, for example, 100 ms. Its setting needs to be balanced, too short may cause false judgment due to normal network jitter or uneven computing load, and too long will reduce the sensitivity of the early warning. During the running of the timer, the engine continuously monitors the subsequent arriving metadata to check whether all the ranks in the participant set of the communication operation have reported their start times (start_timestamp). If the timer expires (i.e., 100 ms has passed), the engine finds that there is at least one metadata corresponding to a rank id in the participant set that has not arrived or has arrived but has no valid start time, and the engine determines that the communication operation has entered the hang state.

[0074] Exemplarily, assume that an all reduce operation involves a communication group containing 8 processes (i.e., rank0 to rank7). The real-time analysis engine of the center node receives the metadata of the operation from rank0 at T0, learns that the participants are {0, 1, 2, 3, 4, 5, 6, 7}, and starts a 100 ms timer. At T0+100 ms, the timer expires, and the engine checks that it has received metadata with start_timestamp from rank0 to rank6, but has not received metadata from rank7. At this time, the engine will determine that the all reduce operation is in the hang state, because rank7 fails to enter the operation on time, causing the other 7 processes to wait indefinitely.

[0075] The second case is the hang caused by all participants having started but not ended. The detection logic of this case is mainly used to find the deadlock or node crash occurring in the communication execution process. The specific implementation principle is as follows: first, the engine must have confirmed (through the detection of the first case described above) that all participants of a communication operation have reported a valid start time, which indicates that the communication operation has been successfully started. After confirming the start of all, the engine starts a new timer for the operation to monitor whether it can be completed within the expected preset time length. This time length is usually longer than the time length used to detect the "start" (i.e. the preset time length in the first case described above), for example, 500 ms or 1 second, because it needs to cover the normal communication execution time. During the running of the timer, the engine continuously monitors to check whether all rank_ids participating in the operation have reported their respective end times (end_timestamp). If the timer expires and the engine finds that at least one rank_id has not reported its end time, the engine determines that the communication operation has entered the hang state.

[0076] Exemplarily, still taking the all_reduce operation involving rank0 to rank7 described above, the engine has confirmed that it has received the start_timestamp of all 8 processes at T0+50 ms, and has started an end timer of 500 ms. At T0+50 ms+500 ms, the timer expires, and the engine checks and finds that it has received the end_timestamp of all 7 processes except rank4. At this time, the engine determines that the operation is in the hang state. This indicates that all processes have entered the communication, but during the execution process, the node where rank4 is located may have a hardware failure or process crash, causing it to fail to complete the operation and report the end time, thereby causing the entire communication group to be stuck.

[0077] In the embodiments of the application, through the combined detection of the above two cases, two different types of hang failures can be accurately distinguished. The first case can quickly locate the "straggler" that fails to enter the communication, and the problem usually lies in the process itself or the connection between the process and the center node. The second case indicates that the failure occurs after all participants enter the communication, and the problem may be in the interaction between processes or the unexpected exit of a process. This fine distinction provides a more explicit troubleshooting direction for the operation and maintenance personnel, significantly improving the efficiency and accuracy of fault diagnosis.

[0078] Based on any of the above embodiments, step 230 specifically comprises: Step 231, for any communication operation in the hang state, among all training nodes participating in the any communication operation, determining the training node that has not detected the start time or the end time within the preset time length as the fault node. Step 232, based on the communication metadata associated with the any communication operation sent by the fault node, generate a pre-warning information and trigger an alarm.

[0079] It should be noted that when the center node determines that a communication operation is in a suspended state according to the steps of the above embodiments, it can further accurately determine the fault node that causes the suspension and generate detailed pre-warning information.

[0080] Specifically, for a certain communication operation detected in a suspended state, in determining the fault node, it can be divided into the following two positioning logics according to the detected suspension type: The first is the fault source positioning for "part of the participants not started" leading to suspension. When a communication operation is determined to be suspended because part of the process has not reported the start time within the preset time, the determination of the fault node is highly accurate and direct. In this scenario, the real-time analysis engine of the center node has identified which rank_id or which rank_id has not reported start_timestamp. The training node corresponding to these "absent" rank_id is directly determined as the fault node of this communication failure.

[0081] Taking the all_reduce operation participated by rank0 to rank7 in the above embodiment as an example, the all_reduce operation triggers a suspension alarm because rank7 has not reported the start time within 100ms. The center node will immediately perform the following operations: first, determine rank7 as the fault source process and query the relevant node information. Here, the system can maintain a mapping relationship from rank_id to its physical location (this mapping relationship can be established at task initialization and stored in the database of the center node). By querying this mapping, the system can obtain the training node IP address (i.e. node_ip) and the physical gpu_id on the node corresponding to rank7. Then, the system will generate a structured pre-warning information, which can include the alarm level (such as serious), the fault type (such as communication suspension-process not started), the occurrence time (i.e. the exact timestamp when the suspension is detected), the suspension duration, the fault source positioning (including node_ip, gpu_id, rank_id) and the fault context, etc. Here, the fault context can include the operation type (such as all_reduce), the communication group identifier (process_group_id), the participant state (such as rank0~rank6 are waiting, rank7 has not entered), etc. Finally, the above pre-warning information is sent to the operation and maintenance team through the alarm system.

[0082] The second is to locate the fault source for the "started but not ended" that causes the hang. When all processes have started, but the communication operation fails to complete within the preset time length, the location of the fault source also has clear logic. In this case, all processes that report the start time but fail to report the end time within the preset time length are determined as the fault nodes (or highly suspected nodes) of the current fault. Those nodes that have successfully reported the end time (if any) can be excluded.

[0083] For example, in the above examples, an all_reduce operation is triggered to hang after all processes start because rank4 fails to report the end time within 500 ms. The processing flow of the center node is as follows: first, rank4 is determined as the fault source process. Similarly, the mapping relationship is queried to obtain the node_ip and gpu_id corresponding to rank4. Then, the warning information is generated, which clearly indicates that the fault occurs in the communication execution phase. Finally, the generated warning information is sent to the operation and maintenance team.

[0084] The method provided by the embodiment of the application converts the fuzzy system stuck problem into specific and operable positioning information. By clearly distinguishing between the two hanging states of not started and not ended and locating the corresponding fault nodes, precise clues are provided for the operation and maintenance personnel. The generated warning information is rich in context, so that the operation and maintenance personnel can quickly focus on the problem source (specific server, specific GPU) without logging into a large number of machines to check the logs, thereby greatly shortening the fault locating and recovery time.

[0085] Based on any of the above embodiments, step 220 specifically includes: Based on the start time and the end time in the communication metadata of the completed communication operation, determining an execution time length of the completed communication operation; In the case where the execution time length exceeds a baseline time length, determining that the completed communication operation has a timeout exception, the baseline time length being pre-set or dynamically learned.

[0086] It should be noted that in addition to being able to detect the hanging state, the center node can also identify another common abnormality that affects training efficiency, that is, the timeout exception, that is, the communication operation is eventually successfully completed, but its time consumption far exceeds the normal level. This exception is a key bottleneck that affects the overall throughput of large-scale training.

[0087] Specifically, in step 220, when the execution state of the communication operation is analyzed in real time according to the communication metadata, in addition to the detection of the hanging state, the following logic for detecting the timeout exception is also included: First, determine the duration of the completed communication operation. This step is only for the communication operation that has been successfully completed. A communication operation is considered completed if the real-time analysis engine of the center node has received the complete communication metadata reported by a certain rank_id for the operation, which contains the valid start time (start_timestamp) and end time (end_timestamp). For each completed communication operation, the analysis engine calculates its execution duration (duration) in the following way: duration = end_timestamp - start_timestamp. For example, if the start_timestamp of a certain all_gather operation reported by rank5 is T0 and the end_timestamp is T0+20ms, the execution duration of the operation is determined to be 20ms.

[0088] Second, compare with the baseline duration to determine whether there is a timeout exception. After calculating the execution duration, the analysis engine compares it with a predefined baseline duration (baseline_duration). The baseline duration represents the expected completion time of the same type of communication operation under healthy conditions, which can be regarded as a kind of service level agreement (SLA). If the execution duration exceeds the baseline duration, the analysis engine determines that the completed communication operation has a timeout exception.

[0089] It can be understood that before performing the timeout exception detection, the baseline duration needs to be determined, which can be obtained by pre-setting or by dynamic learning, and the present embodiment does not make specific limitation thereto.

[0090] In an embodiment, the baseline duration can be obtained by pre-setting (i.e. static SLA). In this way, the baseline duration is a static value determined according to offline benchmarking. Specifically, before the formal training task starts, performance tests can be performed on communication operations of different operation types (such as all_reduce and all_gather) and different communication data volumes (i.e. tensor_shape) according to the current cluster hardware topology and network environment, to obtain the average time consumption under ideal conditions. These test results are configured as a lookup table or rule base and stored in the center node.

[0091] Exemplarily, it is known through testing that in the current 10000-core cluster, the normal baseline duration of an all_reduce operation for a 1GB tensor should be 150ms. Then, when the real-time analysis engine processes an all_reduce operation of the same specification, if the actual execution duration is calculated to be 350ms, since 350ms>150ms, the system will determine it as a timeout exception.

[0092] In another embodiment, to better adapt to the changes of load and different stages of communication patterns in the training process, the baseline duration can also be dynamically learned and adjusted by the system during operation. Specifically, the real-time analysis engine continuously calculates the statistical value of the execution duration of a specific communication operation (with the same operation type, tensor shape, communication group size, etc.) within a recent time window (e.g., the past 10 minutes), such as the moving average or the 95th percentile value (p95), and this dynamically calculated statistical value is used as the current baseline duration.

[0093] For example, the system continuously monitors a broadcast operation, and within the past 5 minutes, the execution duration of the operation has stabilized between 8ms and 10ms, with a p95 value of 9.8ms. The system takes 9.8ms as the current dynamic baseline duration. At this time, if a sudden execution duration of 25ms of the operation occurs, since 25ms is much greater than 9.8ms, the system will also determine it as a timeout exception. The benefit of dynamic learning is that it can automatically adapt to normal performance fluctuations and reduce false positives caused by unreasonable baseline settings.

[0094] In addition, after determining the timeout exception, the center node also determines the faulty node that causes the timeout exception and generates a warning information, and triggers an alarm. The warning information clearly indicates which rank or node is slow, the communication operation type, the actual time consumption and the expected time consumption, etc., guiding the operation and maintenance personnel to focus on the health status of the node.

[0095] The method provided by the embodiment of the application can timely discover and alarm the nodes with timeout exceptions by introducing timeout exception detection, which helps the operation and maintenance personnel to actively intervene and troubleshoot potential problems (such as network congestion, GPU overheating and frequency reduction, node resource contention, etc.), thereby optimizing the training performance and improving the utilization efficiency of computing resources.

[0096] Based on any of the above embodiments, the method further comprises: displaying, through a visual interface, the life cycle, delay distribution, and abnormal trend of the communication operation with the exception.

[0097] Specifically, to further enhance the user experience and troubleshooting efficiency of traffic fault warning and positioning in the distributed training scenario, the center node also provides an interactive visual interface that presents the abnormal data analyzed by the center node in a intuitive graphical manner to the operation and maintenance personnel or algorithm engineers, thereby converting complex and massive communication data into easy-to-understand and operable insights.

[0098] Specifically, when displaying, through the visual interface, the life cycle, delay distribution, and abnormal trend of the communication operation with the exception, the following visual forms are included but not limited to: (1) Visualized presentation of communication operation life cycle (e.g., timeline chart). To visually present the execution process of a single or multiple communication operations, especially the life cycle of a communication operation with abnormality, embodiments of the present application provide a timeline chart, similar to a Gantt chart or Trace view in a distributed tracing system. In this view, the Y-axis usually represents different ranks (rank_id), and the X-axis represents time. A normal communication operation can be rendered as a colored rectangular block with a clear start and end point, where the left boundary corresponds to the start time (start_timestamp), the right boundary corresponds to the end time (end_timestamp), and the length of the block represents the execution duration.

[0099] For abnormal communication operations, they can be presented according to different abnormal types. For example, for the hang-up abnormality caused by the fact that some ranks do not enter the communication operation, at a certain time point in the view, most ranks in the communication group start a new operation (i.e., a new block appears), while the rank determined as the fault source is a blank at the corresponding Y-axis position, forming an obvious "gap" and directly exposing the fault node. For another example, in the view, the block corresponding to the rank determined as timeout is significantly longer than the blocks of other ranks in the communication group, so that the rank or node with timeout abnormality is immediately apparent. In addition, users can also view detailed communication metadata of any block by hovering the mouse over it, such as operation type, tensor size, and precise time consumption, etc.

[0100] (2) Visualized presentation of communication delay distribution (e.g., heat map, histogram). To grasp the communication performance health of the entire cluster from a macro perspective and identify performance bottlenecks, the interface can provide a view of delay distribution. Taking a heat map as an example, a heat map can be generated with ranks as the Y-axis and time windows as the X-axis. The color depth of each grid in the figure represents the average delay of the communication operation of the rank in the time window. Through the color gradient (e.g., green represents low delay and red represents high delay), the operation and maintenance personnel can quickly find out which ranks are consistently underperforming or in which time period the communication pressure of the entire cluster is very large.

[0101] (3) Visualized presentation of abnormality trend (e.g., trend chart). To monitor and predict the long-term stability of the system, the interface can also provide a view of abnormality trend. The trend chart is usually a line chart with time as the X-axis, and the Y-axis can be various key performance or abnormality indicators, such as the number of hang-up events per minute and the number of timeout events per minute, etc. By observing the trend lines of these indicators, the operation and maintenance personnel can judge whether the health condition of the system is improving or deteriorating.

[0102] The visual interface provided by the embodiment of the application is a key bridge connecting automatic anomaly detection and efficient manual decision. It converts original and abstract alarm information into user-friendly graphical language, greatly reducing the cognitive burden of understanding and locating faults. Through comprehensive display of life cycle, delay distribution and abnormal trends, and supplemented by clicking, drilling and other interactive capabilities, the operation and maintenance personnel can go from macro trends to micro details, realize "seeing, seeing clearly and understanding" of communication faults, and thus improve the fault troubleshooting efficiency by an order of magnitude, which is an important support for ensuring the stability of super large-scale distributed training.

[0103] Based on any of the above embodiments, Figure 3 is a second flowchart of the communication fault early warning and positioning method provided by the application, as Figure 3 shown, the method can be applied to a training node in a communication fault early warning and positioning system, and the method comprises the following steps: Step 310, intercepting the communication interface of the distributed training framework to capture each communication operation, and collecting communication metadata associated with the communication operation during the execution of the communication operation; Step 320, sending the communication metadata of each communication operation to the center node, so that the center node analyzes the execution state of each communication operation based on the communication metadata of each communication operation, and determines whether there is an abnormality according to the analysis result; Wherein, the real-time analysis at least includes detecting whether the each communication operation starts or ends within a preset time length to determine whether the each communication operation is in a hovering state; the center node is used to determine a fault node and generate an early warning information based on the communication metadata of the any communication operation in case that the any communication operation is determined to be abnormal.

[0104] Specifically, for each training node in the system, it can be transparent and fine-grained intercepted (i.e. Hook) through monkey patching or decorator technology in the core communication API layer (such as torch.distributed of PyTorch) of the distributed training framework, to accurately capture the life cycle and key metadata of each communication operation (including synchronous and asynchronous operations).

[0105] Before each Hook function is executed, the training node records metadata such as the operation type, start time, communication process identifier, communication group identifier, process list participating in the communication, shape and data type of the communication tensor, and the like. After the communication operation is executed (especially after the wait() method of the Work object of the abnormal operation is completed), the training node records the end time and operation result. Subsequently, the collected communication metadata can be temporarily stored in the local cache, and then uploaded in batches to the central node in an asynchronous manner by a separate background thread or process.

[0106] The data collection module on the central node can provide a data collection service, which is mainly responsible for receiving batch communication metadata uploaded from all training nodes, and decompressing, deserializing, and preliminary checking the data. It should be understood that, in order to implement the data collection service, the data collection module has high concurrent processing capability and can be horizontally expanded to cope with the massive data flow of a million-node cluster.

[0107] After the central node completes the verification of the received communication metadata, it can store the data in the local database, so that the subsequent analysis engine can read the data from the database for real-time analysis. Here, the central node can use a stream processing framework (such as Apache Flink) or a customized background service to build a real-time analysis engine that can continuously query the communication metadata in the database to analyze the execution status of the communication operation (such as the Pending state, timeout exception, etc.). When a communication operation is detected to be in the Pending state or to have a timeout exception, the related communication metadata of the communication operation can be analyzed to accurately infer the most likely faulty node (including node_ip, gpu_id, rank_id, etc.).

[0108] Once the faulty node is determined, the central node immediately generates an early warning information. The information contains detailed context, such as the fault type, occurrence time, hovering duration, affected communication operation type, communication group identifier, and most importantly, the node_ip, gpu_id, and rank_id of the faulty node. Subsequently, the early warning information can be sent to an alarm manager (such as Alertmanager) through a standardized interface to notify the operation and maintenance personnel by email, SMS, or instant message, etc.

[0109] In addition, the central node can also provide intuitive timeline charts, heat maps, trend charts, and the like to display the life cycle of the communication operation, delay distribution, and abnormal trends, and support drilling queries of fault events to assist operation and maintenance personnel to quickly locate and troubleshoot problems.

[0110] The method provided by the embodiment of the application can capture communication metadata associated with communication operations that cannot be reached by a traditional solution by precisely intercepting at the communication interface level of the distributed training framework, thereby laying a foundation for accurate analysis. After receiving the communication metadata associated with each communication operation sent by each training node, the center node can perform real-time analysis according to the metadata to detect the communication suspension state, so that the communication operation in the suspended state of not being initiated or being initiated but not completed can be captured in real time before the fault actually occurs, thereby realizing early warning. In addition, when an abnormal communication operation is detected, the communication metadata can be associated and analyzed to accurately locate the source of the fault and generate warning information containing detailed abnormal conditions, thereby greatly shortening the fault troubleshooting time and ensuring the stability and efficiency of the large-scale training task.

[0111] According to any of the above embodiments, in step 320, the communication metadata of each communication operation is sent to the center node, including: In step 321, the communication metadata of each communication operation is temporarily stored in the local cache. In step 322, the communication metadata in the local cache is uploaded to the center node asynchronously in batches.

[0112] Specifically, after intercepting and collecting the metadata of a communication operation, the first action performed by the intercepting code (i.e., the Hook function) on the training node is not to send it out, but to temporarily store the communication metadata of the communication operation in the local cache, thereby minimizing the local data collection overhead. Here, the local cache refers to an area opened in the memory of the training node, which enables the metadata collection operation to be completed at an extremely fast speed, effectively decoupling the collection action from the subsequent network sending action, and avoiding blocking of the training main thread by any network delay or jitter.

[0113] It can be understood that, in order to achieve high performance and controllable memory occupation, the local cache can be implemented by using a wake-up buffer or a non-blocking queue in the memory. The ring buffer is a fixed-size buffer, and its advantage lies in controllable memory consumption and high-performance writing. The ring buffer can realize lock-free or extremely light-weight concurrent access, thereby providing extremely high write throughput and extremely low delay for data collection. The non-blocking queue is a queue that implements the producer-consumer model. The training main thread acts as a "producer" to quickly put the metadata into the queue, and an independent background thread acts as a "consumer" to take the data out of the queue. The non-blocking feature ensures that when the queue is full or empty, the production or consumption operation will not hang up the thread, but can return immediately, thereby avoiding blocking.

[0114] After the communication metadata is safely and quickly stored in the local cache, a separate background task is responsible for uploading the communication metadata in the local cache to the center node asynchronously in batches. Here, asynchronous means that the uploading task is executed by a separate background thread or process, completely independent of the training main thread. The training main thread only produces data and does not need to care about when the data is uploaded, whether the upload is successful, thereby achieving complete isolation.

[0115] In addition, the background uploading task does not take data from the cache one by one and send it, but adopts a batch uploading mode. For example, it can be set to read a large block (batch) of data from the cache at a time periodically (such as every 500 milliseconds) or when the amount of data in the cache reaches a preset threshold (such as 80% of the cache usage). Then, it serializes (such as using Protocol Buffers) and compresses the batch of data, and sends it to the center node through a single network request (such as a gRPC call). This batch processing mode greatly reduces the overhead of establishing a network connection and the redundancy of the data packet header, significantly improving the network transmission efficiency.

[0116] The method provided by the embodiment of the application greatly reduces the performance intrusion of tracking on the training main process for the massive communication events of the Wan-Ka cluster by using local caching and independent asynchronous batch uploading. First, by using local caching, it minimizes the delay of data acquisition and almost does not cause perceptible intrusion on the performance of the training main process. Second, by using independent background threads for asynchronous batch uploading, it removes the overhead and uncertainty of network transmission from the critical training path, ensuring the stability of training. In addition, the Hook layer only records light metadata and writes to memory, and data transmission is performed through efficient serialization, compression and batch sending, ensuring that the system can still run stably under a super large-scale cluster and the overhead is controllable.

[0117] The communication fault early warning and positioning device provided by the application will be described below. The communication fault early warning and positioning device described below can be correspondingly referred to the communication fault early warning and positioning method described above.

[0118] Based on any of the above embodiments, Figure 4 is one of the structural diagrams of the communication fault early warning and positioning device provided by the application, as Figure 4 shown, the device is applied to a center node, and the device comprises: The data receiving unit 410 is configured to receive communication metadata associated with each communication operation sent by at least one training node, wherein the communication metadata is obtained by intercepting the communication interface of the distributed training framework by the at least one training node. The abnormality analysis unit 420 is configured to analyze the execution state of each communication operation in real time based on the communication metadata, and determine whether the communication operation is abnormal according to the analysis result, wherein the real-time analysis at least includes detecting whether the communication operation starts or ends within a preset time length to determine whether the communication operation is in a suspended state. The positioning and early warning unit 430 is configured to, when determining that any communication operation is abnormal, determine a fault node based on the communication metadata of the any communication operation and generate early warning information.

[0119] The device provided by the embodiment of the application can capture communication metadata associated with a communication operation that cannot be captured by a traditional scheme by precisely intercepting at the communication interface level of a distributed training framework of each training node, thereby laying a foundation for accurate analysis. After receiving the communication metadata associated with each communication operation sent by each training node, the central node can analyze the metadata in real time to detect a communication suspended state, thereby capturing that a communication operation is in a suspended state of not being initiated or being initiated but not completed in real time before a fault actually occurs, so that early warning can be realized. In addition, when detecting that a communication operation is abnormal, the communication metadata can be associated and analyzed to accurately locate a source of the fault and generate early warning information containing detailed abnormal conditions, thereby greatly shortening fault troubleshooting time and ensuring stability and efficiency of a large-scale training task.

[0120] Based on any of the above embodiments, the abnormality analysis unit 420 is specifically configured to: For any communication operation, if a start time is not detected in the communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

[0121] Based on any of the above embodiments, the abnormality analysis unit 420 is specifically configured to: For any communication operation, if a start time exists in the communication metadata of all training nodes participating in the any communication operation, but an end time is not detected in the communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

[0122] Based on any of the above embodiments, the positioning and early warning unit 430 is specifically configured to: For any communication operation in a suspended state, a training node in which a start time or an end time is not detected within a preset time length is determined as the fault node among all training nodes participating in the any communication operation. generate early warning information and trigger an alarm based on the communication metadata associated with the any communication operation sent by the faulty node.

[0123] Based on any of the above embodiments, the anomaly analysis unit 420 is specifically configured to: determine the execution duration of the completed communication operation based on the start time and the end time in the communication metadata of the completed communication operation; determine that the completed communication operation has a timeout anomaly if the execution duration exceeds a baseline duration, the baseline duration being pre-configured or dynamically learned.

[0124] Based on any of the above embodiments, the device further comprises a visual display unit, configured to: display the life cycle, delay distribution and anomaly trend of the communication operation with an anomaly through a visual interface.

[0125] Based on any of the above embodiments, the communication metadata comprises at least one of operation type, start time, communication process identifier, communication group identifier, process list participating in communication, shape and data type of communication tensor, end time, and operation result.

[0126] Based on any of the above embodiments, Figure 5 is a structural diagram of a communication fault early warning and positioning device provided by the present application, as shown in Figure 5 The device is applied to a training node, and the device comprises: An interception and capture unit 510 is configured to perform an interception operation on a communication interface of a distributed training framework to capture each communication operation and collect communication metadata associated with the communication operation during execution of the communication operation; A data sending unit 520 is configured to send the communication metadata of each communication operation to a center node, so that the center node performs real-time analysis on the execution state of each communication operation based on the communication metadata of each communication operation, and determines whether each communication operation has an anomaly according to an analysis result; The real-time analysis at least includes detecting whether each communication operation starts or ends within a preset duration to determine whether each communication operation is in a hovering state; and the center node is configured to determine a faulty node and generate early warning information based on the communication metadata of any communication operation if it is determined that any communication operation has an anomaly.

[0127] The device provided by the embodiment of the present application can capture communication metadata associated with communication operations that cannot be reached by a traditional solution by precisely intercepting at the communication interface level of the distributed training framework of each training node, thereby laying a foundation for accurate analysis. After receiving the communication metadata associated with each communication operation sent by each training node, the center node can perform real-time analysis based on the metadata to detect the communication suspension state, thereby capturing that a communication operation is in a suspension state of not being initiated or having been initiated but not completed in real time before the fault actually occurs, so that early warning is realized. In addition, when an abnormal communication operation is detected, the communication metadata can be associated and analyzed to accurately locate the source of the fault and generate warning information containing detailed abnormal conditions, thereby greatly shortening the fault troubleshooting time and ensuring the stability and efficiency of large-scale training tasks.

[0128] Based on any of the above embodiments, the data sending unit 520 is specifically configured to: temporarily store the communication metadata of the communication operations in a local cache; asynchronously and in batches upload the communication metadata in the local cache to the center node.

[0129] Based on any of the above embodiments, the local cache is a ring buffer or a non-blocking queue.

[0130] Figure 6 is a structural diagram of the center node provided by the present application, as Figure 6 shown, the center node can include a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620, the memory 630 communicate with each other through the communication bus 640. The processor 610 can invoke the logical instructions in the memory 630 to execute the communication fault warning and positioning method, which includes: receiving communication metadata associated with each communication operation sent by at least one training node, the communication metadata being collected by the at least one training node after intercepting the communication interface of the distributed training framework; based on the communication metadata, performing real-time analysis on the execution state of the communication operations, and determining whether the communication operations are abnormal according to the analysis result, wherein the real-time analysis at least includes detecting whether the communication operations start or end within a preset time length to determine whether the communication operations are in a suspension state; in the case where it is determined that any communication operation is abnormal, determining a fault node based on the communication metadata of the any communication operation and generating warning information.

[0131] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] Figure 7 This is a schematic diagram of the training node structure provided by the present invention, as shown below. Figure 7 As shown, the training node may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communication interface 720, and memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a communication fault warning and location method. This method includes: intercepting the communication interface of the distributed training framework to capture each communication operation, and collecting communication metadata associated with the communication operation during its execution; sending the communication metadata of each communication operation to a central node, so that the central node can perform real-time analysis of the execution status of each communication operation based on the communication metadata, and determine whether each communication operation is abnormal based on the analysis results; wherein the real-time analysis includes at least detecting whether each communication operation starts or ends within a preset time period to determine whether each communication operation is in a hovering state; the central node is used to determine the fault node and generate warning information based on the communication metadata of any communication operation if any communication operation is determined to be abnormal.

[0133] In addition, the logic instructions in the memory 730 described above can be implemented in the form of software function units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or contribute to the related art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0134] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the communication fault early warning and positioning method provided by the above-mentioned methods. The method is applied to a center node, and the method comprises the following steps: receiving communication metadata associated with each communication operation sent by at least one training node, the communication metadata being obtained by intercepting the communication interface of a distributed training framework by the at least one training node; based on the communication metadata, analyzing the execution state of the each communication operation in real time, and determining whether the each communication operation is abnormal according to the analysis result, wherein the real-time analysis at least includes detecting whether the each communication operation starts or ends within a preset time length, so as to judge whether the each communication operation is in a hovering state; in the case that it is determined that any communication operation is abnormal, determining a fault node and generating early warning information based on the communication metadata of the any communication operation.

[0135] In addition, the computer program is executed by the processor, and the computer can further execute the communication fault early warning and positioning method provided by each method. The method is applied to a training node, and the method comprises: intercepting a communication interface of a distributed training framework to capture each communication operation, and collecting communication metadata associated with the communication operation during execution of the communication operation; sending the communication metadata of each communication operation to a center node, so that the center node analyzes the execution state of each communication operation based on the communication metadata of each communication operation, and determines whether each communication operation is abnormal according to an analysis result; wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a hovering state; and the center node is configured to determine a fault node and generate early warning information based on the communication metadata of any communication operation in a case where it is determined that the any communication operation is abnormal.

[0136] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the communication fault early warning and positioning method provided by each method. The method is applied to a center node, and the method comprises: receiving communication metadata associated with each communication operation sent by at least one training node, wherein the communication metadata is collected by the at least one training node after intercepting a communication interface of a distributed training framework; based on the communication metadata, analyzing the execution state of each communication operation in real time, and determining whether each communication operation is abnormal according to an analysis result, wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a hovering state; and in a case where it is determined that any communication operation is abnormal, determining a fault node and generating early warning information based on the communication metadata of the any communication operation.

[0137] In addition, the computer program is executed by the processor to implement the communication fault early warning and positioning method provided by the above method, the method is applied to a training node, and the method includes: intercepting a communication interface of a distributed training framework to capture each communication operation and collecting communication metadata associated with the communication operation during execution of the communication operation; and sending the communication metadata of each communication operation to a center node to enable the center node to analyze an execution state of each communication operation based on the communication metadata of each communication operation, and determine whether each communication operation is abnormal according to an analysis result; wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a hovering state; and the center node is configured to determine a fault node and generate early warning information based on the communication metadata of any communication operation in a case where it is determined that the any communication operation is abnormal.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0139] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiment.

[0140] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A communication fault early warning and locating method, characterized in that, The method is applied to a center node, and the method comprises: Receiving communication metadata associated with each communication operation sent by at least one training node, the communication metadata being collected after the at least one training node intercepts a communication interface of a distributed training framework; Based on the communication metadata, the execution state of each communication operation is analyzed in real time, and it is determined whether there is an abnormality in each communication operation according to the analysis result, wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a suspended state; In the case where it is determined that any communication operation is abnormal, based on the communication metadata of the any communication operation, a fault node is determined and a warning information is generated.

2. The communication fault pre-warning and locating method according to claim 1, characterized in that, The detection of whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a suspended state comprises: For any communication operation, if no start time is detected in the communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

3. The communication fault pre-warning and locating method according to claim 1, characterized in that, The detection of whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a suspended state comprises: For any communication operation, if there is a start time in the communication metadata of all training nodes participating in the any communication operation, but no end time is detected in the communication metadata of at least one training node participating in the any communication operation within a preset time length, it is determined that the any communication operation is in a suspended state.

4. The communication fault pre-warning and locating method according to claim 1, characterized in that, The determination of a fault node based on the communication metadata of the any communication operation and the generation of a warning information comprises: For any communication operation in a suspended state, among all training nodes participating in the any communication operation, the training node that does not detect a start time or an end time within a preset time length is determined as the fault node; Based on the communication metadata associated with the any communication operation sent by the fault node, a warning information is generated and an alarm is triggered.

5. The communication fault pre-warning and locating method according to claim 1, characterized in that, The real-time analysis of the execution state of each communication operation based on the communication metadata and the determination of whether there is an abnormality in each communication operation according to the analysis result comprises: Based on the start time and the end time in the communication metadata of a completed communication operation, the execution time length of the completed communication operation is determined; In the case where the execution time length exceeds a baseline time length, it is determined that the completed communication operation has a timeout abnormality, and the baseline time length is pre-set or dynamically learned.

6. The communication fault pre-warning and locating method according to any one of claims 1 to 5, characterized in that, Further comprising: Through a visual interface, the life cycle, delay distribution and abnormality trend of the communication operation with an abnormality are displayed.

7. The communication fault pre-warning and locating method according to any one of claims 1 to 5, characterized in that, The communication metadata comprises at least one of an operation type, a start time, a communication process identifier, a communication group identifier, a process list participating in communication, a shape and data type of a communication tensor, an end time, and an operation result.

8. A communication fault early warning and locating method, characterized in that, The method is applied to a training node, and the method comprises: intercepting a communication interface of a distributed training framework to capture each communication operation and collect communication metadata associated with the communication operation during execution of the communication operation; sending the communication metadata of each communication operation to a central node to enable the central node to perform real-time analysis on execution status of each communication operation based on the communication metadata of each communication operation and determine whether each communication operation is abnormal according to an analysis result; wherein the real-time analysis at least includes detecting whether each communication operation starts or ends within a preset time length to determine whether each communication operation is in a hovering state; and the central node is configured to determine a faulty node and generate a warning information based on the communication metadata of any communication operation in a case where it is determined that the any communication operation is abnormal.

9. The communication fault pre-warning and locating method according to claim 8, characterized in that, The sending of the communication metadata of each communication operation to the central node includes: temporarily storing the communication metadata of each communication operation in a local cache; asynchronously and in batches uploading the communication metadata in the local cache to the central node.

10. The communication fault pre-warning and locating method according to claim 9, characterized in that, The local cache is a ring buffer or a non-blocking queue.

11. A central node comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the communication fault warning and positioning method according to any one of claims 1 to 7 when executing the computer program.

12. A training node comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein, The processor implements the communication fault warning and positioning method according to any one of claims 8 to 10 when executing the computer program.

13. A communication fault early warning and locating system, characterized by The system includes the central node according to claim 11 and at least one training node according to claim 12.

14. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the communication fault warning and positioning method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the communication fault warning and positioning method according to any one of claims 1 to 10.

Images