Fault diagnosis method and device, intelligent agent and product

Abstract

The invention provides a fault diagnosis method and device, an intelligent agent, electronic equipment, a storage medium and a computer program product, relates to the technical field of computers, in particular to the technical fields of cloud computing, distribution, virtualization, artificial intelligence and the like, and can be applied to a diagnosis scene of a communication performance fault of a distributed system. According to the specific implementation scheme, in the task running process of the distributed system, a logic layer log representing state information of coordination operation of a logic layer and a physical layer log representing state information of a communication link of a physical layer are obtained; determining whether a communication performance fault exists in the task running process or not according to the logic layer log and the physical layer log through an artificial intelligence large model; and in response to existence, performing attribution analysis according to the logic layer log and the physical layer log through the depth tracking component, and determining a diagnosis result. According to the method, based on logs of different levels, fault existence judgment of a large model and attribution analysis of a deep tracking component are combined, and the diagnosis efficiency and accuracy are improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of cloud computing, distributed systems, virtualization, and artificial intelligence, and particularly to a fault diagnosis method, device, intelligent agent, electronic device, storage medium, and computer program product, which can be applied to the diagnosis of communication performance faults in distributed systems. Background Technology

[0002] In cloud-native environments, multi-machine distributed deep learning training tasks are popular due to their automated deployment, flexible operator scheduling strategies, and efficient utilization of hardware resources such as GPUs (Graphics Processing Units), CPUs (Central Processing Units), and RDMA (Remote Direct Memory Access). However, the convenient automation and hardware logic abstraction capabilities can actually increase the cost of troubleshooting and diagnosis when communication performance issues such as training hangs or performance slowdowns occur due to communication congestion or hardware failures. Currently, diagnosis of communication performance issues mainly relies on single-machine monitoring or manual methods. Summary of the Invention

[0003] This disclosure provides a fault diagnosis method, apparatus, intelligent agent, electronic device, storage medium, and computer program product.

[0004] According to the first aspect, a fault diagnosis method is provided, including: during the task execution process of a distributed system, acquiring logical layer logs representing the status information of the coordination operation of the logical layer and physical layer logs representing the status information of the communication link of the physical layer; using an artificial intelligence big data model, determining whether there is a communication performance fault during the task execution process based on the logical layer logs and physical layer logs; in response to determining the existence, using a deep tracing component to perform attribution analysis of the communication performance fault based on the logical layer logs and physical layer logs, and determining the diagnosis result.

[0005] According to the second aspect, a fault diagnosis device is provided, comprising: an acquisition unit configured to acquire, during the task execution of a distributed system, logical layer logs representing the status information of the coordination operation of the logical layer and physical layer logs representing the status information of the communication link of the physical layer; a fault determination unit configured to determine, based on the logical layer logs and physical layer logs, whether a communication performance fault exists during the task execution using an artificial intelligence big data model; and a fault diagnosis unit configured to, in response to the determination of the existence of the fault, perform attribution analysis of the communication performance fault based on the logical layer logs and physical layer logs using a deep tracing component, and determine the diagnosis result.

[0006] According to the third aspect, an intelligent agent for fault diagnosis is provided, comprising: an input module for receiving logical layer logs and physical layer logs during the task execution process of a distributed system, wherein the logical layer logs represent the status information of the coordination operation of the logical layer, and the physical layer logs represent the status information of the communication link of the physical layer; a processing module for determining whether a communication performance fault exists during the task execution process based on the logical layer logs and physical layer logs using an artificial intelligence big data model; in response to the determination of the existence, performing attribution analysis of the communication performance fault based on the logical layer logs and physical layer logs using a deep tracing component to determine the diagnostic result; and an output module for outputting the diagnostic result.

[0007] According to a fourth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] According to a fifth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0009] According to a sixth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0010] According to the technology disclosed herein, a fault diagnosis method and apparatus are provided. During the task execution process of a distributed system, logical layer logs representing the state information of the coordination operation of the logical layer and physical layer logs representing the state information of the communication link of the physical layer are acquired. Using an artificial intelligence big data model, the existence of communication performance faults during task execution is determined based on the logical layer logs and physical layer logs. In response to the determination of existence, a deep tracing component performs attribution analysis on the communication performance faults based on the logical layer logs and physical layer logs to determine the diagnostic result. This provides a diagnostic method for communication performance faults in distributed systems. Based on the logical layer logs representing the state information of the coordination operation of the logical layer and the physical layer logs representing the state information of the communication link of the physical layer, combined with the fault existence determination of the big data model and the attribution analysis of the deep tracing component, automated diagnosis of communication performance faults is achieved, improving diagnostic efficiency and accuracy.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure; Figure 2 This is a flowchart of one embodiment of the fault diagnosis method according to the present disclosure; Figure 3 This is a schematic diagram of the log collection process according to this embodiment; Figure 4 This is a schematic diagram illustrating an application scenario of the fault diagnosis method according to this embodiment; Figure 5 This is a schematic diagram of the fault diagnosis architecture under the large model training task according to this embodiment; Figure 6 This is a flowchart of yet another embodiment of the fault diagnosis method according to the present disclosure; Figure 7 This is a structural diagram of one embodiment of the fault diagnosis apparatus according to the present disclosure; Figure 8 This is a structural diagram of one embodiment of the fault diagnosis intelligent agent according to the present disclosure; Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] The technical solutions disclosed herein involve the collection, storage, use, processing, transmission, provision, and disclosure of various types of information, such as user personal information, in accordance with relevant laws and regulations and do not violate public order and good morals.

[0015] Figure 1 An exemplary architecture 100 is shown that can be applied to the fault diagnosis methods and apparatus of this disclosure.

[0016] like Figure 1As shown, the system architecture 100 may include distributed servers 101, 102, and 103, a network 104, and a fault diagnosis server 105. The communication connections between distributed servers 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between distributed servers 101, 102, and 103 and the fault diagnosis server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0017] Distributed servers 101, 102, and 103, in a cloud-native architecture, are multiple independent physical or virtual servers interconnected via a network, collectively forming a unified resource pool. In this system, containers, as lightweight and portable application encapsulation units, are the fundamental carriers for deploying and running distributed services. Container orchestration platforms such as Kubernetes abstract resources such as CPU, GPU, and memory in the distributed server cluster and dynamically schedule containers to appropriate server nodes for execution based on policies. A single distributed task (such as deep learning training) is broken down into multiple container instances, each carrying a portion of the computational load. These containers may be distributed across different servers, collaborating through high-speed networks (such as RDMA). Containers provide process-level isolation, ensuring that tasks do not interfere with each other; simultaneously, the rapid start-up and shutdown characteristics of containers enable distributed systems to scale elastically according to load. Distributed servers provide physical computing resources, while containers are the logical units on which distributed services are scheduled, isolated, and run; together, they support an efficient and elastic cloud-native distributed computing environment.

[0018] The fault diagnosis server 105 can be a server used to diagnose communication performance faults during the task execution of a distributed server. For example, it can be a background processing server that diagnoses communication performance faults based on the logical layer logs representing the status information of the coordination operation of the logical layer and the physical layer logs representing the status information of the communication link of the physical layer during the task execution of distributed servers 101, 102, and 103. As an example, the fault diagnosis server 105 can be a cloud server.

[0019] It should be noted that distributed servers and fault diagnosis servers can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0020] The fault diagnosis server can be deployed in two ways: one is integrated, which is a built-in functional module of the distributed server, sharing physical resources and performing internal communication processes, and is suitable for small-scale scenarios; the other is standalone, which is a dedicated server that communicates with the distributed server via the network and centrally processes fault diagnosis tasks for the entire cluster, and is suitable for medium-to-large-scale cloud computing scenarios.

[0021] It should also be noted that the fault diagnosis method provided in the embodiments of this disclosure is generally executed by a fault diagnosis server, but the possibility of it being executed by a distributed server, or by a distributed server and a fault diagnosis server cooperating with each other, is not excluded. Accordingly, all parts (e.g., units) of the fault diagnosis device can be set in the fault diagnosis server, all parts can be set in the distributed server, or they can be set in the fault diagnosis server and the distributed server respectively.

[0022] It should be understood that Figure 1 The number of networks and servers shown is merely illustrative. Any number of networks and servers can be used depending on implementation needs. When the electronic devices on which the fault diagnosis method runs do not require data transmission with other electronic devices, the system architecture may consist only of the electronic devices on which the fault diagnosis method runs (e.g., fault diagnosis servers).

[0023] Please refer to Figure 2 , Figure 2 A flowchart of a fault diagnosis method provided in this embodiment of the present disclosure. Flowchart 200 includes the following steps: Step 201: During the task execution of the distributed system, obtain the logical layer log representing the status information of the coordination operation of the logical layer and the physical layer log representing the status information of the communication link of the physical layer.

[0024] In this embodiment, the entity executing the fault diagnosis method (e.g., Figure 1 The fault diagnosis server in the distributed system acquires logical layer logs representing the status information of the coordination operation of the logical layer and physical layer logs representing the status information of the communication link of the physical layer during the task execution process.

[0025] A distributed system is a software system in which multiple interconnected computing nodes (such as servers, virtual machines, or container groups) work together to achieve a common business objective. In a distributed system, the business is broken down into multiple subtasks, which are executed in parallel by different nodes, and the nodes must communicate with each other to exchange data and synchronize their states.

[0026] Business operations in a distributed system can be any business scenario that relies on close collaboration and communication between multiple nodes, including but not limited to: Large-scale model training involves parallel parameter optimization on distributed clusters consisting of hundreds to thousands of GPUs for deep learning models (such as large-scale language models and multimodal models) with hundreds of billions or even trillions of parameters.

[0027] Large-scale data processing, such as Apache Spark jobs (which are distributed data processing tasks consisting of a series of computational stages), involves a large amount of data shuffling and exchanging across nodes during its Shuffle stage.

[0028] Distributed scientific computing, such as computational fluid dynamics simulations, requires frequent synchronization of boundary conditions between different computing units.

[0029] Distributed databases, such as distributed transaction processing, involve consistency protocol communication between multiple replicas (such as Raft and Paxos).

[0030] The logical layer, in a distributed software stack, is an abstract layer for collaboration and communication oriented towards business semantics. It defines "why" and "how" nodes cooperate in business logic (e.g., data reduction, synchronization, or aggregation), typically implemented through specific communication libraries or protocol interfaces (such as collection communication libraries). The coordination operations of the logical layer refer to the communication actions in a distributed system where multiple nodes cooperate according to a predetermined pattern to advance a common business goal, such as a global data synchronization (AllReduce) or a barrier wait. Its status information includes, but is not limited to, operation type, initiation and termination timestamps, list of participating nodes, current stage (e.g., waiting, in progress, completed), and the reason for blocking or waiting (e.g., waiting for data from a certain node).

[0031] The physical layer refers to the underlying hardware and basic network infrastructure layer that supports communication in the logical layer. It involves the actual transmission of data bitstreams over physical media (such as network interface cards, switches, and fiber optic cables), and is managed by the operating system kernel, drivers, and network protocol stack. A physical link in the physical layer refers to the physical network path established between two or more communication endpoints for data transmission, involving specific network interfaces, switching equipment, and transmission media. Its status information includes, but is not limited to, link throughput, packet transmission delay, packet loss rate, retransmission count, error packet count, bandwidth utilization, and hardware interrupt frequency.

[0032] Taking the training of large models as an example, coordination operations manifest as frequent and intensive aggregate communication, such as synchronizing gradients computed by all GPUs through AllReduce (global reduction) or collecting distributed samples through AllGather (global aggregation). The progress of a training iteration depends entirely on the successful completion of these cross-GPU coordination operations. The physical link is the high-speed interconnect network supporting these coordination operations, including intra-node GPU interconnects (such as NVLink) and inter-node networks (such as InfiniBand / RDMA). Its transmission performance directly determines the synchronization speed.

[0033] As an example, during the runtime of a distributed system, by calling the built-in performance analysis interface or log output function of the collective communication library it uses (such as NCCL (NVIDIA Collective Communications Library)), detailed records of coordination operations (such as operator call sequences, execution times, and wait events) are obtained and parsed to form a logical layer log. Simultaneously, through performance counters and statistics exposed by the operating system kernel and network drivers (e.g., via ethtool, netstat, or dedicated RDMA performance counters), detailed traffic, error, and latency data of the network interfaces traversed by the service flow are collected in real time to form a physical layer log.

[0034] As another example, a low-intrusion system monitoring agent is deployed on a distributed server hosting distributed services. This agent uses system tracing tools (such as Linux perf and eBPF programs) to dynamically hook critical system calls and library functions, thereby capturing the call stack, duration, and frequency information of inter-process communication. This information is then aggregated to generate logical layer logs reflecting application-layer coordination behavior. Simultaneously, the agent directly captures network packet metadata and queue states from the kernel network stack and hardware driver layer, using flow analysis to obtain throughput, jitter, and packet loss metrics for each service communication link, generating physical layer logs.

[0035] Continue to refer to Figure 3 The diagram illustrates the log collection process.

[0036] In some optional embodiments of this example, the execution entity can perform the process of obtaining the logical layer log in step 201 in the following manner: The first step involves using the logic layer plugins within the container group of the distributed system to collect status information on the coordination operations of the logic layer involved in the container group during task execution, thus obtaining the logic layer sub-logs. The second step is to combine the logic layer sub-logs corresponding to each container group in the distributed system to obtain the logic layer logs.

[0037] In the cloud-native technology ecosystem, a container group (Pod) is the smallest atomic unit for deployment, management, and scheduling in Kubernetes. Essentially, it is a logical host that encapsulates one or more tightly coupled application containers (business containers). These containers share the same network namespace (IP address, port space) and storage volumes, thus being collaboratively created, destroyed, and scheduled as a whole. A business application typically consists of one or more Pods.

[0038] First, when a logic layer plugin (e.g., NCCL_PROFILER_PLUGIN) starts and loads with the business container, its initialization routine locates and hooks the function entry addresses of key operators in the target set communication library (such as NCCL). These operators include, but are not limited to, ncclAllReduce, ncclBroadcast, and ncclAllGather. The hooking is achieved by modifying the jump instructions of the target functions in memory, ensuring that all calls to these standard interfaces are first directed to the plugin's monitoring code.

[0039] Then, during task execution, when the task code calls the hijacked collection communication function, the logic layer plugin performs full data collection: The call context record generates and records the unique trace ID (Identifier), timestamp, process / thread ID, operation type, data size, data type, and a list of processes involved in this call.

[0040] The execution timing capture from the perspective of this container group includes recording the start and end timestamps of the operation, calculating the time taken, and sampling and recording the state sequence observed within the process of this container group at a fixed frequency.

[0041] The system associates wait events with this container group, captures and records the wait event identifiers that cause processes in this container group to block.

[0042] Then, a background thread within the logic layer plugin periodically formats the logic layer sub-logs collected by this container group into predefined structured JSON (JavaScript Object Notation) entries. These entries are then written to a specific file under a shared volume mount point accessible only to containers within this container group.

[0043] Finally, the corresponding logical layer sub-logs for each container group in the distributed system are read and sent to centralized storage or analysis services. All logical layer sub-logs from all container groups are aggregated and aligned in chronological order at the central server to form the logical layer logs.

[0044] In this implementation, full data capture is achieved by hijacking the communication library interface. Without modifying the business code, a complete and accurate runtime sequence and state sequence of collaborative operations in the distributed system are obtained, providing an indispensable and high-fidelity raw data foundation for any subsequent analysis of communication performance faults.

[0045] In some optional implementations of this embodiment, the execution entity performs the first step described above to obtain the logical layer sub-log in the following manner: First, in response to the diagnostic type label included in the configuration file corresponding to the container group to indicate the target fault type, the collection strategy corresponding to the diagnostic type label is determined; then, the collection strategy is adopted to collect the status information of the coordination operation of the logical layer involved in the container group during the task execution, and obtain the logical layer sub-log.

[0046] The data collection strategy refers to a set of rules and parameters that are driven by diagnostic type tags and predefined for dynamically configuring logical layer plugins. It specifically specifies the data dimensions, sampling frequency, level of detail, and types of events that the plugin should collect at runtime, aiming to ensure that the data collection behavior is precisely adapted to the preset diagnostic objectives (such as locating performance degradation or deadlock).

[0047] The diagnostic type label is used to indicate that the system focuses on fault diagnosis of the target fault type, such as deepTracker-trace: "slow" or deepTracker-trace: "hang". deepTracker-trace: "slow" indicates that the above execution entity focuses on fault analysis of the slow type, and deepTracker-trace: "hang" indicates that the above execution entity focuses on fault analysis of the hang type.

[0048] As an example, firstly, when the logic layer plugin initializes, it reads the diagnostic type label value passed in from the environment variable. The plugin internally maintains a policy configuration mapping table, which represents the correspondence between fault diagnosis types and collection policies. For example, if the label value is "slow", a performance profiling policy is loaded; if the label value is "hang", a deadlock detection policy is loaded; if the label does not exist or is other values, a lightweight default monitoring policy is loaded.

[0049] Then, the logic layer plugin performs data collection by hooking the NCCL collection communication operator (such as ncclAllReduce) entry point. The core data collection behavior is as follows, depending on the strategy: 1. The goal of the performance profiling strategy for "slow" performance is to capture detailed time-series data sufficient for performance statistics and trend analysis. The data dimensions include: Complete timing sequence is used to record the precise start and end timestamps of each operation and calculate the time consumption; performance samples are used to sample long-running operations at fixed intervals (e.g., every 100 milliseconds) and record the progress status of the operation (e.g., "X bytes sent / Y bytes in total") for subsequent throughput calculation; context information is used to record the operation type, data size, and list of participating processes; a high sampling frequency is used to focus on generating continuous data sequences that can be used to calculate statistical indicators (e.g., average, quantile).

[0050] The logical sub-logs obtained by the performance profiling strategy have dense log entries, containing a large amount of time series and metric data, and are suitable for time series analysis and regression analysis.

[0051] 2. The goal of the deadlock detection strategy targeting "hang" is to capture complete context information at the moment of process blocking in order to perform deadlock root cause analysis. Its data dimensions include: Critical state transitions are used to record key points of change in operation state (such as changing from "running" to "waiting for signal"), and record the precise timestamp of the change moment; deep snapshots are used to immediately trigger "capture" once the operation exceeds the preset health threshold, recording the current thread call stack, the lock identifier held, and the synchronization object being waited for (such as the address of a condition variable); dependencies are used to enhance the recording of other process or resource identifiers that the operation depends on.

[0052] Data sampling is based on event-driven methods, with a low regular sampling frequency. However, once an abnormal condition is triggered, a one-time, high-overhead deep information capture is performed.

[0053] Log entries obtained using the deadlock detection strategy are sparse during normal periods, but contain very detailed and in-depth system state snapshots at suspected deadlock points, making them suitable for rule-based deadlock pattern matching and context analysis.

[0054] Finally, regardless of the strategy employed, the logic layer plugin will bind the status information collected according to the collection strategy with the tracking ID of that operation, format it into a structured JSON log entry, and write it to the shared volume of this container group, forming a logic layer sub-log. Each log entry will contain a "collection_strategy" field, indicating the strategy mode used to generate that log entry.

[0055] In this implementation, the collection strategy of the logical sub-log is dynamically adapted by driving the diagnostic type label, which realizes intelligent and accurate data collection. While ensuring that key fault information is not lost, the system overhead of invalid data collection is significantly reduced, and the relevance of diagnostic data and the efficiency of subsequent analysis are improved.

[0056] In some optional implementations of this embodiment, the logical layer sub-logs are stored in a shared volume of the container group. A shared volume refers to the same storage area mounted simultaneously by multiple containers within the same container group (e.g., a Kubernetes Pod). Its lifecycle is bound to the container group, providing an inter-process file system-level data sharing channel for each container within the group. In this embodiment, the shared volume serves as an intermediate data buffer between the logical layer plugin (data producer) and the companion container (data consumer). The logical layer plugin writes the logical layer sub-logs to this shared volume, and the companion container reads them from the same location, achieving asynchronous and decoupled data transfer.

[0057] A companion container is an auxiliary container that runs in parallel with the main container (business container) within the same container group. It is designed to work closely with the business container, typically sharing the network namespace and sometimes the process namespace. Its lifecycle is completely synchronized with the business container, providing independent, pluggable auxiliary capabilities (such as log collection, proxying, and monitoring) for core business operations. In this embodiment, the companion container is dedicated to collecting and forwarding diagnostic data in a low-overhead manner, and its existence and operation are transparent to the main business container.

[0058] In this implementation, the execution entity performs the second step above in the following way to obtain the logical layer log: First, it obtains the logical layer sub-log from the shared volume through the companion container of the business container in the container group; then, it combines the logical layer sub-log obtained by each companion container to obtain the logical layer log.

[0059] As an example, firstly, during the execution of the distributed task, the logical layer plugins in each container group run independently, collecting the logical layer coordination operation status information involved in their respective container group and writing it in real time to a designated log file in a shared volume unique to that container group. This designated log file is the logical layer sub-log of that container group.

[0060] Then, within each container group, a companion container runs a lightweight log collector. This collector continuously monitors the child log files in the shared volume of its container group. It uses an append-only read approach, periodically (e.g., every second) checking for new content in the files. Once a new log entry is detected, it is immediately read into the companion container's memory buffer.

[0061] Then, each companion container in the distributed system reads the latest sub-log data, appends necessary metadata (such as source container group ID, host node, and timestamp), and asynchronously sends it to a centralized log aggregation service (such as an Elasticsearch index or Kafka topic) via the cluster's internal network. This service, as part of the "deep tracing component," receives data streams from all companion containers in the container group.

[0062] Finally, the centralized log aggregation service sorts, deduplicates, and merges the sub-log entries reported by all container groups according to a globally unified timestamp, ultimately generating a unified and complete log dataset organized by time series and containing the coordinated operation status of all container groups.

[0063] In this implementation, each companion container collects sub-logs from the local shared volume, reports and integrates them, and realizes unified collection and standardized processing of distributed data collection. This provides complete and consistent data for global fault analysis and helps improve the accuracy of fault diagnosis.

[0064] In some optional implementations of this embodiment, the execution entity can perform the process of obtaining the physical layer log in step 201 in the following manner: The first step is to collect the status information of the physical layer communication link of the node where the container group is located during the task execution by using the physical layer plugin in the container group of the distributed system, and obtain the physical layer sub-log.

[0065] As an example, firstly, within each container group, the physical layer plugin starts with the application container. It is loaded as a kernel module or user-space driver library and hooked to the critical paths of the network protocol stack of the distributed system and the performance counter interface of the GPU driver. Its monitoring scope covers all relevant physical links on the node where the container group resides, including the GPU interconnect bus within the node, the physical ports of the node's network interface card (such as an RDMA network card), etc.

[0066] Then, the physical layer plugin polls or uses an event-driven approach to collect raw performance data from the mount point at a fixed high frequency (e.g., 1000 times per second). Key metrics collected include: Network link metrics, such as the number of bytes sent / received, data packets, erroneous packets, packet loss, retransmission count, and link utilization for a specified port.

[0067] GPU interconnect bus metrics, such as NVLink bandwidth utilization, CRC (Cyclic Redundancy Check) error count, and trainer replay error count.

[0068] System events, such as network card driver error logs and DMA transfer timeout events.

[0069] Finally, the physical layer plugin aggregates the collected high-frequency raw data in user space at the millisecond level (e.g., calculating average and cumulative values ​​per second) and binds it to precise timestamps. Subsequently, the physical layer plugin formats these aggregated metric data into structured log entries (JSON format, containing fields such as node_id, timestamp, nic_throughput, and link_error_count) using the node as the primary key, and continuously appends them to a specified file in the shared volume of this container group. This specified file is the physical layer sub-log corresponding to that node.

[0070] The second step is to combine the physical layer sub-logs corresponding to each container group in the distributed system to obtain the physical layer logs.

[0071] As an example, firstly, the companion container within each container group continuously monitors and reads the physical layer sub-log files generated by the physical layer plugin in the shared volume of this container group.

[0072] The accompanying container then appends necessary metadata to each physical layer log entry it reads, primarily container group identifiers and uniform, high-precision collection timestamps. Subsequently, the log entries are asynchronously sent to a centralized time-series database (such as Prometheus or a proprietary log aggregation service) via a lightweight network transport protocol.

[0073] Then, the centralized service receives data streams reported by companion containers on all nodes of the cluster. It uses timestamps and node identifiers as a composite primary key to clean, sort, and merge all the data. Since the physical layer logs are based on nodes, data reported by different container groups for the same node will be deduplicated and merged to ensure that each node has only one authoritative record of the physical status at each point in time.

[0074] Finally, the integrated data forms a panoramic dataset organized chronologically, covering the physical status of all nodes and links in the cluster. This structured dataset, which can be used for global querying and analysis, is the final physical layer log.

[0075] This implementation method achieves full and standardized collection and integration of the status of the underlying hardware communication links of the distributed cluster, providing an indispensable and accurate data foundation for accurately linking application layer faults and underlying hardware anomalies.

[0076] In some optional implementations of this embodiment, the execution entity can perform the first step described above to obtain the physical layer sub-log in the following manner: First, in response to the diagnostic type label included in the configuration file corresponding to the container group to indicate the target fault type, the collection strategy corresponding to the diagnostic type label is determined; then, the collection strategy is adopted to collect the status information of the physical layer communication link of the node where the container group is located during the task execution to obtain the logical layer sub-log.

[0077] As an example, firstly, when the container group starts, the physical layer plugin reads the diagnostic type tags (such as deepTracker-trace: "slow" or "hang") injected into the container environment variables. The physical layer plugin internally pre-defines the mapping relationship between tag values ​​and collection strategies. For example, if the tag value is "slow," the performance baseline collection strategy is determined to be used. If the tag value is "hang," the anomaly depth collection strategy is determined to be used. Each strategy explicitly defines the range of hardware metrics to be monitored, the sampling frequency, and the anomaly trigger threshold.

[0078] Then, the physical layer plugin executes policy-based data collection by directly calling the hardware performance counter interface and the operating system network stack monitoring interface. For example, the core collection behaviors are as follows, depending on the collection strategy: 1. The objective of the performance baseline acquisition strategy for "slow" performance is to establish a performance benchmark and trend model for the communication link and identify performance degradation. The acquisition actions include: For critical links (such as the GPU interconnect bus within a node and RDMA network interface card ports), aggregated values ​​of throughput, utilization, and average latency are collected at a moderate frequency (such as 10 times per second).

[0079] Physical errors (such as ECC errors and link layer retransmissions) are cumulatively counted, but detailed recording only begins when a higher trigger threshold is used.

[0080] Focus on long-term changes in statistical indicators, such as calculating the standard deviation of throughput per second and observing whether bandwidth utilization consistently exceeds a set threshold.

[0081] The physical layer sub-log entries generated using the performance baseline collection strategy focus on trend and statistical data, with a relatively stable data volume, making them suitable for time series analysis and performance bottleneck modeling.

[0082] 2. The target of the abnormal depth acquisition strategy for "hang" is to capture transient hardware errors and abnormal events that may cause a complete communication interruption. Its acquisition actions include: Significantly improve the sampling and capture frequency of physical errors and abnormal events (e.g., millisecond level), including events such as packet CRC errors, link training failures, DMA timeouts, and hardware queue stalls.

[0083] When any critical error counter shows a non-zero increase within a short period of time (e.g., within 1 second), a deep snapshot of the associated context is immediately triggered to record the complete hardware register state (e.g., network card status register), driver error log, and associated process / thread information at that moment.

[0084] Simultaneously monitor the heartbeat or liveness signal of the link, and immediately record any signal loss as the highest priority event.

[0085] When using the abnormal deep collection strategy, the log volume is small during normal periods. Once a hardware-level anomaly is detected, a high-density diagnostic log entry containing a deep hardware state snapshot is generated.

[0086] Finally, the physical layer plugin binds the physical link status information collected according to the policy with precise timestamps and node identifiers, formats it into structured data (such as JSON), and continuously writes it to a designated file on the shared volume of this container group, forming the physical layer sub-log of the node where the container group resides. The log will contain metadata fields that identify the collection policy.

[0087] In this implementation, the monitoring focus and granularity of the physical layer plugin are dynamically configured through diagnostic type tags, which enables precise targeting of hardware layer monitoring. While effectively capturing low-level abnormal events that cause communication interruptions, it avoids collecting redundant data in performance analysis scenarios, significantly improving the timeliness and accuracy of hardware fault diagnosis.

[0088] In some optional implementations of this embodiment, the physical layer sub-log is stored in the shared volume of the container group. In this implementation, the execution entity performs the second step described above to obtain the physical layer log in the following manner: First, physical layer sub-logs are obtained from the shared volume through the companion containers of the business containers in the container group; then, the physical layer logs are obtained by combining the physical layer sub-logs obtained by each companion container.

[0089] As an example, firstly, during the execution of tasks in a distributed system, the physical layer plugin in each container group continuously collects the physical layer communication link status information of its node, formats this information into structured log entries, and continuously writes them to a designated file in the shared volume of the container group. This designated file is the physical layer sub-log of the container group.

[0090] Then, within each container group, a lightweight log collection process runs within the companion container of the business container. This process continuously monitors changes to the physical layer sub-log files in the shared volume of this container group. Once new content is detected in a file, these new log entries are immediately read and temporarily stored in the companion container's memory buffer.

[0091] Then, each companion container asynchronously and in batches sends the physical layer sub-log entries (with attached metadata such as the source container group identifier and node information) in its buffer to a centralized log aggregation service via the cluster's internal network. This service acts as a data aggregation point, receiving data streams from all companion containers of the container group.

[0092] Finally, after receiving the physical layer sub-logs reported by all container groups, the centralized log aggregation service performs the following integration operations to generate unified physical layer logs: Time alignment and sorting: Sort all log entries according to a globally unified timestamp.

[0093] Node data merging: Since physical layer monitoring is done on a node-by-node basis, logs from different container groups (if any) on the same node will be merged to ensure the uniqueness of the status record for each physical entity (such as each network card or each interconnect bus) at each point in time.

[0094] Structured storage persistently stores the integrated data as a global dataset that can be quickly queried by time range, node, and link type. This complete dataset is the final physical layer log.

[0095] In this implementation, each companion container automatically collects and reports dispersed node-level physical layer data for standardized integration, providing an authoritative and consistent data benchmark for accurately locating distributed communication faults caused by hardware link anomalies.

[0096] Step 202: Using a large-scale artificial intelligence model, determine whether there are communication performance failures during task execution based on the logical layer logs and physical layer logs.

[0097] In this embodiment, the aforementioned execution entity can use an artificial intelligence big data model to determine whether there is a communication performance failure during task execution based on the logical layer logs and physical layer logs.

[0098] Communication performance failure refers to a type of failure in which, during the operation of a distributed system, although the underlying communication links or protocols maintain basic connectivity, the quality of service provided (such as latency, throughput, and reliability) is severely degraded, failing to meet the performance expectations required for the normal advancement of upper-layer collaborative services (such as aggregated communication), thus leading to a significant decrease in overall business efficiency or a complete halt in the process.

[0099] Taking Slow (performance degradation) in communication performance failure as an example, communication is not interrupted, but latency performance is substandard, making it unable to support efficient collaborative computing. Communication link performance (such as latency and throughput) is not completely lost, but it is severely below the threshold required for healthy business operation. Business processes therefore become abnormally slow, but are still progressing at extremely low efficiency.

[0100] For example, in the AllReduce operation during large model training, suppose network congestion occurs between GPU-A and GPU-B. Data packets sent by GPU-A can still reach GPU-B, but the transmission latency surges from the normal 0.1 milliseconds to 50 milliseconds. Since AllReduce requires data synchronization from all GPUs to continue, the entire training task must wait for this slowest link. An iteration that should complete in 1 second is thus stretched to tens of seconds.

[0101] Taking a communication performance failure called Hang (protocol-level stall) as an example, its root cause is not a physical connection interruption (cable being unplugged), but a protocol logic deadlock caused by communication reliability performance dropping to zero (completely unreliable). When communication quality deteriorates to a certain extent (such as 100% packet loss, second-level latency), or when defects in the protocol stack or driver are triggered, a critical communication operation (such as waiting for confirmation, data retransmission) will logically fail to complete, causing all collaborative processes that depend on it to wait indefinitely at the synchronization point, resulting in a complete halt to services.

[0102] For example, during the AllReduce operation, the network card driver of GPU-A malfunctioned, causing all its transmitted data packets to be silently dropped (100% packet loss). After multiple failed retransmissions, the protocol stack of the sending GPU-A triggered its internal timeout mechanism and entered a waiting state; the receiving GPU-B, on the other hand, was waiting indefinitely for the missing data. From the perspective of the upper-layer business logic, all GPUs were "stuck," and the training task was completely stalled.

[0103] As an example, firstly, the original, timestamped logical layer logs (e.g., "[T1] Node-A: Started AllReduce operation X, waiting for node B... [T2] Node-A: Still waiting...") and physical layer logs (e.g., "[T1-T2] Link A->B: Average latency 150ms, packet loss rate 2%) are aligned and integrated into a coherent, descriptive "system state description". Then, a clear role and reasoning rules are designed for the large model. For example: "You are a distributed system performance diagnostic expert. Please strictly analyze the provided system state description according to the following rules: Rule 1: If the logical layer logs show that a critical coordination operation (such as AllReduce) is 'waiting for a long time', 'timeout', or 'incomplete', and the physical layer logs show that the corresponding communication path experiences 'significantly higher latency than baseline', 'packet loss rate > 1%', or 'sudden drop in throughput' during the same time period, then it is determined that 'there is a communication performance failure'. Rule 2: If only one rule is met, or the phenomenon is irrelevant in time, then it is determined that 'there is no clear communication performance failure'."

[0104] Finally, the formatted "system status description" is submitted to the large language model along with the aforementioned system instructions as a user query. The large language model, leveraging its powerful semantic understanding capabilities, will apply rules like an expert to analyze the description and output structured judgments, such as: "A communication performance failure exists. Reason: The waiting time of the AllReduce operation on node A is abnormal, perfectly matching the high latency and high packet loss of the link from node A to node B during the same period, which conforms to rule 1."

[0105] As another example, first, prepare a small number (e.g., 3-5) of high-quality, labeled "example pairs". Each "example pair" includes: Input is a "state description" derived from logs from the logical and physical layers over a certain period of time.

[0106] The output should be a clear conclusion and a brief reason, such as "A fault exists. This is due to the simultaneous occurrence of operation wait and high link latency."

[0107] Then, when diagnostics are needed for a new runtime segment, the following steps are performed: The two types of logs for the current time period are converted into a "state description" to be analyzed. Several of the most relevant "example pairs" are selected from the sample library and used as contextual examples, along with system instructions (such as "Refer to the example and analyze the following new situation in the same format"), to form a complete prompt. The "state description" to be analyzed is appended at the end and submitted to the large model.

[0108] Finally, based on the provided context examples, the large model learns the pattern of "how to associate two types of information from textual descriptions and make judgments", and then imitates this pattern to analyze new "state narratives" and output judgment results with consistent format.

[0109] In some optional implementations of this embodiment, the execution entity also performs the following operation: during the task execution of the distributed system, multi-dimensional operation indicators are collected through the deep tracing component.

[0110] The deep tracing component is deployed as a node monitoring agent in the Kubernetes cluster as a DaemonSet. Ensure that an agent instance is running on every worker node in the cluster (especially GPU nodes).

[0111] The node agent integrates multiple data collectors: DCGM (NVIDIA Data Center GPU Manager) is used to collect data on GPU utilization, memory usage, power consumption, temperature, and PCIe bandwidth utilization; Node Exporter is used to collect data on node CPU utilization, memory usage, network interface throughput, and error packet count. These collectors expose the metrics to HTTP endpoints in Prometheus format.

[0112] The central metric collector of the deep tracking component periodically (e.g., every second) sends pull requests to the endpoints of all node proxies to collect raw metric data. The data is then tagged with tasks, nodes, pods, etc., and stored in a time-series database (such as Prometheus TSDB) to form queryable multidimensional runtime metric time-series data.

[0113] In this implementation, the execution entity performs step 202 as follows to determine whether there is a communication performance failure during task execution: based on the logical layer logs, physical layer logs, multi-dimensional operation indicators, business logs during task execution, and communication logs of the collection communication library used by the distributed system, it determines whether there is a communication performance failure during task execution.

[0114] As an example, firstly, multi-source data preparation and context building. The deep tracing component prepares a time-series data context for the task to be analyzed, including: A multi-dimensional sequence of performance metrics, including time-series values ​​of GPU utilization and power consumption for each node.

[0115] The logical layer log sequence is a record of the logical layer coordination operation status reported by all container groups within the most recent time period, obtained from the central storage.

[0116] Physical layer log sequence, recording the physical link status of all nodes concurrently.

[0117] The raw communication log sequence is generated by a collection of communication libraries (such as NCCL).

[0118] The business log sequence is a collection of business logs from the standard output of the business container. In training task scenarios, we focus on key fields such as step (number of training steps), loss, and iteration time.

[0119] Next, construct the system prompt. When the deep tracing component calls a large model, it provides a strict system prompt, defining its role and analysis logic. The core content is as follows: You are a distributed training anomaly diagnostic expert. Please analyze the task status strictly according to the following data and logical path: Input data: You will receive the following data: 1) GPU and other indicator trends, 2) Logical layer coordination operation logs, 3) Physical layer link logs, 4) Collection communication library raw logs, and 5) Training business logs.

[0120] Analysis path: Metric trend check: If GPU utilization is below 5% for multiple consecutive windows and power consumption is silent, mark it as High-RiskHang; if some nodes have extremely low utilization while others are normal, mark it as Straggler.

[0121] Communication semantic synchronization: Scan the raw communication logs and logic layer logs. If a large number of Timeout and Retransmission records are found, or if the logic logs show that operations are blocked for a long time at a specific operator (such as all_reduce_kernel) without progress, it is determined to be a Communication Deadlock.

[0122] Business logic alignment: Check business logs. If the step counter has not increased for a long time, or the loss has stopped updating, combined with the above anomalies, confirm that progress has stalled.

[0123] Judgment criteria: Normal: GPU utilization >80%, communication has Complete records, and Step progresses steadily.

[0124] Suspicious_Hang: The metric remains below the threshold, communication is blocked at a specific operator, and business logs are completely stopped.

[0125] Suspicious_Slow (Suspected slow): The indicator fluctuates greatly, the Step time is more than twice the historical average, and communication is accompanied by retransmission alarms.

[0126] Output requirements: Output in JSON format, which must include decision (the decision result, such as Normal / Suspicious_Hang / Suspicious_Slow), confidence, reasoning_steps (the reasoning steps, which describe the three analysis results in turn), and suspected_nodes.

[0127] Then, the component organizes the time-series data context prepared in the first step into a natural language description according to time windows, and submits it as a user query to the large model configured with the aforementioned system prompts. The large model then analyzes the data step by step based on the expert logic in the prompts: By comparing the GPU utilization of each window, it determines whether any nodes have fallen behind or the entire system is silent. It cross-references timeout and retransmission events in the original communication logs with operational blockages in the logic layer logs. It verifies whether the progress in the business logs matches the aforementioned anomalies in time. Based on this reasoning chain, the large model generates the final judgment.

[0128] Finally, the large model outputs a standardized JSON result as shown in the example below, which is the determination of "whether there is a communication performance failure": the decision is "suspected performance degradation". The system's confidence level for this decision is 88%. The reasoning process consists of three steps: First, by checking the operating indicators, it was found that the GPU utilization of node Node-2 has been consistently below 40% in the past three observation windows, while other nodes are normal, which is consistent with the characteristics of a "laggard". Second, log analysis shows that the physical layer logs indicate that the network card link of node Node-2 has intermittent high packet loss. At the same time, the original logs of the communication library show that the corresponding node has a packet retransmission alarm. Finally, the comprehensive business logs show that the training iteration time has increased to 2.1 times the historical baseline level. Combining the aforementioned node performance lag and communication link anomalies, the current task status is finally determined to be "suspected performance degradation". Based on this determination, the suspected faulty node is located as Node-2.

[0129] This implementation integrates logical layer logs, physical layer logs, multi-dimensional operational metrics, business logs, and communication logs, and utilizes a large model for intelligent correlation analysis. This enables automated and accurate preliminary judgment of communication performance faults in distributed training, significantly improving the efficiency of fault detection and delimitation, and reducing the cost of manual troubleshooting.

[0130] In some optional implementations of this embodiment, the execution entity can perform the above-mentioned communication performance fault determination process in the following manner: in response to the configuration file corresponding to the container group of the distributed system including a diagnostic type label for indicating the target fault type, the entity determines whether there is a communication performance fault under the target fault type in the task execution process based on the logical layer log, physical layer log, multi-dimensional operation indicators, business log and communication log.

[0131] As an example, the first step of the deep tracing component, when initiating fault diagnosis for a specific task, is to check the configuration file of the container group corresponding to that task. If it finds that the configuration file contains a diagnostic type label (such as deepTracker-trace: "slow" or "hang"), the component will adapt the core focus of the diagnosis strategy accordingly. If the label is "slow", the decision-making strategy will focus on performance degradation and long-tail latency. The analysis will be more sensitive to the trend of declining metrics, the gradual increase in iteration time, and the accompanying occasional communication errors.

[0132] If the label is "hang", the judgment strategy will focus on process blocking and complete stagnation. The analysis will focus on checking for persistent zero-load indicators, permanent waits of synchronization primitives, and complete freeze of business progress.

[0133] The second step involves the component acquiring logical layer logs, physical layer logs, multi-dimensional operational metrics, business logs, and raw communication logs. Instead of performing general correlations, it extracts features strongly correlated with the target fault type from the massive dataset, based on the adaptation strategy established in the first step. For example: The analysis of "slow" includes: 1. Align metrics with business logs, calculate the average iteration time for multiple recent time windows, and compare it with the historical baseline. If it is found to consistently exceed the baseline by a certain percentage (e.g., 1.5 times), it is marked as a critical clue.

[0134] 2. Deep log analysis: In the logical layer logs, filter the top 10% of coordination operations in terms of time consumption, and check the corresponding physical layer logs during these operations to see if there is link utilization saturation or periodic spikes in latency. At the same time, search the raw communication logs for non-fatal warnings (such as lightweight retransmissions).

[0135] 3. Based on comprehensive judgment, if the three factors of "increased iteration time", "existence of long-tail operations" and "periodic link congestion / lightweight errors" overlap in time, it is determined that there is a "slow" type of communication performance failure.

[0136] The analysis of “hang” includes: 1. Align metrics with business logs. Check whether the GPU utilization in the runtime metrics is consistently below the deadlock threshold (e.g., 5%), and whether the training steps in the business logs show no increase within the extended window.

[0137] 2. Deep log analysis: In the logical layer logs, search for coordination operations that have consistently been in a "Waiting" or "Blocked" state and have not completed within a timeout period. In the original communication logs, strictly match the time window of this operation to check for serious communication timeouts, connection aborts, or fatal errors. Simultaneously, check the physical layer logs for records of link disconnections or fatal hardware errors at that time.

[0138] 3. Based on comprehensive judgment, if the four-fold evidence chain of "indicator stagnation", "business progress freeze", "logic operation permanent blockage" and "fatal communication error / physical link interruption" is complete, it is determined that there is a "hang" type communication performance failure.

[0139] The third step, regardless of the type, ultimately outputs a standardized assessment result. The result must clearly identify the fault type (corresponding to the diagnostic type label) and list the key chain of evidence extracted from multi-source data that supports the conclusion. For example, for the "slow" type, the output will emphasize the deterioration trend in performance; for the "hang" type, the output will emphasize the complete stagnation of the state.

[0140] In this implementation, diagnostic type labels guide the large model to perform targeted analysis on multi-source data, realizing the transformation from "general analysis" to "precise focusing", which significantly improves the identification speed and judgment accuracy of specific types of communication performance faults.

[0141] In some optional implementations of this embodiment, the execution entity can perform the above-mentioned communication performance fault determination process in the following way: based on the logical layer logs, physical layer logs, multi-dimensional operation indicators, business logs and communication logs of the distributed system within multiple time windows up to the present, determine whether there is a communication performance fault under the target fault type in the task execution process.

[0142] As an example, firstly, a multi-window time-series data circular buffer is constructed and maintained. Specifically, the deep tracking component maintains a fixed-capacity time-series data circular buffer for the distributed tasks to be monitored. This buffer stores consecutive, equal-length observation windows of data in chronological order (e.g., each window is 60 seconds, with a total capacity of 10 windows, i.e., data from the most recent 10 minutes).

[0143] At the end of each observation window, the component automatically aligns and snapshots the logical layer logs, physical layer logs, multi-dimensional operating indicators, business logs, and communication logs collected in the current window to form a complete "system status snapshot," and pushes it as the latest entry into the circular buffer.

[0144] Then, trend analysis and fault determination are performed based on multi-window data. Specifically, when fault diagnosis is triggered, the component does not only analyze the snapshot of the latest window, but traverses and analyzes the most recent N (e.g., 5) consecutive "system status snapshots" stored in the circular buffer.

[0145] For each data source, extract its trend across multiple windows. For example, calculate the moving average and standard deviation of GPU utilization, observe whether the sliding window quantile of iteration time (e.g., P95) increases continuously, and observe whether the statistical communication error count shows a cumulative increase.

[0146] The extracted multi-window trend patterns are matched against the expected patterns of the target fault type (defined by the diagnostic type label). If the target type is "slow", it checks whether there is a trend where the P95 value of the iteration time or the time spent on a specific operation exceeds the baseline for multiple consecutive windows and continues to deteriorate, accompanied by a gradual increase in physical link utilization or error rate. If the target type is "hang", it checks whether there is a deterministic pattern where the GPU utilization is below the deadlock threshold for multiple consecutive windows without fluctuation, while the number of training steps in the business logs is completely stagnant in all observed windows.

[0147] The component only determines that a communication performance fault of that type "exists" when the trends of multiple windows consistently and continuously point to the characteristic pattern of the target fault type. This effectively eliminates false judgments caused by momentary jitter or single-window anomalies.

[0148] In this implementation, by analyzing the data trends of multiple consecutive time windows, the persistence and progressive characteristics of faults are effectively identified, which significantly improves the robustness and accuracy of fault determination for slow or steady-state deadlock faults and avoids false alarms caused by instantaneous interference.

[0149] Step 203: In response to the confirmed existence, the deep tracing component performs attribution analysis on the communication performance failure based on the logical layer logs and physical layer logs to determine the diagnostic results.

[0150] In this embodiment, in response to the determination of existence, the aforementioned execution entity performs attribution analysis of communication performance failure based on logic layer logs and physical layer logs using a deep tracing component to determine the diagnostic result.

[0151] The Deep Tracing component is a centralized intelligent diagnostic engine deployed in a cloud-native distributed environment. Its core responsibility is to aggregate, correlate, and deeply analyze multi-source heterogeneous data collected from various business container groups, representing different levels of system status, aiming to automatically locate performance bottlenecks and root causes of failures affecting business collaboration processes. This component implements the following key features: 1. Cross-layer data fusion: It can receive and correlate logs from the logical layer (business collaboration semantics) and the physical layer (hardware transmission status) to build a complete observation view from application to infrastructure.

[0152] 2. Intelligent Attribution Analysis: Built-in or integrated advanced analysis algorithms (such as AI (Artificial Intelligence) models) can automatically identify abnormal patterns and their causal relationships in multi-layer data, tracing surface performance phenomena (such as "slower training") back to specific, actionable fault entities (such as "high latency caused by congestion in the RDMA link between node A and node B").

[0153] 3. Precise location and decision support: Its output is not a general alarm, but a structured diagnostic report that includes specific abnormal objects (such as nodes, GPUs, network cards), fault time windows and the most likely root causes, providing direct basis for automatic recovery or operation and maintenance decisions.

[0154] As an example, firstly, a spatiotemporal attribute graph is constructed based on logical layer logs and physical layer logs, where each node in the spatiotemporal attribute graph represents a business process or computing unit. Based on the logical layer logs, logical edges are established in the spatiotemporal attribute graph between nodes where coordination operations occur. The attributes of the logical edges include operation type (e.g., AllReduce), start / end time, final state (success, timeout), and waiting time. Based on the physical layer logs, physical edges are established between node pairs that have actual data communication. The attributes of the physical edges include metrics such as average latency, packet loss rate, and throughput within the time window.

[0155] The edges of the two levels are associated with their corresponding nodes and time windows to form a fused "logical-physical" association graph with temporal attributes.

[0156] Then, within the time period where the fault was identified, the logical edges are traversed, and all edges with abnormal states (such as timeouts and failures) are marked. For each abnormal logical edge, the set of physical edges it depends on (i.e., the actual links carrying the data transmission of that logical operation) is traced back. Root cause localization algorithms (such as contribution analysis based on random walks or minimum cut set algorithms) are applied. The core logic of this algorithm is to find a minimum set of physical edges or nodes that, when the state of this set is abnormal (such as extremely high physical edge latency), can explain all observed logical edge anomalies to the greatest extent. For example, the algorithm might find that all timed-out AllReduce operations depend on a certain physical link "from node A to switch S", and that this link has an extremely high packet loss rate during the fault period, thus inferring it as the root cause.

[0157] Finally, a diagnostic result is generated, and the output clearly identifies the core fault entity (such as "NIC-1 of Node A" or "physical link between Node B and Node C") and the fault type (such as "hardware error causing high packet loss" or "bandwidth congestion") that caused this communication performance failure, and attaches key log summaries as evidence.

[0158] As another example, firstly, key performance indicators, such as "average time of each coordination operation" and "operation timeout rate," are extracted from the logical layer logs to form a sequence of logical performance indicators. Secondly, performance indicators for each link, such as "link latency," "packet loss rate," and "utilization rate," are extracted from the physical layer logs to form a sequence of physical status indicators.

[0159] Then, all metrics are aligned according to a unified timestamp to form a multivariate synchronous time-series dataset.

[0160] Then, within the failure time window, causal discovery algorithms (such as constraint-based PC algorithm and time-based Granger causality test) are applied to analyze the multivariate time-series data. The goal is to identify which changes in physical state indicators statistically significantly "caused" the deterioration of logic performance indicators.

[0161] For example, the algorithm might discover that "the packet loss rate of link L" is a Granger cause of "the time taken for operation O", and that the two are highly correlated during the period of failure.

[0162] Furthermore, by using contribution quantification models (such as Shapley values ​​and regression weight analysis), the contribution of each physical metric identified as a cause to the overall logic performance degradation is calculated.

[0163] Finally, the diagnostic results are generated, outputting a sorted list of root causes. For example: "1. Primary root cause: The packet loss rate of the link (ID: Link_XY) between node X and node Y has surged (contribution 75%), directly causing AllReduce operations that rely on this link to time out; 2. Secondary factor: The PCIe bandwidth utilization of node Z is saturated (contribution 20%), exacerbating data exchange latency."

[0164] In some optional implementations of this embodiment, the execution entity performs step 203 as follows: by using the deep tracing component, it performs attribution analysis of communication performance failures based on the logical layer logs, physical layer logs, business logs during task execution, and communication logs of the collection communication library used by the distributed system, and determines the diagnostic results.

[0165] As an example, the first step is spatiotemporal alignment of multi-source logs and construction of event chains. The deep tracing component first obtains four types of logs related to the fault period: logical layer logs, containing the state sequence of coordination operations (such as AllReduce); physical layer logs, containing state indicators of inter-node network links and GPU interconnect buses; business logs, containing business progress records such as training steps, loss, and iteration time; and communication logs, which are raw communication event records generated by communication libraries (such as NCCL).

[0166] The component uses a globally unified high-precision timestamp as a benchmark to clean, sort, and align all the log entries in time and space, constructing a unified timeline that includes software calls, hardware status, business progress, and original communication events.

[0167] The second step is cross-layer correlation analysis and fault propagation path tracing. The component runs a built-in causal correlation analysis engine, which executes the following core logic based on an aligned timeline: To pinpoint the origin of the anomaly, within the unified timeline, first locate the starting point T_f in the business logs where the training progress stagnated or slowed significantly.

[0168] Trace back to the logical blockage point. Within the time window near T_f, scan the logical layer log to find the first critical coordination operation (denoted as Op_f) that entered the "waiting" or "timeout" state and failed to recover, and extract the object it was waiting for (such as a specific node Rank).

[0169] Tracing down to the physical anomaly, for the execution time period of Op_f and the nodes involved, the physical layer logs are retrieved to check whether there are abnormal states such as high latency, high packet loss, sudden drop in bandwidth or surge in hardware error count between the corresponding nodes, and the start time T_p is located.

[0170] Verify the original communication evidence. During the time period from T_p to T_f, retrieve the communication logs to verify whether there are any records of communication timeouts, excessive retransmissions, connection errors, or protocol-level anomalies that match physical anomalies and logical blockages.

[0171] By establishing a causal chain, the engine constructs a fault propagation hypothesis chain from physical layer anomaly to logical layer blockage and then to business layer performance based on the chronological order (T_p -> communication anomaly -> Op_f blocking -> T_f business anomaly) and the correlation strength of events.

[0172] The third step is root cause localization and diagnostic result generation. The analysis engine evaluates the established fault propagation chain, calculates its confidence level, and generates structured diagnostic results. These results explicitly include: The root cause entity is the final physical or logical entity that caused the failure, such as: "the RDMA link (NIC port eth1) between node A and node B" or "the NVLink connection of GPU 2 on node C".

[0173] The fault type and the specific fault mechanism corresponding to the initial judgment, such as "synchronization timeout caused by continuous high packet loss in the link" or "data exchange congestion caused by GPU memory bandwidth saturation".

[0174] Time interval: The exact time window from when a fault occurs to when it affects business operations.

[0175] The evidence summary lists the key pieces of evidence extracted from the logs at each level that support the conclusion.

[0176] In this implementation, by systematically associating and tracing the abnormal event chain in the logs of the four layers of business, logic, communication, and physical, it achieves automated inference from vague performance phenomena to precise and locatable hardware or software root causes.

[0177] The aforementioned executing entity can check whether the faulty entity has been accurately located in the report, specifically by determining whether the "faulty node" and "faulty process ranking" fields are null values. This judgment uses "OR" logic; as long as the target is successfully located at either the physical node level or the distributed application process level, the diagnostic conclusion is deemed executable, and the response mechanism should be triggered immediately. This design ensures that the system can effectively handle both hardware failures at the infrastructure layer and application process anomalies at the service layer simultaneously.

[0178] Once the judgment conditions are met, the executing entity will perform two core actions sequentially. The primary action is to trigger a precise alarm. This alarm is not a simple threshold exceedance notification, but rather directly encapsulates the intelligent analysis results from the diagnostic report, namely the "anomaly type" and "root cause analysis." This upgrades the alarm information itself from "reporting phenomena" to "reporting diagnostic conclusions," for example, from "high GPU utilization" to "GPU Y on node X experiences a sudden performance drop due to a memory hardware error," thereby greatly reducing the time required for operations and maintenance personnel to locate the root cause of the problem.

[0179] Following this, the aforementioned execution entity will also perform automated remediation actions. These actions invoke predefined response scripts or workflows entirely based on the "remediation suggestions" generated in the diagnostic report. Its execution strategy is distinctly layered and targeted. For faulty nodes, typical operations include isolating them from the resource pool, restarting critical services, or performing hardware resets via out-of-band management; for faulty processes, this might involve safely restarting the specific process within a distributed task framework or rescheduling its workload. This process achieves a direct transformation from analysis to action, initially containing the impact of the fault and attempting to restore services without manual intervention.

[0180] See also Figure 4 , Figure 4This is a schematic diagram of an application scenario 400 of the fault diagnosis method for a distributed system according to this embodiment. A training task for a large language model is running in the distributed server cluster 401. During the execution of the training task in the distributed system, the fault diagnosis server 402 periodically acquires logic layer logs 403, representing the status information of the coordination operation of the logic layer, and physical layer logs 404, representing the status information of the communication link of the physical layer. Using the large artificial intelligence model 405, it determines whether a communication performance fault exists during the task execution based on the logic layer logs and physical layer logs. In response to the determination of its existence, the deep tracing component 406 performs attribution analysis of the communication performance fault based on the logic layer logs and physical layer logs to determine the diagnostic result.

[0181] In some optional implementations of this embodiment, before performing step 201 above, the fir tree execution entity also performs the following operations through the depth tracking component: First, the configuration file corresponding to the container group in the distributed system includes a plugin injection tag. The plugin packages of the logical layer plugin and the physical layer plugin are copied to the shared volume of the container group through the initialization container in the container group.

[0182] For example, when the YAML configuration of a user-submitted task contains a specific plugin injection tag (such as deepTracker: "on") in the container group's metadata, the Kubernetes API Server will be intercepted by a pre-registered Mutating Admission Webhook when it receives the request.

[0183] This webhook dynamically modifies the container group creation request based on the tag, adding an initialization container (InitContainer) to the container group's specification. The image of this initialization container contains the complete plugin package.

[0184] When the container group starts, the initialization container runs first. It copies all the binary files and dependency libraries of the logical and physical layer plugins stored in its image to a shared volume defined within the Pod. After completion, the initialization container exits.

[0185] Then, inject the relevant environment variables for the logic layer plugin and the physical layer plugin into the business container in the container group.

[0186] When adding and initializing a container, the same webhook also modifies the configuration of the business containers within the container group. It injects a series of predefined environment variables into the environment variable list of the business containers, such as NCCL_DEBUG_FILE, NCCL_NET_PLUGIN="trace", NCCL_PROFILER_PLUGIN="trace", etc. These variables specify the log output path and the loading location of the plugin library for subsequent plugins.

[0187] Finally, using a preset installation script, based on the plugin package in the shared volume, the logical layer plugin and physical layer plugin are installed in the business container in the container group.

[0188] For example, in addition to the plugin package, the shared volume also contains a pre-installed script. The webhook modifies the original startup command of the business container. Typically, the original command (such as `python train.py`) is wrapped as an argument in a new command that calls the installation script.

[0189] When the application container starts, this installation script is executed first. The core logic of the script is to move or link the plugin library files in the shared volume to the application container's standard library search path. Based on injected environment variables, it configures the dynamic linker or runtime library preloading mechanism (such as setting LD_PRELOAD) to ensure that diagnostic plugins (physical layer plugins and logical layer plugins) are automatically loaded when the training process starts. The original training command (python train.py) is then executed. From this point on, the training task runs in an environment with the diagnostic plugins loaded.

[0190] In this implementation, plugin deployment and environment configuration are automatically completed through cloud-native standard interfaces, enabling zero-code intrusion and dynamically pluggable deep monitoring capabilities for distributed training tasks, which greatly reduces the threshold for enabling diagnostic functions and the complexity of operation and maintenance.

[0191] Continue to refer to Figure 5 This diagram illustrates a fault diagnosis architecture for a large model training task. To further explain the fault diagnosis process, a large model training scenario is used as an example, and the following explanation is provided: The entire process begins with the user submitting a training task definition named PyTorchJob.yaml, which specifies the use of the pytorch-mnist:2.2.1 image. When Kubernetes begins creating this training Pod, a crucial webhook_mutate server (i.e., an admission controller) automatically intervenes, injecting an additional sidecar container into the Pod, with the image deepTrackerSidecar:latest. Thus, the final Pod simultaneously runs the main training container and the sidecar container responsible for tracing.

[0192] Subsequently, the sidecar container injects the necessary environment variables into the main training container, completing the initial configuration of the tracing tool. After system startup, the specific tracing work unfolds sequentially. First, the deep_tracker (deep tracing component) background service within the sidecar container establishes a connection with the main training container via SSH. When a Trace_task command is received from the outside (which may represent area B on the operations or management side in the diagram), the sidecar issues tracing commands to the deep_tracker service within the training container through this SSH channel.

[0193] Next, within the main training container, the deep_tracker service invokes the aick_trace tracing tool. This tool reads a hostfile that defines all nodes in the cluster, thus initiating unified deep tracing across all worker nodes involved in the distributed training task, collecting data on system calls, network activity, and computing resources in real time. Simultaneously, Kubernetes' kube-probe continuously monitors the health of the Pods to ensure that tracing does not affect the stable operation of the training task.

[0194] After the tracing task is completed, the data collected from each node is aggregated, ultimately generating a detailed Trace_result report for performance analysis and optimization. Areas A and B illustrate the security and logical division of the entire architecture: Area A represents the environment running the business training task, while Area B represents the operational management environment for initiating monitoring and receiving results. The core value of this system lies in achieving transparent, one-stop observability collection of distributed deep learning training tasks through the sidecar pattern and non-intrusive automatic injection. Users can gain in-depth performance insights without modifying the training code.

[0195] Continue to refer to Figure 6 The illustration shows a schematic flow 600 of another embodiment of the fault diagnosis method according to the present disclosure. Flow 600 includes the following steps: Step 601: In response to the fact that the configuration file corresponding to the container group of the distributed system includes a plugin injection tag, the plugin packages of the logical layer plugin and the physical layer plugin are copied to the shared volume of the container group through the initialization container in the container group.

[0196] Step 602: Inject the relevant environment variables of the logic layer plugin and the physical layer plugin into the business container in the container group.

[0197] Step 603: Using a preset installation script, based on the plugin package in the shared volume, install the logical layer plugin and the physical layer plugin in the business container of the container group.

[0198] Step 604: During task execution, the status information of the coordination operations of the logical layer involved in the container group is collected through the logical layer plugin in the distributed system to obtain the logical layer sub-log.

[0199] Step 605: Obtain logical layer sub-logs from the shared volume through the companion container of the business container in the container group.

[0200] Step 606: Combine the logical layer sub-logs obtained from each companion container to obtain the logical layer logs.

[0201] Step 607: During task execution, the physical layer communication link status information of the node where the container group is located is collected through the physical layer plugin in the container group of the distributed system to obtain the physical layer sub-log.

[0202] Step 608: Obtain physical layer sub-logs from the shared volume through the companion container of the business container in the container group.

[0203] Step 609: Combine the physical layer sub-logs obtained by each companion container to obtain the physical layer logs.

[0204] Step 610: Using the large-scale artificial intelligence model, based on the logical layer logs, physical layer logs, multi-dimensional operational metrics, business logs during task execution, and communication logs from the collective communication library used by the distributed system, determine whether there are communication performance failures during task execution.

[0205] Step 611: In response to the confirmed existence, the deep tracing component performs attribution analysis of communication performance failure based on logical layer logs, physical layer logs, business logs during task execution, and communication logs of the collective communication library used by the distributed system, and determines the diagnostic results.

[0206] The fault diagnosis method process 600 in this embodiment, compared with the above process 200, specifically describes the injection process of logic layer plugins and physical layer plugins, the collection process of logic layer logs and physical layer logs, and the diagnosis process of communication performance faults.

[0207] Continue to refer to Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a fault diagnosis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0208] like Figure 7 As shown, the fault diagnosis device 700 includes: an acquisition unit 701, configured to acquire, during the task execution of a distributed system, a logic layer log representing the status information of the coordination operation of the logic layer and a physical layer log representing the status information of the communication link of the physical layer; a fault determination unit 702, configured to determine, through an artificial intelligence big data model, whether a communication performance fault exists during the task execution based on the logic layer log and the physical layer log; and a fault diagnosis unit 703, configured to, in response to the determination of the existence of the fault, perform attribution analysis of the communication performance fault based on the logic layer log and the physical layer log using a deep tracing component to determine the diagnosis result.

[0209] In some optional implementations of this embodiment, the acquisition unit 701 is further configured to: collect the status information of the coordination operation of the logical layer involved in the container group during the task execution through the logical layer plugin in the container group of the distributed system, and obtain the logical layer sub-log; and combine the logical layer sub-logs corresponding to each container group of the distributed system to obtain the logical layer log.

[0210] In some optional implementations of this embodiment, the acquisition unit 701 is further configured to: in response to the configuration file corresponding to the container group including a diagnostic type label for indicating the target fault type, determine the acquisition strategy corresponding to the diagnostic type label; and use the acquisition strategy to acquire the status information of the coordination operation of the logic layer involved in the container group during the task execution to obtain the logic layer sub-log.

[0211] In some optional implementations of this embodiment, the logical layer sub-logs are stored in the shared volume of the container group, and the acquisition unit 701 is further configured to: acquire the logical layer sub-logs from the shared volume through the companion container of the service container in the container group; and obtain the logical layer logs by combining the logical layer sub-logs acquired by each companion container.

[0212] In some optional implementations of this embodiment, the acquisition unit 701 is further configured to: collect the status information of the physical layer communication link of the node where the container group is located during the task execution through the physical layer plugin in the container group of the distributed system, and obtain the physical layer sub-log; combine the physical layer sub-logs corresponding to each container group of the distributed system to obtain the physical layer log.

[0213] In some optional implementations of this embodiment, the acquisition unit 701 is further configured to: in response to the configuration file corresponding to the container group including a diagnostic type label for indicating the target fault type, determine the acquisition strategy corresponding to the diagnostic type label; and use the acquisition strategy to acquire the status information of the physical layer communication link of the node where the container group is located during the task execution to obtain the logical layer sub-log.

[0214] In some optional implementations of this embodiment, the physical layer sub-logs are stored in the shared volume of the container group, and the acquisition unit 701 is further configured to: acquire the physical layer sub-logs from the shared volume through the companion container of the service container in the container group; and obtain the physical layer logs by combining the physical layer sub-logs acquired by each companion container.

[0215] In some optional implementations of this embodiment, the above apparatus further includes: a preprocessing unit (not shown in the figure), configured to: perform the following operations through the deep tracing component: in response to the configuration file corresponding to the container group of the distributed system including a plugin injection tag, copy the plugin packages of the logical layer plugin and the physical layer plugin to the shared volume of the container group through the initialization container in the container group; inject the relevant environment variables of the logical layer plugin and the physical layer plugin into the business container in the container group; and install the logical layer plugin and the physical layer plugin in the business container in the container group based on the plugin package in the shared volume using a preset installation script.

[0216] In some optional implementations of this embodiment, the above apparatus further includes: an indicator acquisition unit (not shown in the figure), configured to: acquire multi-dimensional operation indicators through the deep tracing component during the task execution of the distributed system; and a fault determination unit 702 further configured to: determine whether there is a communication performance fault during the task execution based on the logical layer log, the physical layer log, the multi-dimensional operation indicators, the business log during the task execution, and the communication log of the aggregated communication library used by the distributed system.

[0217] In some optional implementations of this embodiment, the fault determination unit 702 is further configured to: in response to the configuration file corresponding to the container group of the distributed system including a diagnostic type label for indicating the target fault type, determine whether there is a communication performance fault under the target fault type in the task execution process based on the logical layer log, the physical layer log, the multi-dimensional operation index, the business log and the communication log.

[0218] In some optional implementations of this embodiment, the fault determination unit 702 is further configured to: determine whether there is a communication performance fault under the target fault type in the task execution process based on the logical layer logs, physical layer logs, multi-dimensional operation indicators, business logs and communication logs of the distributed system within multiple time windows up to the present.

[0219] In some optional implementations of this embodiment, the fault diagnosis unit 703 is further configured to: perform attribution analysis of communication performance faults based on the logical layer logs, the physical layer logs, the business logs during the task execution process, and the communication logs of the collection communication library used by the distributed system, and determine the diagnosis result.

[0220] This embodiment provides a fault diagnosis device. The acquisition unit in the fault diagnosis device acquires logic layer logs representing the state information of the coordination operation of the logic layer and physical layer logs representing the state information of the communication link of the physical layer during the task execution of a distributed system. The fault determination unit uses an artificial intelligence big data model to determine whether a communication performance fault exists during task execution based on the logic layer logs and physical layer logs. In response to the determination of the fault's existence, the fault diagnosis unit uses a deep tracing component to perform attribution analysis of the communication performance fault based on the logic layer logs and physical layer logs to determine the diagnostic result. This provides a diagnostic method for communication performance faults in distributed systems. Based on the logic layer logs representing the state information of the coordination operation of the logic layer and the physical layer logs representing the state information of the communication link of the physical layer, combined with the fault existence determination of the big data model and the attribution analysis of the deep tracing component, automated diagnosis of communication performance faults is achieved, improving diagnostic efficiency and accuracy.

[0221] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an intelligent agent for fault diagnosis, which is similar to... Figure 2 The method embodiments shown correspond to specific deployments in various electronic devices.

[0222] like Figure 8As shown, the intelligent agent used for fault diagnosis includes: an input module 801, used to receive logical layer logs and physical layer logs during the task execution process of the distributed system, wherein the logical layer logs are used to characterize the status information of the coordination operation of the logical layer, and the physical layer logs characterize the status information of the communication link of the physical layer; a processing module 802, used to determine whether there is a communication performance fault during the task execution process based on the logical layer logs and physical layer logs through an artificial intelligence big data model; in response to the determination of the existence, the deep tracing component performs attribution analysis of the communication performance fault based on the logical layer logs and physical layer logs to determine the diagnostic result; and an output module 803, used to output the diagnostic result.

[0223] In some optional implementations of this embodiment, the logic layer logs are obtained as follows: during the task execution process, the status information of the coordination operation of the logic layer involved in the container group of the distributed system is collected by the logic layer plugin in the container group to obtain the logic layer sub-logs; the logic layer logs are obtained by combining the logic layer sub-logs corresponding to each container group of the distributed system.

[0224] In some optional implementations of this embodiment, the logic layer sub-log is obtained as follows: in response to the configuration file corresponding to the container group including a diagnostic type label for indicating the target fault type, a collection strategy corresponding to the diagnostic type label is determined; using the collection strategy, the status information of the coordination operation of the logic layer involved in the container group is collected during the task execution to obtain the logic layer sub-log.

[0225] In some optional implementations of this embodiment, the logical layer sub-logs are stored in the shared volume of the container group, and the logical layer logs are obtained in the following way: the logical layer sub-logs are obtained from the shared volume through the companion container of the business container in the container group; the logical layer logs are obtained by combining the logical layer sub-logs obtained by each companion container.

[0226] In some optional implementations of this embodiment, the physical layer logs are obtained as follows: the physical layer plugin in the container group of the distributed system collects the status information of the physical layer communication link of the node where the container group is located during the task execution to obtain the physical layer sub-logs; and the physical layer logs are obtained by combining the physical layer sub-logs corresponding to each container group of the distributed system.

[0227] In some optional implementations of this embodiment, the physical layer sub-log is obtained as follows: in response to the configuration file corresponding to the container group including a diagnostic type label for indicating the target fault type, a collection strategy corresponding to the diagnostic type label is determined; using the collection strategy, the status information of the physical layer communication link of the node where the container group is located is collected during the task execution to obtain the logical layer sub-log.

[0228] In some optional implementations of this embodiment, the physical layer sub-logs are stored in the shared volume of the container group, and the physical layer logs are obtained in the following way: the physical layer sub-logs are obtained from the shared volume through the companion container of the service container in the container group; the physical layer logs are obtained by combining the physical layer sub-logs obtained by each companion container.

[0229] In some optional implementations of this embodiment, the logic layer plugin and the physical layer plugin are deployed as follows: The deep tracing component performs the following operations: In response to the configuration file corresponding to the container group of the distributed system including a plugin injection tag, the plugin packages of the logic layer plugin and the physical layer plugin are copied to the shared volume of the container group through the initialization container in the container group; the relevant environment variables of the logic layer plugin and the physical layer plugin are injected into the business container in the container group; and the logic layer plugin and the physical layer plugin are installed in the business container in the container group using a preset installation script based on the plugin package in the shared volume.

[0230] In some optional implementations of this embodiment, the processing module 802 is further configured to: determine whether there is a communication performance failure during the task execution process based on the logical layer log, the physical layer log, the multi-dimensional operation indicators, the business log during the task execution process, and the communication log of the collection communication library used by the distributed system, wherein the multi-dimensional operation indicators are collected by the deep tracing component during the task execution process of the distributed system.

[0231] In some optional implementations of this embodiment, the processing module 802 is further configured to: in response to the configuration file corresponding to the container group of the distributed system including a diagnostic type label for indicating the target fault type, determine whether there is a communication performance fault under the target fault type in the task execution process based on the logical layer log, the physical layer log, the multi-dimensional operation index, the business log and the communication log.

[0232] In some optional implementations of this embodiment, the processing module 802 is further configured to: determine whether there is a communication performance failure under the target fault type in the task execution process based on the logical layer logs, physical layer logs, multi-dimensional operation indicators, business logs and communication logs of the distributed system within multiple time windows up to the present.

[0233] In some optional implementations of this embodiment, the processing module 802 is further configured to: perform attribution analysis of communication performance failures based on the logical layer logs, the physical layer logs, the business logs during the task execution process, and the communication logs of the collection communication library used by the distributed system, and determine the diagnostic results.

[0234] This embodiment provides an intelligent agent for fault diagnosis. The input module receives logical layer logs and physical layer logs from the task execution process of a distributed system. The logical layer logs represent the state information of the logical layer's coordination operations, and the physical layer logs represent the state information of the physical layer's communication links. The processing module uses a large-scale artificial intelligence model to determine whether a communication performance fault exists during task execution based on the logical layer logs and physical layer logs. In response to the determination of a fault, a deep tracing component performs attribution analysis on the communication performance fault based on the logical layer logs and physical layer logs to determine the diagnostic result. The output module outputs the diagnostic result. This provides an intelligent agent for diagnosing communication performance faults in distributed systems. By combining logical layer logs representing the state information of the logical layer's coordination operations and physical layer logs representing the state information of the physical layer's communication links with the fault existence determination of a large-scale model and the attribution analysis of a deep tracing component, automated diagnosis of communication performance faults is achieved, improving diagnostic efficiency and accuracy.

[0235] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the fault diagnosis method described in any of the above embodiments when executed.

[0236] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the fault diagnosis method described in any of the above embodiments when executed.

[0237] This disclosure provides a computer program product that, when executed by a processor, can implement the fault diagnosis method described in any of the above embodiments.

[0238] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0239] like Figure 9 As shown, device 900 includes a processor 901, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0240] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0241] Processor 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 901 performs the various methods and processes described above, such as fault diagnosis methods. For example, in some embodiments, the fault diagnosis method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by processor 901, one or more steps of the fault diagnosis method described above may be performed. Alternatively, in other embodiments, processor 901 may be configured to perform fault diagnosis methods by any other suitable means (e.g., by means of firmware).

[0242] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0243] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable fault diagnosis device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0244] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0245] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0246] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0247] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers integrated with blockchain technology.

[0248] According to the technical solution of the embodiments of this disclosure, a fault diagnosis method and apparatus are provided. During the task execution process of a distributed system, logical layer logs representing the state information of the coordination operation of the logical layer and physical layer logs representing the state information of the communication link of the physical layer are acquired. Using an artificial intelligence big data model, the existence of a communication performance fault during task execution is determined based on the logical layer logs and physical layer logs. In response to the determination of the fault's existence, a deep tracing component performs attribution analysis on the communication performance fault based on the logical layer logs and physical layer logs to determine the diagnostic result. This provides a diagnostic method for communication performance faults in distributed systems. Based on the logical layer logs representing the state information of the coordination operation of the logical layer and the physical layer logs representing the state information of the communication link of the physical layer, combined with the fault existence determination of the big data model and the attribution analysis of the deep tracing component, automated diagnosis of communication performance faults is achieved, improving diagnostic efficiency and accuracy.

[0249] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0250] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A fault diagnosis method, comprising: During the task execution process of a distributed system, logical layer logs representing the status information of the coordination operation of the logical layer and physical layer logs representing the status information of the communication link of the physical layer are obtained. Using a large-scale artificial intelligence model, the system determines whether there are communication performance failures during task execution based on the logical layer logs and the physical layer logs. In response to the determination of existence, the deep tracing component performs attribution analysis on the communication performance failure based on the logical layer logs and the physical layer logs to determine the diagnostic results.

2. The method according to claim 1, wherein, The logical layer log, which acquires status information representing the coordination operations of the logical layer during the task execution process of the distributed system, includes: By using the logic layer plugin in the container group of the distributed system, the status information of the coordination operation of the logic layer involved in the container group is collected during the task execution, and the logic layer sub-log is obtained. The logical layer log is obtained by combining the logical layer sub-logs corresponding to each container group in the distributed system.

3. The method according to claim 2, wherein, The process of collecting status information of the coordination operations of the logical layer involved in the container group during the execution of the task yields logical layer sub-logs, including: In response to the fact that the configuration file corresponding to the container group includes a diagnostic type label for indicating the target fault type, the collection strategy corresponding to the diagnostic type label is determined. Using the aforementioned collection strategy, the status information of the coordination operations of the logical layer involved in the container group is collected during the task execution to obtain the logical layer sub-log.

4. The method according to claim 2, wherein, The logical layer sub-logs are stored in the shared volume of the container group, and The logical layer logs are obtained by combining the logical layer sub-logs corresponding to each container group in the distributed system, including: The logical layer sub-log is obtained from the shared volume through the companion container of the business container in the container group; The logical layer log is obtained by combining the logical layer sub-logs obtained from each of the accompanying containers.

5. The method according to claim 1, wherein, During the execution of tasks in a distributed system, physical layer logs, which acquire state information representing the communication links at the physical layer, are included: By using the physical layer plugin in the container group of the distributed system, the status information of the physical layer communication link of the node where the container group is located is collected during the execution of the task, and physical layer sub-logs are obtained. The physical layer logs are obtained by combining the physical layer sub-logs corresponding to each container group in the distributed system.

6. The method according to claim 5, wherein, The process of collecting the status information of the physical layer communication link of the node where the container group is located during the execution of the task, and obtaining the physical layer sub-log, includes: In response to the fact that the configuration file corresponding to the container group includes a diagnostic type label for indicating the target fault type, the collection strategy corresponding to the diagnostic type label is determined. Using the aforementioned acquisition strategy, the status information of the physical layer communication link of the node where the container group is located is acquired during the task execution to obtain the logical layer sub-log.

7. The method according to claim 5, wherein, The physical layer sub-logs are stored in the shared volume of the container group, and The physical layer logs are obtained by combining the physical layer sub-logs corresponding to each container group in the distributed system, including: The physical layer sub-logs are obtained from the shared volume through the companion container of the service container in the container group; The physical layer log is obtained by combining the physical layer sub-logs obtained from each of the accompanying containers.

8. The method according to any one of claims 1-7, wherein, Before acquiring the logical layer log representing the state information of the coordination operation of the logical layer and the physical layer log representing the state information of the communication link of the physical layer during the task execution process of the distributed system, the method further includes: The following operations are performed using the depth tracking component: In response to the fact that the configuration file corresponding to the container group of the distributed system includes a plugin injection tag, the plugin packages of the logical layer plugin and the physical layer plugin are copied to the shared volume of the container group through the initialization container in the container group; Inject the relevant environment variables of the logic layer plugin and the physical layer plugin into the business container in the container group; Using a preset installation script, the logical layer plugin and the physical layer plugin are installed in the business container of the container group based on the plugin package in the shared volume.

9. The method according to claim 1, wherein, Also includes: During the task execution of the distributed system, multi-dimensional operational metrics are collected through the deep tracing component. as well as The step of determining whether a communication performance failure exists during task execution based on the logical layer log and the physical layer log includes: Based on the logical layer logs, the physical layer logs, the multi-dimensional operating metrics, the business logs during the task execution process, and the communication logs of the aggregated communication library used by the distributed system, it is determined whether there are communication performance failures during the task execution process.

10. The method according to claim 9, wherein, The step of determining whether there is a communication performance failure during the task execution process based on the logical layer logs, the physical layer logs, the multi-dimensional operation metrics, the business logs during the task execution process, and the communication logs of the aggregated communication library used by the distributed system includes: In response to the configuration file corresponding to the container group of the distributed system including a diagnostic type label for indicating the target fault type, the system determines whether there is a communication performance fault under the target fault type during the task execution process based on the logical layer log, the physical layer log, the multi-dimensional operation indicators, the business log, and the communication log.

11. The method of claim 10, wherein, The step of determining whether a communication performance fault exists under the target fault type during the task execution process based on the logical layer log, the physical layer log, the multi-dimensional operation indicators, the business log, and the communication log includes: Based on the logical layer logs, physical layer logs, multi-dimensional operating metrics, business logs, and communication logs of the distributed system within multiple time windows up to the present, determine whether there is a communication performance failure under the target fault type during the task execution process.

12. The method according to claim 1, wherein, The process of performing attribution analysis on communication performance failures using a deep tracing component based on the logical layer logs and the physical layer logs to determine the diagnostic results includes: Using the deep tracing component, attribution analysis of communication performance failures is performed based on the logical layer logs, the physical layer logs, the business logs during task execution, and the communication logs of the collective communication library used by the distributed system, to determine the diagnostic results.

13. A fault diagnosis device, comprising: The acquisition unit is configured to acquire, during the task execution process of the distributed system, logical layer logs representing the status information of the coordination operation of the logical layer and physical layer logs representing the status information of the communication link of the physical layer. The fault determination unit is configured to determine whether there is a communication performance fault during task execution based on the logical layer logs and the physical layer logs using an artificial intelligence big data model. The fault diagnosis unit is configured to determine the diagnosis result in response to the determination of the existence of a communication performance fault by performing attribution analysis based on the logical layer log and the physical layer log through a deep tracing component.

14. An intelligent agent for fault diagnosis, comprising: The input module is used to receive logical layer logs and physical layer logs during the task execution process of the distributed system. The logical layer logs are used to represent the status information of the coordination operation of the logical layer, and the physical layer logs represent the status information of the communication link of the physical layer. The processing module is used to determine whether there is a communication performance failure during the task execution process based on the logical layer logs and the physical layer logs using an artificial intelligence big data model; in response to the determination that there is a failure, the deep tracing component performs attribution analysis of the communication performance failure based on the logical layer logs and the physical layer logs to determine the diagnostic result. The output module is used to output the diagnostic results.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, Computer instructions are used to cause a computer to perform the method of any one of claims 1-12.

17. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images