Congestion detection methods, devices, equipment and storage media

By constructing a training semantic temporal graph and a probabilistic semantic network envelope, the problem that existing congestion detection methods have difficulty distinguishing between malicious congestion and benign anomalies is solved, and more accurate congestion detection is achieved.

CN122870702BActive Publication Date: 2026-11-17PENG CHENG LAB
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611318148.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-28
Publication Date
2026-11-17
Estimated Expiration
2046-08-28

AI Technical Summary

Technical Problem

Existing congestion detection methods struggle to distinguish between malicious congestion and benign anomalies during training alignment, leading to a high risk of missed and false alarms.

Method used

By acquiring training semantic data and network telemetry data, a training semantic time series graph is constructed to determine the probabilistic semantic network envelope of network observation indicators in the network telemetry data under different semantic conditions, and malicious congestion events are detected based on the probabilistic semantic network envelope.

Benefits of technology

It enables accurate differentiation between malicious congestion and benign anomalies without relying on fixed thresholds, reducing the risk of missed and false alarms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122870702B_ABST
    Figure CN122870702B_ABST
Patent Text Reader

Abstract

This application discloses a congestion detection method, apparatus, device, and storage medium, relating to the field of network communication technology. The method includes: acquiring training semantic data and network telemetry data for the current distributed training task; constructing a training semantic time sequence graph for the current distributed training task based on the training semantic data; determining the probabilistic semantic network envelopes corresponding to network observation indicators in the network telemetry data under different semantic conditions based on the training semantic time sequence graph; and detecting whether malicious congestion events exist in each training step based on the probabilistic semantic network envelopes to obtain network congestion detection results. This application addresses the technical problem of existing congestion detection methods struggling to distinguish between training-aligned malicious congestion and benign anomalies, leading to a high risk of false negatives and false positives. By establishing probabilistic semantic network envelopes to describe the normal value range of each network indicator instead of fixed thresholds, the determination of malicious congestion from network anomaly detection no longer relies on fixed thresholds.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of network communication technology, and in particular to congestion detection methods, apparatus, devices and storage media. Background Technology

[0002] As the scale of Large Language Models (LLMs) continues to grow, distributed training systems widely employ Remote Direct Memory Access (RDMA) networks to achieve high-throughput, low-latency communication between multi-GPU servers. Existing RDMA network congestion handling techniques mainly rely on congestion control methods based on network signals, anomaly detection methods based on traffic statistics, and congestion localization methods based on paths or topology. However, these methods are typically analyzed from the perspective of network metrics or single nodes. In large model collaborative training, ensemble communication (such as All-Reduce and All-to-All) is not completed independently by a single node, but rather by multiple training ranks co-executing on their respective hosts and network paths. Under this mechanism, communication delays of a few ranks can propagate to all participating ranks through ensemble communication synchronization barriers, amplifying local network anomalies into an increase in the overall training step's time consumption.

[0003] In wide-area RDMA collaborative training scenarios, a new and difficult-to-counter hidden threat exists: "malicious congestion for training alignment." For this type of attack, attackers do not need to continuously send large amounts of traffic; they only need to inject short bursts of congestion traffic with low average rates within a critical set communication window. This can cause a few training ranks to become trailing ranks, and then, by leveraging the set communication synchronization barrier, amplify local latency into an increase in the overall training step's time consumption. Because the average traffic rate of this type of attack is low, its instantaneous queue fluctuations may resemble normal communication convergence or benign anomalies. This makes it difficult for traditional congestion detection methods based on fixed thresholds to distinguish this malicious behavior from normal training fluctuations, GPU computation tailing, link jitter, multi-tenant contention, and other benign anomalies, resulting in a high risk of false negatives and false negatives. Summary of the Invention

[0004] The main objective of this application is to provide a congestion detection method, apparatus, device, and storage medium, aiming to solve the technical problem that existing congestion detection methods are unable to distinguish between malicious congestion and benign anomalies during training alignment, resulting in a high risk of missed and false alarms.

[0005] To achieve the above objectives, this application proposes a congestion detection method, the method comprising: Acquire training semantic data and network telemetry data for the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task. The network telemetry data is used to describe the network state during the operation of the current distributed training task. Based on the training semantic data, a training semantic time sequence diagram of the current distributed training task is constructed. The training semantic time sequence diagram is used to describe the relationship between the computation events, the collective communication events and the network resources in each training step of the current distributed training task. Based on the training semantic time sequence diagram, the probabilistic semantic network envelopes corresponding to different semantic conditions of the network observation indicators in the network telemetry data are determined. The probabilistic semantic network envelopes are used to describe the normal value range of the network observation indicators under each semantic condition. Based on the probabilistic semantic network envelope detection, the network congestion detection result of the current distributed training task is obtained to determine whether there is a malicious congestion event in each training step.

[0006] In one embodiment, the step of constructing the training semantic time sequence graph of the current distributed training task based on the training semantic data includes: Based on the training semantic data, determine the training computation event node, set communication event node, and collaboration node corresponding to each training node in each training step of the current distributed training task; Determine the dependencies between the computation events corresponding to the training computation event nodes, the set communication events corresponding to the set communication event nodes, and the collaborative events corresponding to the collaborative nodes; Based on the aforementioned dependency relationships and preset dependency edge establishment rules, directed dependency edges are established between the training computation event nodes, the set communication event nodes, and the collaborative nodes. The training semantic time sequence graph of the current distributed training task is constructed based on the training computation event nodes, the set communication event nodes, the collaborative nodes, and the directed dependency edges.

[0007] In one embodiment, the step of constructing the training semantic time sequence graph of the current distributed training task based on the training computation event node, the set communication event node, the collaborative node, and the directed dependency edge includes: Determine the start time and completion time of each of the aforementioned collective communication events; Based on the event start time and the event completion time, each of the set communication events is expanded into several rank-level communication sub-events that are executed by the respective rank; Based on the candidate path distribution, a weighted mapping edge is established between each rank-level communication sub-event and the candidate path segment; Determine the shared bottleneck probability between each pair of the computational event, the rank-level communication sub-event, and the cooperative event; The training semantic time sequence graph of the current distributed training task is constructed based on the training computation event nodes, the rank communication sub-events, the collaborative nodes, the directed dependency edges, the weighted mapping edges, and the shared bottleneck probabilities.

[0008] In one embodiment, the step of determining the probabilistic semantic network envelope corresponding to different semantic conditions of the network observation index in the network telemetry data based on the training semantic time sequence graph includes: The semantic context corresponding to each set of communication events in the current distributed training task is determined based on the training semantic sequence diagram. The semantic context includes at least one of the following: training location information, communication attributes, participating resources, candidate path distribution, message structure, and communication criticality. Based on the preset index condition distribution and the semantic context, the network observation indexes in the network telemetry data are divided into several envelope buckets; Based on the historical sample set, determine the normal value range of the network observation index in each envelope bucket under the corresponding semantic conditions, and define the normal value range as the probabilistic semantic network envelope.

[0009] In one embodiment, the step of detecting whether a malicious congestion event exists in each training step based on the probabilistic semantic network envelope includes: The envelope violation score of each training step is determined based on the degree of deviation between the real-time network observation index in each training step and the probabilistic semantic network envelope under the corresponding semantic conditions. If the envelope violation score exceeds a first preset threshold, it is determined that a network congestion event exists in the corresponding training step; Determine the phase lock score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event; The malicious congestion score corresponding to the network congestion event is determined based on the envelope violation score, the phase lock score, the causal order score, the path overlap score, and the attack amplification score. If the malicious congestion score exceeds a second preset threshold, a malicious congestion event is determined to exist in the training step.

[0010] In one embodiment, after the step of detecting whether malicious congestion events exist in each training step based on the probabilistic semantic network envelope and obtaining the network congestion detection result of the current distributed training task, the method further includes: Determine the positive evidence score and negative evidence score of each candidate source in the candidate source set of the current distributed training task. The positive evidence score includes the target stage locking score, target path overlap score, rank skew persistence score, target attack amplification score, and target causal order score. The negative evidence score includes benign interpretation strength and uncertainty score. The malicious semantic congestion score of each candidate source is determined based on the positive evidence score and the negative evidence score. If the malicious semantic congestion score exceeds a preset malicious threshold, the candidate source is determined to be the malicious congestion event.

[0011] In one embodiment, after determining that the candidate source is the malicious congestion event, the method further includes: The victimized local communication sub-event nodes affected by the malicious congestion event are determined from the training semantic temporal graph, and the distribution of candidate path segments that have a mapping relationship with the victimized local communication sub-event nodes is obtained. The path segment evidence score corresponding to each candidate path segment is determined based on the congestion evidence strength on each candidate path segment in the candidate path segment distribution. Based on the target sharing bottleneck probability between each candidate source and the victim local communication sub-event corresponding to the victim local communication sub-event node, the burst traffic time location corresponding to each candidate source, and the repeated similar anomaly records of each candidate source in the historical training steps, an attribution confidence score is generated. The malicious congestion location result of the current distributed training task is generated based on the attribution confidence.

[0012] Furthermore, to achieve the above objectives, this application also proposes a congestion detection device, the device comprising: The data acquisition module is used to acquire training semantic data and network telemetry data of the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task, and the network telemetry data is used to describe the network state during the operation of the current distributed training task. The timing graph construction module is used to construct a training semantic timing graph of the current distributed training task based on the training semantic data. The training semantic timing graph is used to describe the relationship between the computation events, the collective communication events and network resources in each training step of the current distributed training task. An envelope determination module is used to determine the probabilistic semantic network envelope of the network observation index in the network telemetry data under different semantic conditions based on the training semantic time sequence diagram. The probabilistic semantic network envelope is used to describe the normal value range of the network observation index under each semantic condition. The congestion detection module is used to detect whether there are malicious congestion events in each training step based on the probabilistic semantic network envelope, and to obtain the network congestion detection result of the current distributed training task.

[0013] In addition, to achieve the above objectives, this application also proposes a congestion detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the congestion detection method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the congestion detection method described above.

[0015] This application provides a congestion detection method. It discloses acquiring training semantic data and network telemetry data for the current distributed training task. The training semantic data characterizes computational events and aggregated communication events corresponding to multiple training nodes in the current distributed training task, while the network telemetry data describes the network state during the operation of the current distributed training task. A training semantic time sequence graph for the current distributed training task is constructed based on the training semantic data. This graph describes the relationships between computational events, aggregated communication events, and network resources in each training step of the current distributed training task. Based on the training semantic time sequence graph, the probabilistic semantic network corresponding to network observation indicators in the network telemetry data under different semantic conditions is determined. The probabilistic semantic network envelope is used to describe the normal value range of network observation indicators under various semantic conditions. Based on the probabilistic semantic network envelope, the network congestion detection result of the current distributed training task is obtained by detecting whether there are malicious congestion events in each training step. Since this application can establish probabilistic semantic network envelopes for different semantic conditions based on training semantic time sequence graphs and network telemetry data to describe the normal value range of each network indicator, it can replace fixed thresholds for anomaly judgment. This makes the entire process from network anomaly detection to malicious congestion judgment no longer dependent on fixed thresholds, thus solving the technical problem that existing congestion detection methods are difficult to distinguish between training-aligned malicious congestion and benign anomalies, resulting in a high risk of false negatives and false positives. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the congestion detection method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the congestion detection method of this application; Figure 3 This is a flowchart illustrating Embodiment 3 of the congestion detection method of this application; Figure 4 This is a flowchart illustrating the reverse congestion source tracing and reverse tracking direction in the congestion detection method of this application; Figure 5 This is a flowchart illustrating the overall congestion detection method of this application; Figure 6 A system block diagram provided for the congestion detection method of this application; Figure 7 This is a schematic diagram of the module structure of the congestion detection device according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the congestion detection method in this application embodiment.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is as follows: First, obtain training semantic data and network telemetry data for the current distributed training task. The training semantic data characterizes computational events and aggregated communication events corresponding to multiple training nodes in the current distributed training task, while the network telemetry data describes the network state during the operation of the current distributed training task. Second, construct a training semantic time sequence diagram for the current distributed training task based on the training semantic data. This diagram describes the relationships between computational events, aggregated communication events, and network resources in each training step of the current distributed training task. Third, determine the probabilistic semantic network envelopes corresponding to network observation indicators in the network telemetry data under different semantic conditions based on the training semantic time sequence diagram. These envelopes describe the normal value range of the network observation indicators under each semantic condition. Fourth, detect whether malicious congestion events exist in each training step based on the probabilistic semantic network envelopes, thereby obtaining the network congestion detection results for the current distributed training task.

[0023] Existing congestion detection methods based on fixed thresholds are difficult to distinguish between malicious attacks and normal training fluctuations, GPU computation tailing, link jitter, multi-tenant contention, and other benign anomalies, resulting in a high risk of false negatives and false negatives.

[0024] This application provides a solution that can establish probabilistic semantic network envelopes based on training semantic time sequence graphs and network telemetry data for different semantic conditions to describe the normal value range of each network indicator, in order to replace fixed thresholds for anomaly judgment. This makes the entire process from network anomaly detection to malicious congestion judgment no longer dependent on fixed thresholds, thereby solving the technical problem that existing congestion detection methods have difficulty distinguishing between training-aligned malicious congestion and benign anomalies, leading to a high risk of false negatives and false positives.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or congestion detection device capable of performing the above functions. The following description uses a congestion detection device (hereinafter referred to as the device) as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide a congestion detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the congestion detection method of this application.

[0027] In this embodiment, the congestion detection method includes steps S10 to S40: Step S10: Obtain the training semantic data and network telemetry data of the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task. The network telemetry data is used to describe the network state during the operation of the current distributed training task.

[0028] It should be noted that the current distributed training task can be a running computational task that requires multiple training nodes to collaboratively complete model training, such as a task of training a large language model based on a data parallel strategy, or a task of cross-node collaborative training based on a Mixture-of-Experts (MoE) architecture. This embodiment does not impose any limitations on this. In practical applications, the current distributed training task typically includes multiple training steps, and each training node in each training step can perform operations such as forward computation, backward computation, gradient synchronization, and parameter updates.

[0029] It should also be noted that current distributed training tasks can utilize remote direct memory access networks to achieve high-throughput, low-latency communication between multiple nodes, and employ aggregate communication operations (such as All-Reduce, All-to-All, etc.) to achieve gradient synchronization or parameter exchange across nodes. For example, a large language model training job containing 64 GPU servers, each equipped with 8 GPUs, with a total of 512 training ranks collaboratively updating the model parameters, constitutes a distributed training task.

[0030] Understandably, a training node can be an independent computing unit participating in the current distributed training task. At the physical level, it can be a Graphics Processing Unit (GPU), a Central Processing Unit (CPU) core, or a computing device containing multiple computing chips; at the logical level, it can be a training process, a training thread, or a training rank. In practical applications, a current distributed training task typically includes multiple training nodes, which can collaboratively execute computational tasks and perform collective communication tasks. For example, in data-parallel training, each GPU corresponds to a training rank, and the GPU, its host, and network interface card together constitute a training node. Furthermore, each training node has a unique node identifier or rank identifier used to distinguish different participants in collective communication.

[0031] It should be understood that training semantic data can be data used to describe the behavioral characteristics generated by each training node during computation and communication during the execution of a distributed training task. This data can reflect the execution progress and event sequence of the training task at the logical level. In this embodiment, training semantic data can come from at least one of the following: training framework (such as PyTorch), ensemble communication library (such as NCCL), performance analyzer (such as PyTorch Profiler), host proxy, or eBPF (such as extended Berkeley packet filtering) events. It can include, but is not limited to, training step number (used to identify which iteration in the current training task), training stage identifier (such as forward propagation, back propagation, gradient synchronization, optimizer update), ensemble communication operation type (such as All-Reduce, All-Gather, All-to-All), communication algorithm type (such as Ring algorithm, Tree algorithm), participation rank set (i.e., the identifier set of all training ranks participating in the current ensemble communication operation), message size, event start timestamp, event completion timestamp, checkpoint status, expert routing load, etc.

[0032] Understandably, network telemetry data can be data collected from network devices (such as switches and network interface cards) and the host side to reflect the real-time status of the network during the current distributed training task. It can include, but is not limited to, switch port queue depth (i.e., the number of data packets currently queued for forwarding), queue growth slope (i.e., the rate at which queue depth changes over time), queue peak duration, ECN (Explicit Congestion Notification) marking ratio (i.e., the proportion of data packets marked as congested to all data packets), number and growth rate of ECN markings, number and rate of CNP (Congestion Notification Packets), number and total duration of PFC (Priority-based Flow Control) pause frames, link utilization, network interface card congestion count, RDMA queue pair statistics, host-side burst data volume, traffic matrix, candidate path set, link error count, routing change records, and port status change records, etc. For example, a network telemetry data record can show that: the queue depth of the switch port at time t is 150 packets, the ECN tagging ratio is 3%, 5 CNP messages were received in the last second, and the cumulative duration of PFC pause frames is 2 milliseconds.

[0033] It should be understood that computational events can be computational operations executed independently on a training node without relying on cross-node data transfer, such as forward propagation computation, backpropagation computation, gradient calculation, optimizer parameter updates, expert calculations, and data loading. In practical applications, the execution scope of computational events is limited to within a single training node and can be completed independently without waiting for data from other training nodes. In other words, computational events do not involve cross-node data transfer and can be executed solely based on local data.

[0034] It should also be understood that a collective communication event can be a record of data transmission and synchronization operations collaboratively completed between multiple training nodes, such as All-Reduce, All-Gather, All-to-All, Reduce-Scatter, and Broadcast operations. In practical applications, a collective communication event is usually defined by the communication operation type, communication algorithm type, participant rank set, message size, and communication protocol. A logical collective communication event (e.g., "All-Reduce operation of the data parallel group in the 100th training step") is composed of local communication sub-events executed by multiple training nodes. The logical collective communication event is only considered complete when all local communication sub-events of all participating nodes have been completed. Furthermore, a collective communication event is not completed independently by a single node, but rather jointly completed by multiple training ranks on different hosts, network cards, RDMA queue pairs, and network paths.

[0035] Understandably, network status can be the real-time condition of the network during the execution of a distributed training task. This status can be quantified through various network metrics, including but not limited to network congestion levels, transmission quality, link health, and resource utilization. In practical applications, network status can be specifically represented by various metrics in network telemetry data. For example, switch port queue depth reflects network congestion levels, ECN tagging ratio reflects network congestion trends, CNP packet count reflects congestion feedback perceived by the receiver, PFC pause frame count reflects link-level flow control status, link error count reflects link health, and link utilization reflects bandwidth utilization, etc.

[0036] In its implementation, the device can read the training step number, computation event records of each training node, and ensemble communication event records from the training log of the training framework. It can also read the ensemble communication operation type, communication algorithm, participating rank set, message size, RDMA queue pair number, and start and end times of each rank communication sub-event from the ensemble communication library's tracking log. Furthermore, it can read the timeline information of forward propagation, backward propagation, gradient synchronization, and parameter updates from the performance analyzer. Simultaneously, it can read the host-side bursts, traffic matrix, and RDMA queue pair statistics from the host agent or eBPF events. By integrating all this information, the device can obtain the training semantic data for the current distributed training task. Simultaneously, the device can read the queue depth, ECN tag ratio, CNP packet count, PFC pause frame count, and link utilization of each switch port through the switch's management interface or telemetry protocols (such as INT, P4). It can also read the RDMA rate, retransmission count, timeout count, congestion count, and queue pair statistics from the network card's driver or firmware interface. Furthermore, it can read the host-side burst data, peak rate, packet length distribution, and traffic matrix from the host agent, and link error counts, routing change records, port status change records, device anomaly records, and cross-domain link status information from the wide-area link monitoring system. All of this information is used as the network telemetry data for the current distributed training task. The network telemetry data can be collected in real-time (e.g., once every millisecond) or periodically (e.g., once every 10 milliseconds). This embodiment does not limit the collection method or frequency.

[0037] Step S20: Construct a training semantic time sequence diagram of the current distributed training task based on the training semantic data. The training semantic time sequence diagram is used to describe the relationship between the computation events, the aggregate communication events and the network resources in each training step of the current distributed training task.

[0038] It should be noted that the training semantic temporal graph can be a data structure used to describe the temporal relationships, dependencies, and resource mapping relationships between various events in the training task. In this embodiment, the training semantic temporal graph can be expressed in the form of a graph structure, including a set of nodes, a set of edges, and a set of attributes. The set of nodes can include computational event nodes, set communication event nodes, cooperative nodes, and network resource nodes. The set of edges can include dependency edges between computational events and local communication sub-events, convergence edges between local communication sub-events and logical set communication events, mapping edges between communication sub-events and network resources, topological connection edges between network resources, and congestion propagation edges between candidate source bursts, queue growth, congestion signals, rank tails, and synchronization waits. The set of attributes is used to record additional information about nodes and edges, such as event time windows, dependency types, and mapping probability weights.

[0039] It should also be noted that in the training semantic timing graph, computation event nodes are used to represent computational operations on a single training node (such as forward computation and backward computation); collective communication event nodes are used to represent a logical collective communication event or multiple local communication sub-events corresponding to training ranks; cooperative nodes are used to represent cooperative operations such as gradient readiness, parameter readiness, expert routing completion, and synchronization barriers; and network resource nodes are used to represent network hardware or logical resources such as hosts, network interface cards, RDMA queue pairs, switch ports, switch queues, and wide area link segments. In this embodiment, the directed edges in the training semantic timing graph can be used to describe the relationships between computation events, collective communication events, and network resources in each training step of the current distributed training task. These relationships can include the temporal dependency between computation events and collective communication events (such as collective communication that triggers gradient synchronization only after backward computation is completed), the convergence relationship between local communication sub-events on multiple training nodes and logical collective communication events (such as all 64 local All-Reduce sub-events converging into a single logical All-Reduce event), and the mapping relationship between collective communication events and network resources (such as which RDMA queue pair a local communication sub-event uses and which candidate path segment it traverses). This embodiment constructs a training semantic time sequence graph, which can accurately pinpoint the abnormal fluctuations of a network queue to specific training steps, specific aggregate communication events, and specific training nodes, thereby facilitating subsequent malicious congestion detection.

[0040] Understandably, a training step can be a complete iteration in a distributed training task. In a single training step, the training task can process one or more micro-batch data, including stages such as forward propagation, backpropagation, gradient synchronization, and parameter updates, ultimately updating the model parameters. In practical applications, current distributed training tasks typically contain multiple training steps, with all training nodes in each step executing their respective computation and communication tasks in parallel. Furthermore, each training step has a globally unique identifier (i.e., a step number), such as the 100th training step, the 1000th training step, etc.

[0041] It is also understood that network resources can be the hardware and logical resources involved in data communication in a distributed training task. These can include, but are not limited to, hosts (i.e., the physical servers where training nodes reside), network interface cards (NICs) (i.e., network interface controllers on hosts), RDMA queue pairs (i.e., logical queues on NICs used to manage RDMA communication, including send and receive queues), switches (i.e., forwarding devices in the network), switch ports (i.e., physical interfaces on switches), switch queues (i.e., logical queues in switch ports used to buffer data packets to be forwarded), and wide area link segments (i.e., network links spanning data centers or regions). In this embodiment, network resources can be a node type in the training semantic time sequence graph. The training semantic time sequence graph can associate a set of communication events with the specific network resources they use through mapping edges, thereby realizing a complete resource link description of "communication event → RDMA queue pair → NIC → host → switch port → wide area link segment".

[0042] In its implementation, the device can construct a training semantic time sequence graph for the current distributed training task based on the collected training semantic data. During this construction process, the device categorizes the computational events and collective communication events recorded in the training semantic data according to the training steps in which they occur, and determines the training computational event nodes, collective communication event nodes, and collaborative nodes corresponding to each training node in each training step, as well as the directed dependencies between these nodes. Then, the device can establish directed dependency edges between these event nodes according to preset dependency edge establishment rules, and simultaneously establish mapping edges between collective communication event nodes and network resource nodes based on network topology and routing information. Finally, the device can integrate the nodes and edges into a training semantic time sequence graph for the current distributed training task, describing the temporal order, logical dependencies, and resource mapping relationships between computational events, collective communication events, and network resources in each training step.

[0043] Step S30: Based on the training semantic time sequence diagram, determine the probabilistic semantic network envelope corresponding to the network observation index in the network telemetry data under different semantic conditions. The probabilistic semantic network envelope is used to describe the normal value range of the network observation index under each semantic condition.

[0044] It should be noted that network observation metrics can be quantifiable parameters extracted from network telemetry data to measure network operation status. These may include, but are not limited to, maximum port queue depth, queue growth slope, peak queue duration, ECN tag ratio, ECN tag growth rate, CNP packet count, CNP packet rate, CNP packet direction, CNP first occurrence time, PFC pause frame count, total PFC pause duration, maximum PFC pause duration, PFC propagation range, queue-level traffic matrix, number of bytes sent on the host side, set communication duration, rank completion time skew, GPU waiting time or idle time, etc.

[0045] Understandably, different semantic conditions can be the training context of various ensemble communication events in a distributed training task. These can include, but are not limited to, training phases (such as forward propagation, back propagation, gradient synchronization, parameter prefetching, optimizer update, and checkpoint saving), communication operation types (such as All-Reduce, All-Gather, and All-to-All), communication algorithms (such as Ring, Tree, and Recursive Doubling), participation rank groups (such as data parallel groups, tensor parallel groups, pipeline parallel groups, and expert parallel groups), message size categories (small messages, medium messages, large messages, and very large messages), topological regions (same machine, same rack, cross rack, cross cluster, and cross data center), model layer numbers, micro-batch numbers, and parallel group types. In this embodiment, different ensemble communication events may be under different semantic conditions. Semantic conditions are used to divide network observation metrics into different semantic buckets, with each semantic bucket corresponding to a set of network observation metric samples under a specific combination of semantic conditions. For example, the semantic conditions “All-Reduce + Ring algorithm + 64 rank + 256MB message + cross rack + backpropagation” constitute a specific semantic bucket. Under this semantic bucket, the normal range of port queue depth may be significantly different from the normal range of the semantic conditions “All-to-All + 32 rank + 1MB message + same machine + forward propagation”.

[0046] It should be noted that the probabilistic semantic network envelope can be a data model used to describe the distribution characteristics and normal value range of network observation indicators under given semantic conditions. The normal value range for each semantic condition can be the statistical distribution interval of the network observation indicator under each semantic condition during historical normal operation. This range can be established based on sample data from normal operation periods and employs an anti-pollution update strategy to avoid contamination by attacked samples.

[0047] In practical applications, the probabilistic semantic network envelope can be established on training samples with the same or similar semantic conditions by statistically analyzing historical network observation indicators. Specific forms include, but are not limited to: median and median absolute deviation envelope, quantile envelope, mean and standard deviation envelope, etc. Specifically, the median and median absolute deviation envelope can be established using the median of historical samples as the center and the median absolute deviation as the measure of dispersion, with a normal range of values ​​of [median - k × median absolute deviation, median + k × median absolute deviation], where k is an adjustable coefficient. The quantile envelope can be established using the lower quantile (e.g., 5th percentile) of historical samples as the lower boundary and the higher quantile (e.g., 95th percentile) as the upper boundary, with a normal range of values ​​of [lower quantile, higher quantile]. The mean and standard deviation envelope can be established using the mean of historical samples as the center and the standard deviation as the measure of dispersion, with a normal range of values ​​of [mean - k × standard deviation, mean + k × standard deviation].

[0048] It should also be noted that, compared to traditional fixed-threshold detection methods, the probabilistic semantic network envelope is dynamic and semantically aware. That is, under the same semantic condition, the envelope defines the normal range under that semantic condition; under different semantic conditions, the envelope automatically adjusts its range to match the expected behavior under that semantic condition. For example, under the semantic condition of "All-Reduce + 64 rank + 256MB", the queue depth may reach 150 data packets during normal training. This value would be misjudged as abnormal under a fixed-threshold method (e.g., a fixed threshold of 100), but in the probabilistic semantic network envelope, since historical samples show that the normal range under this semantic condition is [50, 200], 150 is considered a normal value.

[0049] In its implementation, the device first determines the semantic conditions of communication events in each set within the current distributed training task based on the training semantic time sequence diagram. It then categorizes network observation indicators from the network telemetry data according to their corresponding semantic conditions, grouping those with the same or similar semantic conditions into the same envelope bucket. Next, the device utilizes historical network observation indicator data under each semantic condition, employing robust statistical methods (such as median and median absolute deviation) or quantile methods (such as 5th percentile to 95th percentile) to statistically determine the normal value range of the network observation indicators, thereby obtaining the probabilistic semantic network envelope under that semantic condition. Furthermore, when the number of historical samples under a specific semantic condition is insufficient, the device can use samples from the previous level for envelope estimation according to a preset fallback hierarchy (e.g., gradually falling back from the specific semantic condition to a broader semantic condition) to ensure the reliability of the envelope.

[0050] Step S40: Based on the probabilistic semantic network envelope detection, detect whether there is a malicious congestion event in each training step, and obtain the network congestion detection result of the current distributed training task.

[0051] It should be understood that malicious congestion events can be network anomalies deliberately injected by external attackers to cause communication delays and increased training step times in distributed training tasks. In practical applications, malicious congestion events do not require attackers to continuously send large amounts of traffic. Instead, they only need to send short bursts of congestion traffic with low average rates to the shared bottleneck link within the critical set communication window of the current distributed training task. Because the average rate of this congestion traffic is low, traditional detection methods based on average rate or fixed thresholds are difficult to detect. Furthermore, because the short bursts of traffic are precisely aligned with the critical set communication window, the communication completion time of a few training ranks is slowed down. This, in turn, propagates the local delay to all participating training ranks through the synchronization barrier mechanism of set communication, ultimately leading to a significant increase in the overall training step time and affecting training performance.

[0052] It should also be understood that network congestion detection results can be a set of information indicating whether malicious congestion events and related circumstances exist in the current distributed training task. This can include detection conclusions, event time information, event location information, malicious congestion scores, etc. The detection conclusions indicate whether malicious congestion events exist in the current training step or the current time window, including "malicious congestion events exist" and "malicious congestion events do not exist". The event time information indicates the time window or training step number corresponding to the detected malicious congestion event. The event severity information indicates the degree of impact of the malicious congestion event on the training task (e.g., the increase in training step time, the increase in GPU waiting time).

[0053] In practical applications, the device can detect real-time network observation metrics collected in each training step based on a defined probabilistic semantic network envelope to determine whether malicious congestion events exist in each training step and generate network congestion detection results. Specifically, the device can compare the real-time network observation metrics in each training step with the probabilistic semantic network envelope under the corresponding semantic conditions. If the real-time network observation metrics exceed the normal range, the envelope violation degree can be calculated based on the degree of exceedance. Subsequently, the device can calculate the malicious congestion score by integrating evidence such as the envelope violation degree, stage lockout score (used to describe the degree of overlap between congestion events and highly critical communication windows), causal order score (used to describe whether the propagation order between bursts, queue growth, congestion signals, rank tails, and synchronization waits is valid), path overlap score (used to describe the degree of path overlap between candidate sources and victim rank communication), and attack amplification score (used to describe the increase in training step time caused by a unit burst of traffic). When the malicious congestion score exceeds a preset threshold, it can be determined that a malicious congestion event exists in that training step; otherwise, it is determined that no malicious congestion event exists. Finally, the device can generate network congestion detection results for the current distributed training task, which include a determination of whether a malicious congestion event exists, and a malicious congestion score when a malicious congestion event exists.

[0054] This embodiment provides a congestion detection method. The method discloses acquiring training semantic data and network telemetry data for the current distributed training task. The training semantic data characterizes computational events and aggregated communication events corresponding to multiple training nodes in the current distributed training task, while the network telemetry data describes the network state during the operation of the current distributed training task. A training semantic time sequence graph for the current distributed training task is constructed based on the training semantic data. This graph describes the relationships between computational events, aggregated communication events, and network resources in each training step of the current distributed training task. Based on the training semantic time sequence graph, the probabilistic semantic network corresponding to network observation indicators in the network telemetry data under different semantic conditions is determined. The probabilistic semantic network envelope is used to describe the normal value range of network observation indicators under various semantic conditions. Based on the probabilistic semantic network envelope, the network congestion detection result of the current distributed training task is obtained by detecting whether there are malicious congestion events in each training step. Since this embodiment can establish probabilistic semantic network envelopes for different semantic conditions based on training semantic time sequence graphs and network telemetry data to describe the normal value range of each network indicator, it can replace fixed thresholds for anomaly judgment. This makes the entire process from network anomaly detection to malicious congestion judgment no longer dependent on fixed thresholds, thereby solving the technical problem that existing congestion detection methods are difficult to distinguish between training-aligned malicious congestion and benign anomalies, resulting in a high risk of false negatives and false positives.

[0055] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating Embodiment 2 of the congestion detection method of this application.

[0056] In this embodiment, the step of constructing the training semantic time sequence graph of the current distributed training task based on the training semantic data includes steps S21 to S24: Step S21: Determine the training computation event node, set communication event node, and collaboration node corresponding to each training node in each training step of the current distributed training task based on the training semantic data.

[0057] It should be noted that training computation event nodes can be graph nodes used to represent computation events in the training semantic temporal graph, which may include node identification information, event type information, time window information, information of the training node to which they belong, and additional context information, etc. In this embodiment, training computation event nodes can be represented as follows:

[0058] in, As an event identifier, For event type, For training positions (including training step number, stage, layer number, and micro-batch number, etc.). For the event time window, To participate in the order or execute resources, For input / output data objects, For additional context.

[0059] It should also be noted that training computation event nodes can serve as the start or end point of directed edges in the training semantic temporal graph, used to express the dependencies between computation events and other events. For example, in a training semantic temporal graph, "the reverse computation of training node 0 in the 100th training step" is represented as a training computation event node. The attributes of this node include event identifier, event type (reverse computation), time window (start time t1 to end time t2), and the training node to which it belongs (rank 0).

[0060] It should be noted that the set communication event node can be a graph node in the training semantic temporal sequence graph used to represent the set communication event, which may contain node identification information, communication operation type information, communication algorithm type information, participation rank set information, message size information, and time window information, etc. In this embodiment, the set communication event node can be a vertex in the graph data structure, used to carry relevant information of the set communication event. For any set communication event... Its corresponding semantic context It can be uniformly represented as:

[0061] in, It represents training location information (including training step, stage, layer, and microbatch). It represents communication attributes (including communication operation type, communication algorithm, and parallel group type). This represents the participating resources (including the participating rank set, the participating host set, and the RDMA queue pair set). Indicates the distribution of candidate paths. Represents the message structure (including message size, fan-in and fan-out, and burst patterns). This indicates the criticality of the communication.

[0062] It should be understood that a collaborative node can be a graph node in the training semantic sequence graph used to represent collaborative or synchronous operations between multiple training nodes. In the training semantic sequence graph, it typically serves as a common successor node after multiple events converge, expressing the synchronous semantics that "subsequent events are only allowed to execute after all multiple events are completed." In this embodiment, collaborative nodes may include, but are not limited to, gradient-ready nodes, parameter-ready nodes, expert routing completion nodes, and synchronization barrier nodes. Gradient-ready nodes represent that the gradient calculation of a certain layer or gradient bucket on a certain training node has been completed, and the gradient data is ready for ensemble communication. Parameter-ready nodes represent that all parameter fragments of a certain layer on a certain training node have been collected and are available for calculation. Expert routing completion nodes represent that expert routing calculation in the MoE scenario has been completed, and the target and message size of All-to-All communication have been determined. Synchronization barrier nodes represent that all participating ranks of a logical ensemble communication event have completed their respective local communication, and subsequent events (such as optimizer updates, next layer calculations, or the next training step) are only allowed to begin execution.

[0063] It should also be understood that, unlike training computation event nodes and ensemble communication event nodes, collaborative nodes do not represent specific computational or communication operations. Instead, they represent a "wait-trigger" synchronization mechanism used to characterize the synchronization waiting relationship in multi-node collaborative training. For example, in a data-parallel training, a synchronization barrier node connects a logical All-Reduce event node and a subsequent optimizer update event node, indicating that any rank is allowed to begin executing optimizer updates only after all 64 ranks of All-Reduce have been completed.

[0064] In its implementation, the device can read the training step number from the training semantic data, categorize each event record in the training semantic data according to the training step number, and then divide the data according to the training node to which the event belongs within the same training step. For each training node, the device can identify the computational operations performed by that node, such as forward computation, backward computation, gradient computation, and optimizer update, as computational events, and obtain the event identifier for each computational event. Event Type Training position Event Time Window , participate in order or execute resources Input / output data objects and additional context This information is then combined into training computation event nodes. Simultaneously, the device can identify the All-Reduce, All-Gather, and All-to-All communication operations performed by this node as aggregated communication events, and obtain the training location information for each aggregated communication event. , communication attributes Participating resources Candidate path distribution Message Structure and the criticality of communication The device combines this information into a set of communication event nodes. Furthermore, it can identify synchronization states recorded in the training semantic data, such as gradient-ready states, parameter-ready states, and synchronization barrier states, that require multiple training nodes to reach as collaborative events, and generate corresponding collaborative nodes for each collaborative event.

[0065] Step S22: Determine the dependencies between the computation events corresponding to the training computation event nodes, the set communication events corresponding to the set communication event nodes, and the collaboration events corresponding to the collaboration nodes.

[0066] Understandably, a collaborative event can be an event in distributed training that involves multiple training nodes and requires coordination to complete. These events can include, but are not limited to, gradient ready events (indicating that gradient calculations for all participating ranks have been completed), parameter ready events (indicating that parameter fragments for all participating ranks have been collected), expert routing completion events (indicating that routing decisions for all source ranks have been determined), and synchronization barrier events (indicating that overall communication of the logical set is complete and subsequent operations can begin).

[0067] It is also understood that a dependency relationship can be a logical sequence or triggering relationship between two event nodes in a training semantic sequence graph. It can be used to express that the start or completion of one event depends on the completion of another event. In this embodiment, dependencies include, but are not limited to, control order dependencies, data dependencies, micro-batch or pipeline dependencies, collective communication participation dependencies, synchronization barrier dependencies, communication resource order dependencies, expert routing dependencies, and parameter sharding dependencies. Among these, control order dependencies characterize that one event must wait for another event to complete before it can begin execution. For example, the All-Reduce local communication of that rank can only be triggered after the reverse computation is completed. The time constraints; data dependencies are used to characterize that the tensors, gradients, parameter slices, optimizer states, or expert tokens output by one event are input data for another event; micro-batch or pipeline dependencies are used to characterize that two events belong to the same micro-batch or adjacent pipeline stages and have a specified sending, receiving, or execution order; set communication participation dependencies are used to characterize that multiple rank communication sub-events belong to the same logical set communication instance; synchronization barrier dependencies are used to characterize that subsequent optimizer updates, next layer computations, or the next training step must wait for the logical set communication to complete; communication resource order dependencies are used to characterize that two communication events use the same communication stream, the same queue pair, the same network card sending queue, or are constrained by the execution order of the communication library; expert routing dependencies are used to characterize that the All-to-All communication target and message size can only be determined after MoE gating and token counting are completed; parameter slice dependencies are used to characterize that the ownership, collection, and merging relationships of ZeRO parameter or gradient slices determine the order of the corresponding All-Gather or Reduce-Scatter events.

[0068] In practical applications, the device can traverse all the identified event nodes and identify whether there are dependency constraints between any two events according to the logical relationship and time order between the events. Specifically, the device can identify dependency relationships such as control order dependency, data dependency, micro-batch or pipeline dependency, set communication participation dependency, synchronization barrier dependency, communication resource order dependency, expert routing dependency, and parameter fragmentation dependency.

[0069] Step S23: Based on the dependency relationship and the preset dependency edge establishment rules, establish directed dependency edges between the training computation event node, the set communication event node and the collaborative node.

[0070] It should be noted that the preset dependency edge establishment rules can be a pre-defined set of judgment conditions and establishment methods used to determine whether a directed dependency edge needs to be established between any two event nodes. In this embodiment, the preset dependency edge establishment rules may specifically include, but are not limited to: if an event must start only after another event has completed, then a control order dependency edge is established; if the data object output by one event is the input of another event, then a data dependency edge is established; if multiple rank-level communication sub-events belong to the same logical set communication instance, then a set communication participation dependency edge is established; if subsequent events must wait for the logical set communication to complete, then a synchronization barrier dependency edge is established; if two communication events use the same communication stream or the same queue pair, then a communication resource order dependency edge is established.

[0071] It should also be noted that a directed dependency edge can be an edge connecting two nodes in the training semantic sequence graph and indicating the direction of dependency. It can point from the dependent event node to the event node that depends on that event, indicating that the dependent event must occur before the event that depends on it. In this embodiment, a directed dependency edge can include attribute information such as dependency type identifier, minimum allowed time interval, maximum allowed time interval, applicable rank range, and mandatory or soft dependency identifier. It can be recorded as:

[0072] in, Indicates the type of dependency. This represents the allowed event interval (i.e., the minimum and maximum allowed time difference between the dependent event node and the start of the event node that depends on this event). This indicates the rank range to which the dependency applies. This indicates a mandatory dependency or a soft dependency that allows overlap.

[0073] In its implementation, the device can convert each identified dependency relationship into a directed edge connecting the corresponding nodes based on preset dependency edge establishment rules. Specifically, for control order dependencies, the device can establish a directed edge from the dependent event node to the event node that depends on it, indicating that the dependent event must complete before the dependent event. For data dependencies, the device can establish a directed edge from the event node that outputs data to the event node that uses the data, indicating that the data must be ready before it can be used. For set communication participation dependencies, the device can establish directed edges from the rank-level communication sub-event nodes of each participating rank to the logical set communication event node, indicating that the local communications of all ranks together constitute the logical set communication event. For synchronization barrier dependencies, the device can establish directed edges from the logical set communication event node to the subsequent computation event node or optimizer update event node, indicating that subsequent operations must wait for the logical set communication to complete before execution. For communication resource order dependencies, the device can establish a directed edge from the communication event node that occupies it first to the communication event node that occupies it later, indicating that the two communication events share the same communication stream or queue pair and must be executed sequentially.

[0074] Step S24: Construct the training semantic time sequence graph of the current distributed training task based on the training computation event node, the set communication event node, the collaborative node, and the directed dependency edge.

[0075] It should be noted that the training semantic sequence graph in this embodiment can be represented in the form of a directed graph:

[0076] in, Indicates training steps Cross-node semantic-communication-topology joint timing graph Represents a set of nodes. Denotes the set of edges. A set of attributes representing nodes and edges.

[0077] The set of nodes can be further represented as:

[0078] in, It is a set of training computation event nodes at each rank, including data loading, forward computation, backward computation, expert computation, and optimizer update, etc. It is a set of communication nodes, including logical set communication event nodes and rank-level communication sub-event nodes; It is a collection of network resource nodes, including hosts, network interface cards, RDMA queue pairs, switch ports, switch queues, and wide area link segments, etc. It is a set of coordinating nodes, including gradient ready, parameter ready, expert route completion, and synchronization barrier.

[0079] The edge set can be further represented as:

[0080] in, This refers to the data and control dependency edges within a single training node (i.e., a portion of the directed dependency edges). This refers to the set communication participation and synchronization dependency edges between different training nodes (i.e., another part of the directed dependency edges). The mapping edges from communication sub-events to network resources and candidate paths; These are topological connection edges between network resources; Congestion propagation edges are defined for candidate source bursts, queue growth, congestion signals, rank tails, and synchronization waits.

[0081] Further, step S24 includes: Step S241: Determine the start time and completion time of each of the aforementioned collective communication events.

[0082] It should be understood that the event start time can be a timestamp indicating that a set of communication events begins execution on a single participant rank; correspondingly, the event completion time can be a timestamp indicating that a set of communication events completes execution on a single participant rank. In this embodiment, the event start time and event completion time together define the participant rank. In collective communication events Local communication time window .

[0083] It should also be understood that the event start time and event completion time can be obtained from the collection communication library's trace logs, performance analyzer, or eBPF events, or through the host-side RDMA completion queue event timestamps.

[0084] Step S242: Based on the event start time and the event completion time, each set communication event is expanded into several rank-level communication sub-events that are executed by the participating rank.

[0085] Understandably, the participation rank can be an identifier of the training rank of participants in the same logical set of communication events. It is a logical identifier of a training node in a distributed training task, with each training node corresponding to a unique rank number. For a given set of communication events... All of its participating ranks constitute the participating rank set. .

[0086] It should be noted that a rank-level communication sub-event can be a communication sub-event that is executed independently by each participating rank after a logical set of communication events is broken down according to the participating rank. It can describe the local communication behavior of a specific participating rank in the logical set of communication events, including the communication start time, communication end time, host, network card, RDMA queue pair used, and the number of bytes sent or received. For example, an All-Reduce event involving 32 participating ranks can be expanded into 32 rank-level communication sub-events, and each rank-level communication sub-event corresponds to a local send and receive operation of a GPU in that All-Reduce.

[0087] It should also be noted that, for a set of communication events The device does not only represent it as a single event, but also as each participating rank Establish rank-level communication sub-events Among them, rank-level communication sub-events It can be represented as:

[0088] in, To participate in the rank identifier, Indicates participation in the order The host where it is located Indicates participation in the order The network card used Indicates participation in the order The RDMA queue pair used The start time of the event. The time for the event to be completed. Indicates participation in the order In collective communication events The number of bytes sent or received.

[0089] In this embodiment, the event completion time of the aggregate communication event node can be determined by the slowest participant rank, that is:

[0090] The synchronization wait time caused by the rank factor waiting for the overall completion of communication of the complete partial communication set is:

[0091] It should also be noted that in the training semantic time sequence graph, set communication event nodes can serve as the start or end point of directed edges, used to express the dependencies and convergence relationships between set communication events and other events. Furthermore, set communication event nodes can be further distinguished into logical set communication event nodes and rank-level communication sub-event nodes. A logical set communication event node represents a complete set communication operation (e.g., All-Reduce completed by all 64 ranks), while a rank-level communication sub-event node represents the local communication behavior of a training rank within that logical set communication (e.g., the send and receive operations of rank 0 in All-Reduce). For example, in a training semantic time sequence graph, "the All-Reduce logical event of the data parallel group in the 100th training step" can be represented as a logical set communication event node, while "the local communication of rank 0 in this All-Reduce" can be represented as a rank-level communication sub-event node.

[0092] In the specific implementation, for each set of communication events The device can first obtain the set of participating rank events of the set of communication events. It also retrieves the local communication records of each participating rank corresponding to the communication events in the set from the training semantic data, including the event start time of each participating rank r. Event completion time Host The network card used The RDMA queue used and the number of bytes sent or received in the set communication event. Then, the device can target each participating rank. Generate a rank-level communication sub-event node This allows the logical set communication event to be expanded into multiple rank-level communication sub-event nodes, the same number as the number of participating ranks. Furthermore, the device can calculate the logical set communication completion time and the synchronization wait time for each rank, and record these times in the attributes of the corresponding set communication event node and cooperating node.

[0093] Step S243: Establish weighted mapping edges between each rank-level communication sub-event and the candidate path segment according to the candidate path distribution.

[0094] It should be noted that the candidate path distribution can be a set of all candidate network paths or path segments that a communication event may traverse and their corresponding probabilities. In wide-area RDMA networks, due to the influence of factors such as ECMP (Equal Cost Multipath) hashing, routing strategies, and telemetry granularity limitations, the precise end-to-end path is often difficult to uniquely determine. Therefore, this embodiment can use a probability distribution to describe the set of possible paths.

[0095] In this embodiment, for rank-level communication sub-events The corresponding candidate path distribution It can be represented as:

[0096] It satisfies:

[0097] in, Indicates the first Candidate paths or candidate path segments, express Use candidate paths or candidate path segments The probability of.

[0098] In practical applications, the probability in the candidate path distribution can be jointly determined based on information such as host and network interface card (NIC) locations, routing tables, ECMP hash rules, switch port counts, ECN tag locations, CNP feedback directions, PFC propagation range, host traffic matrices, and historical overlap evidence. For example, for a rank-level communication sub-event from host A to host B, the ECMP hash rule might hash the traffic to two equivalent paths. and ,in The probability is 0.6. If the probability is 0.4, then the distribution of this candidate path is: .

[0099] It should also be noted that candidate path segments can be network paths connecting two adjacent network nodes in the network topology. These can be part of a complete end-to-end path or a segment passing through a specific switch port or WAN link. In this embodiment, candidate path segments can be observable or inferable basic units in the network topology, used for congestion localization and source analysis under uncertain path conditions. They can include network segments of different granularities, such as path segments between hosts and network cards within the same physical server, path segments between switches within the same rack, path segments between aggregation switches across racks, and WAN link segments across data centers. For example, a complete end-to-end path from host A to host B can sequentially pass through "host A → access switch A → aggregation switch → access switch B → host B," where "access switch A → aggregation switch" is a candidate path segment.

[0100] It should be noted that the weighted mapping edge can be an edge in the training semantic temporal graph used to connect rank-level communication sub-event nodes and candidate path segment nodes, carrying probability weights. It can be used to represent the probability that the communication traffic corresponding to the rank-level communication sub-event passes through the candidate path segment. In this embodiment, for rank-level communication sub-events... and candidate path segments The weight of the weighted mapping edge between the two can be expressed as:

[0101] in, This indicates a rank-level communication sub-event. This represents a candidate path segment, and the weight indicates the communication sub-event at a given rank. Under these conditions, communication traffic uses candidate path segments. The probability of.

[0102] In practical applications, for each rank-level communication sub-event The device can determine the candidate path segments that the communication traffic of a rank-level communication sub-event may traverse, based on the host location, network interface card (NIC) location, destination host location, network topology, routing table information, and ECMP hash rules. Subsequently, the device can query the routing table to determine multiple possible equivalent paths from the source host to the destination host, estimate the probability of each path being selected using ECMP hash rules, and then organize the candidate path segments and their corresponding probabilities into a candidate path distribution. Finally, the device can calculate the rank-level communication sub-event for each candidate path segment. The probability of using the candidate path segment is used as a weight to establish a weighted mapping edge from the rank communication sub-event node to the candidate path segment node in the training semantic temporal graph, and the weight value is recorded in the edge attribute.

[0103] Step S244: Determine the shared bottleneck probability between each pair of the computation event, the rank communication sub-event, and the cooperative event.

[0104] It should be noted that the shared bottleneck probability can be the probability that two events (e.g., two rank-level communication sub-events, or a computation event and a rank-level communication sub-event) will both pass through the same bottleneck link or the same network resource during network transmission. It can be used to determine whether two communication flows might encounter resource contention at a certain point in the network. In this embodiment, the two rank-level communication sub-events... and Shared bottleneck probability The corresponding calculation formula can be:

[0105] in, and They represent and The set of candidate paths, and They represent and Use path segment The probability of sharing a bottleneck ranges from [0,1]. The closer the value is to 1, the more likely the two communication sub-events are to share the same network bottleneck resource (e.g., the same switch queue, the same egress port, or the same WAN link segment).

[0106] In practical applications, for any two event nodes (e.g., two rank-level communication sub-events) and The device can obtain its corresponding candidate path set respectively. and The device calculates the corresponding path usage probabilities and takes the intersection of the candidate path sets of the two event nodes to obtain a common candidate path set. For each path segment in the common candidate path set, the device multiplies the probabilities of the two events using that path segment to obtain the probability that the two events share the same bottleneck under that path segment. Subsequently, the device sums the products of all common candidate path segments to obtain the shared bottleneck probability between the two events. Afterward, the device can perform the above calculations between computation events and rank-level communication sub-events, between computation events and cooperative events, and between rank-level communication sub-events and cooperative events to obtain the shared bottleneck probability between each event pair.

[0107] Step S245: Construct the training semantic time sequence graph of the current distributed training task based on the training computation event node, the rank communication sub-event, the collaborative node, the directed dependency edge, the weighted mapping edge, and the shared bottleneck probability.

[0108] In practical applications, the device can expand the node set of the training semantic temporal graph to include training computation event nodes, rank-level communication sub-event nodes (replacing the previous set communication event nodes), cooperative nodes, and network resource nodes, and expand the edge set to include directed dependency edges, weighted mapping edges, and probabilistic edges corresponding to shared bottleneck probabilities. In this embodiment, the device can select two event nodes whose shared bottleneck probabilities exceed a preset threshold to establish a probabilistic edge based on the magnitude of the shared bottleneck probability, record the shared bottleneck probability value in the edge's attributes, and then combine the node set and edge set to generate the training semantic temporal graph. ,in Record the attribute information of each node and each edge, including the weight value of the weighted mapping edge and the probability value of the shared bottleneck.

[0109] Furthermore, when evidence that conforms to the conditions of temporal order and shared path is observed, the device can also establish congestion propagation chains in the training semantic temporal graph:

[0110] in, Indicates candidate source Short-term outbreak, Represents path segment The queue is growing. This indicates an ECN, CNP, or PFC congestion signal. Indicates the victim's rank Communication trailing, This indicates a synchronization wait caused by the incomplete communication of other rank-factor logical sets.

[0111] Furthermore, for training tasks involving multiple data parallel ranks, the device can also perform training on each rank. Establish on:

[0112] All rank-level All-Reduce sub-events converge at the logical set communication node and are updated via the synchronization barrier connection optimizer:

[0113]

[0114] If a candidate path for a certain victim rank shares a wide-area bottleneck link with the attack flow, then that rank will first experience a CNP or completion time tail, and then proceed through... and This causes other ranks to wait. This structure is used to distinguish between "late computation causing late communication startup" and "communication starting on time but completing late due to network congestion".

[0115] For source rank To target expert rank During the token sending process, the device can establish:

[0116] The message size from source rank to target expert rank is:

[0117] in, The number of tokens routed to the target expert rank. This represents the amount of data corresponding to a single token. The device can use the expert routing matrix as part of the semantic context to distinguish between normal communication skew caused by uneven expert load and rank tailing caused by normal message size but abnormal network paths.

[0118] For ZeRO-3, the device can be in rank Before performing the first-level computation, parameter fragments need to be obtained from other ranks:

[0119]

[0120] Gradient piecewise merging in backpropagation is represented as:

[0121] Thus, the device can represent how the delay of a certain parameter fragment sender or a certain candidate path is abnormal, blocking the layer calculation of multiple remote ranks and continuing to propagate along the model layer dependencies.

[0122] In this embodiment, the start and end times of each set of communication events are determined, and each set of communication events is expanded into several rank-level communication sub-events executed by the participating ranks based on the start and end times. Then, weighted mapping edges are established between each rank-level communication sub-event and the candidate path segment according to the candidate path distribution, and the shared bottleneck probability between each pair of computation events, rank-level communication sub-events, and cooperative events is determined. Finally, a training semantic time sequence graph is constructed based on training computation event nodes, rank-level communication sub-events, cooperative nodes, directed dependency edges, weighted mapping edges, and shared bottleneck probabilities, forming a joint graph structure that integrates cooperative training semantics, fine-grained communication time information, and network resource probability mapping. This enables subsequent congestion detection to identify malicious congestion behavior from the complete causal chain of "participating rank tail → logical set communication delay → synchronization waiting amplification," and to use shared bottleneck probabilities and weighted mapping edges for probabilistic congestion tracing. Thus, even in a wide-area network environment with uncertain paths, quantifiable attribution evidence can still be provided, ensuring the accuracy and interpretability of malicious congestion detection and location.

[0123] Further, the step of determining the probabilistic semantic network envelope corresponding to the network observation index in the network telemetry data under different semantic conditions based on the training semantic time sequence graph includes: Step S31: Determine the semantic context corresponding to each set of communication events in the current distributed training task based on the training semantic time sequence diagram. The semantic context includes at least one of training location information, communication attributes, participating resources, candidate path distribution, message structure, and communication criticality.

[0124] It should be noted that semantic context can be a dataset describing the complete training environment in a distributed training task, used to uniquely determine the location, role, and resource configuration of the collective communication event within the training logic. This includes at least one of the following: training location information, communication attributes, participating resources, candidate path distribution, message structure, and communication criticality. For example, training location information describes which training step, training stage, layer, and micro-batch the event occurs in; communication attributes describe the communication operation performed, the algorithm used, and the parallel group to which the event belongs; participating resources describe which training ranks, hosts, and RDMA queue pairs are involved; candidate path distribution describes the network paths the event's communication traffic may traverse and their probabilities; message structure describes the amount of data transmitted, fan-in / fan-out scale, and burst pattern; and communication criticality describes the impact of the event on the overall training step time.

[0125] In this embodiment, for any set communication event Its corresponding semantic context It can be represented as:

[0126] in, It represents training location information (including training step, stage, layer, and microbatch). It represents communication attributes (including communication operation type, communication algorithm, and parallel group type). This represents the participating resources (including the participating rank set, the participating host set, and the RDMA queue pair set). Indicates the distribution of candidate paths. Represents the message structure (including message size, fan-in and fan-out, and burst patterns). This indicates the criticality of the communication.

[0127] It should be noted that training location information can be a description of the temporal and spatial locations of collective communication events in a distributed training task. This information can be used to correlate network observation metrics with specific training spatiotemporal locations, thereby determining whether network anomalies are related to a specific training phase. Specifically, it can be represented as:

[0128] in, Indicates the global training step or iteration number; This refers to the training phase, which includes data loading, forward propagation, back propagation, gradient synchronization, parameter prefetching, optimizer updates, and checkpoint saving. The model layer is represented by Transformer blocks, MoE layers, expert layers, or logistic gradient buckets, etc. This indicates the micro-batch number within the current training step. For events that do not involve model layers or micro-batches, the corresponding field is set to null, a wildcard value, or a predefined invalid identifier.

[0129] It should also be noted that communication attributes can be characteristic information of the communication operations performed by a set of communication events. These attributes can be used to distinguish different communication operation types and algorithms' different usage patterns of network resources, specifically as follows:

[0130] in, Indicates the type of communication operation, including All-Reduce, Reduce-Scatter, All-Gather, All-to-All, Broadcast, Send, or Recv, etc. Indicates the type of communication algorithm, including Ring, Tree, Recursive Doubling, Pairwise Exchange, or hierarchical communication algorithms; Indicates the type of parallel group, including data parallel group, tensor parallel group, pipeline parallel group, expert parallel group, or ZeRO fragmentation group, etc.

[0131] Understandably, participating resources can be information on the computational and network resources involved in a communication event. This information can be used to determine the scale of participation and the distribution of resources in the communication event. Different scales of participation and resource distributions have a significant impact on the convergence pattern and degree of competition of network traffic, which can be specifically expressed as follows:

[0132] in, This is the set of participating rank (i.e., the set of logical numbers of the training rank of the communication events participating in this set). For the set of participating hosts (i.e., the set of physical hosts that participate in the communication events of that set). This refers to the set of RDMA queue pairs involved (i.e., the set of identifiers for the RDMA queue pairs used for communication events in this set).

[0133] It is also understandable that the message structure can be the organizational characteristic information of the data transmitted in a collection of communication events. This information can be used to determine whether the traffic characteristics of the communication event meet expectations, thereby distinguishing between normal and abnormal communication traffic. Specifically, it can be represented as follows:

[0134] in, Indicates the message size (i.e., the total amount of data transmitted). and These represent the fan-in and fan-out sizes (i.e., the number of nodes participating in sending and receiving, respectively). It indicates the nature of a sudden event, including the peak value, duration, interval, and periodicity of the event.

[0135] It is also understandable that communication criticality can be a quantitative indicator of the impact of a set of communication events on the overall training step time. It can be used to semantically weight envelope violations and comprehensive risks, guide telemetry encryption sampling of high-criticality events, determine alarm and handling priorities, and identify amplified features of "small-volume attacks on critical windows causing large training losses". The corresponding calculation formula can be:

[0136] in, This indicates whether the event occurred before the synchronization barrier and the degree of blocking it on subsequent execution; This indicates the degree to which the event is close to the end of the training step or the end of the critical path; Indicates the scale of participation and the scope of affected synchronization; This indicates the proportion of the historical variation in the duration of the event that contributes to the training step time. This is the normalization function. Furthermore, to Let be the weighting coefficients, and satisfy:

[0137] In a specific implementation, the device can determine the semantic context corresponding to each set communication event in the current distributed training task based on the training semantic sequence diagram. Specifically, the device can traverse the set communication event nodes (including logical set communication event nodes and rank communication sub-event nodes) in the training semantic sequence diagram, read the corresponding event attribute information from the attribute set of each node, and read the training location information, communication attributes, participating resources, candidate path distribution, and message structure of each set communication event from the training semantic sequence diagram. Then, it can calculate the communication criticality of the event and finally combine this information into the complete semantic context of the set communication event.

[0138] Step S32: Based on the preset index condition distribution and the semantic context, divide the network observation index in the network telemetry data into several envelope buckets.

[0139] It should be noted that the preset index condition distribution can be a pre-defined rule used to classify and group network observation indicators in network telemetry data according to their corresponding semantic context. In practical applications, the preset index condition distribution limits the combination of semantic condition dimensions used when dividing network observation indicators into each envelope bucket. For example, the preset index condition distribution can specify division according to the combination of "communication operation type + communication algorithm + participation rank group + message size category + topology region + training stage", or it can specify division according to a simplified combination such as "communication operation type + message size category" when there are insufficient samples.

[0140] It should also be noted that an envelope bucket can be a container for storing network observation indicator data samples with the same or similar semantic context. It is associated with specific semantic conditions, and network observation indicator samples within the same envelope bucket all originate from a set of communication events with the same or similar semantic conditions. In this embodiment, the preset indicator condition distribution can specify the bucket key as:

[0141] in, Indicates the type of communication operation; Indicates the type of communication algorithm; Indicates participation in the rank group or parallel group; Indicate message size category (divided into small message, medium message, large message, and extra-large message according to communication library protocol switching point, link bandwidth product, historical message size quantile or logarithmic interval); Indicates the topology region category, including same machine, same rack, across racks, across clusters, or across data centers; Indicates the training phase category. Used to generate fast retrieval keys, which are not limited to cryptographic hash algorithms.

[0142] In practical applications, the device can generate bucket keys corresponding to each set of communication events based on the semantic context of each dimension, according to preset indicator conditions. Network observation indicators corresponding to set communication events with the same bucket key are then grouped into the same envelope bucket. Furthermore, if the number of samples in a precise bucket is insufficient, bucket rollback can be performed according to a preset rollback level. The preset rollback level can be:

[0143]

[0144]

[0145]

[0146] The equipment can be arranged according to The process rolls back sequentially, only when the number of samples in the current level is lower than the predetermined minimum number of samples, until the number of samples meets the requirement.

[0147] Step S33: Determine the normal value range of the network observation index in each envelope bucket under the corresponding semantic conditions based on the historical sample set, and determine the normal value range as the probabilistic semantic network envelope.

[0148] It should be understood that the historical sample set can be a data set of network observation index samples collected and stored from historical training steps within the same envelope bucket (i.e., the same or similar semantic context). It can originate from historical network telemetry data collected under normal training conditions and is used to statistically analyze the index distribution characteristics under various semantic conditions. In this embodiment, for any network observation index... and semantic context The historical sample set can be represented as:

[0149] in, to This refers to the same network observation metric (e.g., port queue depth) under the same semantic conditions. One historical observation value.

[0150] It should also be understood that sample data in the historical sample set is used to calculate statistics such as the median, median absolute deviation, or quantiles, thereby determining the normal range of values. When the exact envelope bucket sample is insufficient, the system can merge it into a coarser-grained envelope bucket to obtain historical samples according to a hierarchical backoff strategy.

[0151] In practical applications, the device can collect all historical network observation indicator samples within each envelope bucket, forming a historical sample set, and use robust statistical methods to calculate the normal value range of any indicator (e.g., port queue depth) in this historical sample set. Specifically, the device can calculate the median of the historical sample set:

[0152] And calculate the median absolute deviation:

[0153] in, To prevent small constants with a denominator of zero. Furthermore, current observations... The robust standardized residual can be:

[0154] At this point, the device can define the normal value range as ,in This is an adjustable coefficient.

[0155] In addition, the device can also use the quantile envelope method to calculate the lower and higher quantiles of the historical sample set to determine the normal range of values, which can be specifically expressed as:

[0156] in, and These are the lower quantile (e.g., 5th percentile) and the upper quantile boundary (e.g., 95th percentile).

[0157] At this point, the device can calculate the normal value range. Determined as a network observation index under this semantic condition The probabilistic semantic network envelope is calculated, and the above calculation is performed on each network observation index in each envelope bucket to obtain the probabilistic semantic network envelope corresponding to each network observation index under each semantic condition. If the number of samples in the precise envelope bucket is less than the predetermined minimum number of samples, the hierarchical backoff order can be followed. (in For precise semantic buckets, To coarsen semantic buckets, To further coarsen the semantic bucket, The coarsest semantic bucket is merged into a coarser-grained envelope bucket, and the normal range of values ​​is determined using the merged historical sample set.

[0158] Furthermore, in this embodiment, the device only allows low-risk windows with no strong attribution evidence and complete telemetry to enter the normal envelope update set. Let the current window's suspicion level be:

[0159] in, To lock the score for the stage, For causal order fractions, To amplify the score for the attack, This is a benign interpretation of the intensity.

[0160] window A set of envelope updates can only be entered if all of the following conditions are met:

[0161] in, This is a low-risk suspicion threshold used to exclude windows with strong evidence of envelope violation, phase lock-in, or attack amplification. The attribution confidence exclusion threshold for the envelope update phase is used to exclude windows that can be associated with candidate sources with high confidence. This is the telemetry missing rate threshold, used to exclude windows with severe missing key observations.

[0162] In practical applications, , and All of these can be determined through "offline calibration and online calibration." The offline phase involves collecting data from confirmed, normal pre-training windows, calculating the distribution of each indicator, and selecting quantile thresholds while meeting the target false alarm rate and the maximum allowable envelope contamination rate. For example, We can take the high quantile of the normal window suspicion distribution; It can be determined based on the degree of separation of attribution confidence between normal windows and marked abnormal windows; The threshold can be determined based on the missing rate during normal operation of the telemetry system and the envelope estimation error under different missing rates. Only minor adjustments are allowed during the online phase, and minimum sample size, maximum variation range, and hysteresis interval are set to prevent attackers from gradually contaminating the threshold. To avoid confusion with subsequent malicious score thresholds, this application uses... , and The thresholds for benign explanations, malicious scores, and final attribution outputs will not be reused. or .

[0163] In this embodiment, the semantic context corresponding to each set of communication events is determined based on the training semantic time sequence diagram. Based on the preset index condition distribution and semantic context, the network observation index in the network telemetry data is divided into several envelope buckets. Then, the normal value range of the network observation index in each envelope bucket under the corresponding semantic conditions is determined based on the historical sample set. The normal value range is determined as the probabilistic semantic network envelope, so that the judgment benchmark for normal network behavior is no longer a single fixed threshold, but a probabilistic statistical boundary driven by historical normal operation data under each semantic condition. Furthermore, since the envelope of the probabilistic semantic network is a normal range of values ​​learned under specific semantic conditions, malicious congestion with a low average rate but aligned with the key communication window and deviating from the normal distribution under these semantic conditions can be detected because it exceeds the envelope. Benign anomalies such as normal aggregate communication convergence, GPU computation tailing, and uneven expert load are reduced in false alarms because they fall within the corresponding envelope range or can be semantically explained reasonably. At the same time, the statistical reliability of the envelope estimation is guaranteed by semantic bucketing and hierarchical backoff mechanism when samples are insufficient. Thus, semantically aware adaptive congestion detection is realized, which effectively solves the technical problem of high false alarm and missed alarm risks caused by the lack of semantic understanding in existing solutions.

[0164] Furthermore, the step of detecting whether malicious congestion events exist in each training step based on the probabilistic semantic network envelope includes: Step S41: Determine the envelope violation score of the training step based on the degree of deviation between the real-time network observation index in each training step and the probabilistic semantic network envelope under the corresponding semantic conditions.

[0165] Understandably, real-time network observation metrics can be quantifiable parameters collected in real time during the current distributed training task to measure the network's operating status. They share the same set of metric definitions as the network observation metrics in the historical sample set, including but not limited to maximum port queue depth, queue growth slope, ECN tagging ratio, CNP packet rate, total PFC pause duration, and changes in link utilization.

[0166] It is also understandable that the degree of deviation can be the deviation of real-time network observation indicators from the normal range of values ​​defined by the probabilistic semantic network envelope. It can be used to measure whether the current network observation values ​​deviate significantly from the statistical distribution range under historical normal operating conditions.

[0167] It should be noted that the envelope violation score can be a comprehensive score used to quantify the degree to which real-time network observation indicators deviate from the probabilistic semantic network envelope under the corresponding semantic conditions. It can be obtained by weighting the degree of deviation of each network indicator. The higher the envelope violation score, the more significant the deviation of the current network state from the normal semantic envelope.

[0168] In this embodiment, for the currently observed real-time network observation indicators The corresponding comprehensive envelope violation score The calculation formula can be:

[0169] in, For network observation index vectors, The weighting coefficients for each indicator (satisfying) and ), For a single indicator envelope violation degree, This is the normalization function.

[0170] The network observation vector can be represented as:

[0171] In the formula, This indicates the port queue depth, queue growth slope, and peak queue duration. This indicates the proportion, number, and growth rate of ECN tags; Indicates the number, direction, rate, and first occurrence time of CNP messages; Indicates the number of paused frames, total pause duration, maximum pause duration, and propagation range; This represents the host-side or queue-side traffic matrix; This indicates that the completion time of the participating order is skewed; Indicates the duration of the collective communication; This indicates the GPU's waiting or idle time.

[0172] When the envelope violation of a single metric can be defined as:

[0173] in, and These are the boundaries of the lower and higher quantiles, respectively. This represents the median absolute deviation under the corresponding semantic conditions.

[0174] In practical applications, for any real-time network observation metric in the current training step, the device can first determine the semantic condition corresponding to the observation metric and the probabilistic semantic network envelope under that semantic condition, and read the lower quantile boundary and higher quantile boundary, or the median absolute deviation, of the envelope. If the real-time observation value falls between the lower quantile boundary and the higher quantile boundary, the envelope violation degree of the metric can be determined to be 0; if the real-time observation value is lower than the lower quantile boundary or higher than the higher quantile boundary (i.e., y_t>Q_c^{high}(y)), the envelope violation degree can be calculated using the corresponding formula. Next, the device can calculate the comprehensive envelope violation score of multiple network metrics, and map the envelope violation degree of each network metric to the [0,1] interval through a normalization function, and then calculate the comprehensive envelope violation score according to the preset weight coefficients. For malicious congestion detection, the weights of metrics such as queue growth, ECN, CNP, PFC, rank skew, communication duration, and GPU wait can be higher than those of ordinary average byte count metrics.

[0175] Step S42: If the envelope violation score exceeds a first preset threshold, it is determined that a network congestion event exists in the corresponding training step.

[0176] Understandably, the first preset threshold can be a pre-set threshold for determining whether there is a network congestion event in the training step, which can be calibrated offline based on the user-configured target false positive rate and maximum allowable false negative rate. For example, by collecting confirmed normal warm-up training windows and labeled abnormal windows, the quantile threshold can be selected under the condition of meeting the target false positive rate and maximum allowable false negative rate.

[0177] It is also understandable that a network congestion event can be an event in a training step where one or more network observation metrics deviate significantly from normal expectations due to network resource contention or abnormal traffic injection. It is a preliminary judgment result of malicious congestion detection, indicating that an abnormal signal at the network level has been observed in the current training step, but it has not yet been distinguished whether the anomaly is caused by malicious attack, normal communication convergence or other benign reasons.

[0178] Step S43: Determine the phase lock score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event.

[0179] It should be noted that the phase lock-in score can be an indicator used to quantify the temporal overlap between network congestion events and highly critical communication windows. In this embodiment, the phase lock-in score... The calculation formula can be:

[0180] In the formula, For congestion events and highly critical communication windows (i.e. The number of times that high-value collective communication events overlap. The total number of observations. The stage lockout score ranges from [0,1]. A higher score indicates that the congestion event is more likely to occur within a critical communication window, and the greater the suspicion of malicious intent.

[0181] It should also be noted that the causal order score can be a rating metric used to verify whether the propagation path of a network congestion event conforms to the causal propagation order of "candidate source burst, queue growth, congestion signal, victim rank tail, and synchronization wait." It can be used to determine whether there is a causal relationship between the network event and the training tail, rather than merely a random temporal overlap. In this embodiment, the device can determine the burst time of the candidate source. Path segment queue growth time Congestion signal generation time (Including ECN tags, CNP messages, or PFC pause frames), victim-rank communication tail time and synchronization barrier waiting time Do the following order relationships and time interval constraints satisfy:

[0182] Furthermore, the time difference between adjacent events must fall within the corresponding allowed window:

[0183]

[0184]

[0185]

[0186] If all of the above complete order and time constraints are satisfied, the causal order score will be a higher value (e.g., 0.9); if any one of them is not satisfied, the causal order score will be reduced accordingly.

[0187] It should be noted that the path overlap score can be an indicator used to describe the degree of overlap between candidate sources and victim-rank communications on the network path. In this embodiment, the path overlap score... The corresponding calculation formula can be:

[0188] In the formula, Indicates candidate source The set of candidate paths, Indicates candidate source Use path segment The probability, Represents path segment In congestion events The congestion evidence strength is determined by the path overlap score, which ranges from [0,1]. A higher score indicates a greater probability that the candidate source and the victim rank share a network bottleneck.

[0189] It should also be noted that the attack amplification score can be used as an indicator to describe the increase in training step time caused by a unit burst of traffic. It can be used to characterize the malicious congestion feature of "small traffic causing large training loss," with a higher score indicating a greater training loss per unit burst of traffic. In this embodiment, the attack amplification score... The corresponding calculation formula can be:

[0190] In the formula, To increase the amount of time spent on training steps, Increase the amount of time the GPU is idle or waiting. The number of burst bytes in the candidate source within the exception window. For preset weighting coefficients, To prevent small constants with a denominator of zero.

[0191] Step S44: Determine the malicious congestion score corresponding to the network congestion event based on the envelope violation score, the phase lock score, the causal order score, the path overlap score, and the attack amplification score.

[0192] It should be noted that the malicious congestion score can be a quantitative indicator used to comprehensively determine whether the current network congestion event is malicious semantic congestion. It can be calculated by combining positive evidence such as envelope violation score, phase lock score, causal order score, path overlap score, and attack amplification score, as well as negative evidence such as benign interpretation strength and uncertainty score.

[0193] Step S45: If the malicious congestion score exceeds the second preset threshold, it is determined that there is a malicious congestion event in the training step.

[0194] Understandably, the second preset threshold can be a pre-set critical value for determining the malicious congestion score of malicious congestion events in the training step. It can be calibrated offline based on the target false positive rate and the maximum allowable false negative rate. For example, by collecting confirmed normal warm-up training windows and labeled abnormal windows, the quantile threshold can be selected under the condition of satisfying the target false positive rate and the maximum allowable false negative rate.

[0195] In this embodiment, an envelope violation score is determined based on the deviation between real-time network observation metrics and the probabilistic semantic network envelope. When the envelope violation score exceeds a first preset threshold, a network congestion event is identified. This achieves preliminary screening of network anomalies based on a semantic statistical baseline, avoiding in-depth analysis of all normal fluctuations. Simultaneously, by determining the stage lock score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event, the malicious suspicion of the congestion event can be comprehensively quantified from multiple dimensions such as time alignment, causal propagation integrity, path sharing degree, and attack efficiency. This allows for effective differentiation between benign anomalies such as normal aggregated communication convergence and GPU tailing and genuine malicious semantic congestion, thereby achieving accurate detection of malicious congestion events.

[0196] In practical applications, for identified network congestion events, the device can count the number of times the network congestion event overlaps with a high-criticality communication window within a preset observation window, and count the total number of observations to calculate the stage lockout score. Simultaneously, the device can verify whether the temporal order of candidate source burst events, path segment queue growth events, congestion signal events, victim rank tail events, and synchronization wait events meets the set order relationship and time interval constraints, and whether the time difference between adjacent events falls within the allowable window. If all order relationships are valid and the time differences all fall within the allowable window, the causal order score can be determined to be high (e.g., 0.9); if some order relationships are missing, the score is determined based on the proportion that is valid. Then, the device can obtain the probability of each candidate source using each path segment from the candidate path distribution, and obtain the congestion evidence strength of each path segment in the congestion event to calculate the path overlap score. Furthermore, the device can obtain the increase in training step time, the increase in GPU idle or waiting time, and the number of burst bytes of the candidate source within the abnormal window to calculate the attack amplification score. Subsequently, the device determines the malicious congestion score corresponding to the network congestion event based on the envelope violation score, phase lock score, causal order score, path overlap score, and attack amplification score. The calculated malicious semantic congestion score is then compared with a second preset threshold. If there is a candidate source that makes the malicious semantic congestion score greater than or equal to the second preset threshold, it can be determined that there is a malicious congestion event in the current training step. If the malicious semantic congestion scores of all candidate sources are lower than the second preset threshold, it can be determined that there is no malicious congestion event in the current training step.

[0197] Furthermore, in practical applications, since training events and network telemetry outputs typically come from different acquisition sources, the device can define a total time error:

[0198] In the formula, This refers to the clock synchronization error between the host, network card, and switch. Error introduced by telemetry sampling interval, This addresses the collector processing and transmission latency.

[0199] If network events timestamp Training communication events satisfy:

[0200] This assumes that network events can be aligned with training events. For rank analysis, time windows corresponding to sub-events can be used. Perform finer-grained alignment.

[0201] In this embodiment, training computation event nodes, collective communication event nodes, and collaborative nodes corresponding to each training node in each training step are determined based on training semantic data. The dependencies between these computation events, collective communication events, and collaborative events are then determined. Directed dependency edges between nodes are established based on the dependencies and preset dependency edge establishment rules. Finally, a training semantic time sequence graph is constructed based on each node and the directed dependency edges. This allows the constructed training semantic time sequence graph to accurately describe the sequential constraints and synchronous waiting relationships between different events in each training step. Consequently, subsequent congestion detection can understand the execution logic and time expectations of each event from the training semantic level, rather than relying solely on the time overlap of network signals for judgment. This provides a structured semantic basis for distinguishing between normal collective communication dependencies and malicious congestion interference, while ensuring the interpretability and logical consistency of malicious congestion detection and attribution reasoning, thus improving the accuracy and reliability of malicious congestion detection.

[0202] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating Embodiment 3 of the congestion detection method of this application.

[0203] In this embodiment, after step S40, the method further includes steps S51 to S53: Step S51: Determine the positive evidence score and negative evidence score of each candidate source in the candidate source set of the current distributed training task. The positive evidence score includes the target stage locking score, target path overlap score, rank skew persistence score, target attack amplification score, and target causal order score. The negative evidence score includes benign interpretation strength and uncertainty score.

[0204] It should be noted that the candidate source set can be a collection of data representing potential attack sources that may trigger malicious congestion events. It consists of multiple candidate sources (i.e., potential sources of malicious traffic), all of which could potentially cause the currently detected malicious congestion event. In this embodiment, the candidate source set... It can be represented as:

[0205]

[0206] in, For flows that share bottleneck links or ports with victim rank candidate paths; This refers to a sudden occurrence of queue pairs or hosts within the allowed time frame before the exception window; The flow at the point where the rate changes; For sources that do not belong to the current set of communication rank groups but send RDMA traffic during the critical window; A source that belongs to the current rank group but whose sending window, rate, queue pair, or burst pattern exceeds its own envelope; A cross-task flow that is periodically synchronized with the victim training steps.

[0207] It should be understood that the positive evidence score can be a quantitative score of various pieces of evidence supporting a candidate source as a malicious congestion source. This score may include at least one of the following: target phase lock score, target path overlap score, rank skew persistence score, target attack amplification score, and target causal order score. A higher positive evidence score indicates a greater likelihood that the candidate source is a malicious congestion source.

[0208] It should also be understood that the negative evidence score can be a quantitative score of various pieces of evidence opposing a candidate source as a malicious congestion source, that is, the strength of evidence used to explain that the current anomaly may be caused by benign reasons. It can include the strength of benign explanation and the uncertainty score. The higher the negative evidence score, the lower the probability that the candidate source is a malicious congestion source.

[0209] Understandably, the target phase lock score can be a phase lock score calculated for the current candidate source, which can be used to quantify the degree of temporal overlap between the abnormal traffic corresponding to the candidate source and the high-criticality communication window; the target path overlap score can be a path overlap score calculated for the current candidate source, which can be used to describe the degree of overlap between the candidate source and victim-rank communication on the network path; the target attack amplification score can be an attack amplification score calculated for the current candidate source, which can be used to characterize the degree to which the burst traffic of the candidate source amplifies the training step time; and the target causal order score can be a causal order score calculated for the current candidate source, which can be used to verify whether the propagation order among the burst traffic, queue growth, congestion signal, victim-rank tail, and synchronization wait corresponding to the candidate source holds true.

[0210] It should be noted that the rank skewness persistence score can be used as a metric to quantify whether a significant rank tail persists or recurs across multiple training steps. In practical applications, the device can first define the training steps. The communication completion time skew for each participating rank is as follows:

[0211] In the formula, For training steps Middle Rank Communication completion time, and These are the 95th percentile and the 50th percentile (i.e., the median), respectively. To prevent small constants with denominators of zero, the device then processes rank-skewed continuous fractions. Defined as:

[0212] In the formula, The set of training steps within the observation range. The skewness threshold is defined as follows: rank skewness persistence indicates whether a significant rank tail persists or recurs across multiple training steps, rather than whether skewness occurs sporadically in a single training step.

[0213] Furthermore, if it is also necessary to determine whether the tailing is caused by the same or similar ranks, then the training step can be defined. The tail rank set is:

[0214] And define the identity consistency of the trailing rank:

[0215] It should be understood that benign explanatory strength can be a metric used to quantify the degree to which a currently observed anomaly can be explained by benign causes (such as GPU computation tailing, data loading congestion, checkpoint input / output, link failure, routing changes, legitimate set communication convergence, or uneven expert load). In this embodiment, benign explanatory strength... It can be represented as:

[0216] in, to These represent the degree to which GPU computation tailing, data loading blockage, checkpoint input / output, link failure, routing change, legitimate set communication, and uneven expert load can explain the current anomaly. These are preset weighting coefficients.

[0217] It should also be understood that the uncertainty score can be a comprehensive indicator used to quantify path uncertainty, the degree of telemetry data missingness, the magnitude of time synchronization error, and the degree of unexplainable background load. A higher uncertainty score indicates less sufficient evidence for attribution, and even with a high positive evidence score, the final attribution confidence should be reduced. In this embodiment, the uncertainty score can consider path entropy (the dispersion of candidate path distribution), telemetry missing rate (the proportion of missing key observation data), time synchronization error (clock synchronization deviation between acquisition sources), and the degree of unexplainable background load (background traffic that cannot be attributed to any known flow).

[0218] Step S52: Determine the malicious semantic congestion score of each candidate source based on the positive evidence score and the negative evidence score.

[0219] It should be noted that the malicious semantic congestion score can be a quantitative indicator used to determine whether a candidate source is a malicious congestion source, calculated by combining positive and negative evidence for each candidate source. In this embodiment, the malicious semantic congestion score... The corresponding calculation formula can be:

[0220] In the formula, The sum of positive evidence. This represents the sum of negative evidence.

[0221] Among them, the positive evidence is:

[0222] In the formula, to The following scores are represented in order: target phase lock score, envelope violation score, target causal order score, target path overlap score, rank skew persistence score, tail rank identity consistency score, and target attack amplification score. .in, The degree to which the same candidate source causes similar anomalies when repeated across training steps.

[0223] Negative evidence includes:

[0224] In the formula, The factors considered are path entropy, telemetry missing data, time synchronization error, and the degree of unexplained background load. High path uncertainty or severe telemetry missing data will reduce the final attribution confidence, even if other scores are high.

[0225] Step S53: If the malicious semantic congestion score exceeds a preset malicious threshold, determine the candidate source as the malicious congestion event.

[0226] Understandably, the preset malicious threshold can be a pre-set critical value for the malicious semantic congestion score used to determine whether a candidate source is a malicious congestion source. It can be calibrated offline based on the target false positive rate and the maximum allowable false negative rate. For example, by collecting confirmed normal preheating training windows and labeled abnormal windows, the quantile threshold can be selected under the condition of meeting the target false positive rate and the maximum allowable false negative rate.

[0227] In its implementation, after determining that a malicious congestion event exists in the current training step, the device can further identify the specific source of the malicious congestion. Specifically, for each candidate source in the candidate source set, the device can calculate its corresponding positive evidence scores, including target stage locking score, target path overlap score, rank skew persistence score, target attack amplification score, and target causal order score. It also calculates the benign interpretation strength and uncertainty score in the negative evidence score. Subsequently, the device can sum the positive evidence scores of each candidate source according to preset weights to obtain the total positive evidence score, and sum the benign interpretation strength and uncertainty score in the negative evidence score according to preset weights to obtain the total negative evidence score. Finally, the device calculates the malicious semantic congestion score based on the total positive evidence score and the total negative evidence score. Subsequently, the device can compare the malicious semantic congestion score of each candidate source with a preset malicious threshold. If there is a candidate source whose malicious semantic congestion score is greater than or equal to the preset malicious threshold, then the candidate source can be identified as a malicious congestion source, that is, the candidate source is the source that caused the current malicious congestion event. At this time, the device can output the identification information of the malicious congestion source as the attribution and localization result of the malicious congestion event.

[0228] Furthermore, after determining that the candidate source is the malicious congestion event, the method further includes: Step S61: Determine the victimized local communication sub-event nodes affected by the malicious congestion event from the training semantic time sequence graph, and obtain the distribution of candidate path segments that have a mapping relationship with the victimized local communication sub-event nodes.

[0229] It should be understood that the victimized local communication sub-event node can be a graph node corresponding to a rank-level communication sub-event in the training semantic temporal graph that is affected by a malicious congestion event. The corresponding rank-level communication sub-event can be a local communication operation of a specific participating rank whose completion time is significantly delayed during the malicious congestion event. In this embodiment, the victimized local communication sub-event node can include attribute information such as the communication start time, communication end time, host, network interface card, RDMA queue pair used, and the number of bytes sent or received for that rank.

[0230] It should also be understood that the candidate path segment distribution can be a set of candidate network path segments and their corresponding probabilities that the communication traffic corresponding to the affected local communication sub-event may traverse. In this embodiment, the candidate path segment distribution... It can be represented as:

[0231] In the formula, and , Indicates the first Candidate path segments, This indicates that the victim's local communication sub-event uses the candidate path segment. The probability of a path is determined by factors including topology, routing tables, ECMP hash rules, switch port counts, ECN marker locations, CNP directions, PFC propagation range, host traffic matrices, and historical overlap evidence across training steps.

[0232] In practical applications, the device can search for the corresponding rank-level communication sub-event node in the node set of the training semantic temporal graph based on the training rank affected by the malicious congestion event determined in the preceding steps. It then reads the host information, network interface card information, RDMA queue pair information, and communication time window information from the rank-level communication sub-event node to determine that this node is the victimized local communication sub-event node. Next, the device can search for weighted mapping edges originating from this victimized local communication sub-event node in the edge set of the training semantic temporal graph to obtain all candidate path segments with mapping relationships to this node and their corresponding probability weights, forming a distribution of candidate path segments corresponding to the victimized local communication sub-event.

[0233] Step S62: Determine the path segment evidence score corresponding to each candidate path segment based on the congestion evidence strength on each candidate path segment in the candidate path segment distribution.

[0234] It should be noted that the congestion evidence strength can be a comprehensive quantified value of various congestion-related network anomalies observed on a candidate path segment, which can be obtained by weighting multiple congestion-related indicators on that candidate path segment. In this embodiment, the path segment... and congestion events The corresponding strength of congestion evidence It can be represented as:

[0235] in:

[0236] In the formula, Represents path segment The degree of abnormality of the queue. Represents path segment The degree of abnormality of the ECN marker on it Represents path segment The degree of CNP feedback anomaly on the surface Represents path segment The degree of PFC pause abnormality on the above Represents path segment Abnormal link utilization on the network Represents path segment The degree of link error or device malfunction on the connection. This is the weight vector corresponding to each anomaly.

[0237] It should also be noted that the path segment evidence score can be a score used to quantify the probability that a candidate path segment is the location of congestion in a malicious congestion event, and it can be calculated based on the strength of congestion evidence.

[0238] In practical applications, for each candidate path segment in the candidate path segment distribution, the device can acquire various congestion-related network anomaly signals on that path segment, and read from network telemetry data the degree of queue anomaly (e.g., the maximum queue depth of switch ports on that path segment), the degree of ECN marking anomaly (e.g., the degree to which the ECN marking ratio deviates from the normal value), the degree of CNP feedback anomaly (e.g., the abnormal increment of CNP packet rate), the degree of PFC pause anomaly (e.g., the abnormal increment of the number of PFC pause frames), the degree of link utilization anomaly (e.g., the abnormal change in link utilization), and the degree of link error or device anomaly (e.g., the abnormal increment of link error count). Then, the device can calculate the congestion evidence strength according to a preset weight vector, and map the congestion evidence strength to the [0,1] interval through a normalization function to obtain the path segment evidence score. Subsequently, the device can perform the above calculation for each candidate path segment to obtain the path segment evidence score for each candidate path segment.

[0239] Step S63: Based on the target shared bottleneck probability between each candidate source and the victim local communication sub-event corresponding to the victim local communication sub-event node, the burst traffic time location corresponding to each candidate source, and the repeated similar anomaly records of each candidate source in the historical training steps, generate attribution confidence.

[0240] It should be understood that the target shared bottleneck probability can be the probability that a candidate source and a victim local communication sub-event share the same network bottleneck resource (e.g., the same switch queue, the same egress port, or the same WAN link segment). In this embodiment, the candidate source... Communication sub-events with the victim's local area The probability of shared bottlenecks between targets It can be calculated based on the candidate path distribution corresponding to the candidate source and the candidate path segment distribution corresponding to the victim local communication sub-event. The corresponding calculation formula is as follows:

[0241] Among them, the higher the probability of target sharing bottleneck, the more likely the traffic of the candidate source and the victim local communication sub-event is to pass through the same network bottleneck resource, and the greater the possibility that the candidate source will cause congestion in the victim local communication sub-event.

[0242] It should also be understood that the burst traffic time location can be the moment or time window at which the candidate source generates the burst traffic, which can include the start time, peak time, end time of the burst traffic, and its relative position to the start and end times of the affected local communication sub-events.

[0243] Understandably, the historical training step can be one or more training steps prior to the current training step. It can be used to provide a record of the candidate source’s behavior during the historical execution process, serving as a basis for determining whether the current anomaly is a recurring similar anomaly.

[0244] It is also understood that repeated similar anomaly records can be a historical set of records in which the same candidate source has caused similar congestion anomalies multiple times in historical training steps. This can include, but is not limited to, information such as the training step number, victim rank identifier, congestion duration, and increase in training step time caused by the candidate source in historical training steps. In this embodiment, repeated similar anomaly records can be used to calculate a repetition score SR, which is the degree to which the same candidate source repeatedly causes similar anomalies across training steps. The more abundant the repeated similar anomaly records, the greater the suspicion that the candidate source is continuously causing malicious congestion.

[0245] It should be noted that attribution confidence can be a comprehensive metric used to quantify the reliability of attributing a current malicious congestion event to a candidate source. In this embodiment, attribution confidence... The corresponding calculation formula can be:

[0246] in:

[0247] In the formula, BC represents evidence of benign causes such as legitimate set communication, link failure, and ECMP polarization, while U represents path entropy, telemetry missing data, and time error. The preset weighting coefficients are used. The device can determine the final attribution output granularity based on the attribution confidence level. If the confidence level meets the queue pair level threshold, the source queue pair is output; if it only meets the host level threshold, the source host is output; if the evidence is insufficient, "suspicious but cannot be attributed" is output.

[0248] Step S64: Generate the malicious congestion location result of the current distributed training task based on the attribution confidence.

[0249] It should be understood that malicious congestion localization results can be output information indicating the specific source location of a malicious congestion event. This information may include attribution conclusions (indicating whether the malicious congestion event has been successfully attributed to a specific candidate source), localization granularity level (indicating the granularity level of the attribution result, including at least one of RDMA queue pair level, host level, tenant level, rack level, switch port level, and WAN segment level), localization entity identifier (indicating the identifier information of the specific entity located as the attack source, such as queue pair number, host identifier, tenant identifier, rack number, switch port number, or WAN segment identifier), and attribution confidence (indicating the confidence value of the localization result). In practical applications, the granularity of the malicious congestion localization result depends on the sufficiency of the evidence. The more sufficient the evidence, the finer the localization granularity; insufficient evidence results in a coarser-grained output or a "cannot be attributed" conclusion.

[0250] In practical applications, the device can acquire the candidate path distribution of candidate sources and the candidate path segment distribution of affected local communication sub-events to calculate the target-shared bottleneck probability between candidate sources and affected local communication sub-events. Simultaneously, the device can read the start time, peak time, and end time of burst traffic from candidate sources from network telemetry data, and obtain the communication start time and end time of affected local communication sub-events. It then verifies whether the burst traffic time location satisfies the following conditions: burst traffic end time < communication end time (i.e., burst traffic occurred before the affected communication tail appears), and whether the burst traffic peak time falls near the affected communication time window, thus obtaining the burst traffic time location corresponding to the candidate source. Furthermore, the device can query historical records of congestion anomalies caused by candidate sources in previous training steps, statistically analyzing the frequency of similar anomalies, the consistency of affected rank, and the consistency of increased training step time, to obtain repeated similar anomaly records of candidate sources in historical training steps. Finally, based on the target-shared bottleneck probability, the rationality of the burst traffic time location, and the repeated similar anomaly records, the device can calculate the attribution confidence score and apply the calculated attribution confidence score to the network. Compared with the preset confidence threshold (including the queue pair level confidence threshold) Host-level confidence threshold Rack-level confidence threshold Wide-area link segment-level confidence threshold Compare with (etc.), if If the positioning granularity is determined to be at the source queue pair level, then the source queue pair identifier corresponding to the candidate source is output; if but If the location granularity is determined to be at the source host level, then the source host identifier corresponding to the candidate source is output; if but If the positioning granularity is determined to be at the rack level, then the rack identifier corresponding to the candidate source is output; if but If so, then the wide area link segment identifier corresponding to the candidate source is output. If the location is below all preset thresholds, the output will be "suspicious but cannot be attributed", along with the coarsest granular location information that can be determined at present and the attribution confidence value.

[0251] In this embodiment, by determining the victim local communication sub-event nodes from the training semantic temporal graph and obtaining the distribution of candidate path segments, and determining the path segment evidence score based on the congestion evidence strength on each candidate path segment, and then generating attribution confidence based on the target shared bottleneck probability, the time location of the burst traffic and historical repeated similar anomaly records, the credibility of the candidate source can be comprehensively evaluated from three dimensions: path overlap degree, temporal causality and historical behavior consistency, which is conducive to improving the accuracy of malicious congestion location.

[0252] In the specific implementation, refer to Figure 4 , Figure 4 This is a flowchart of the reverse congestion tracing and reverse tracking direction in the congestion detection method of this application. Figure 4The left side represents the forward propagation direction from the source to the victim, and the right side represents the reverse tracing direction from the victim to the source. In the forward propagation direction, the malicious congestion propagation process sequentially goes through six stages. Stage 1 involves a burst of traffic generated by the source queue pair, source host, or source user. The average rate of this burst may be low, but its timing is precisely aligned with the critical set communication window in distributed training. Stage 2 involves the burst propagating through the network, passing through congestion queues, physical links, or wide-area link segments, and starting to form queue accumulation at network resources. Stage 3 involves the switch detecting that the queue growth exceeds a preset threshold and triggering a congestion notification mechanism, generating congestion signals such as ECN tags, CNP messages, or increased queuing delay. Congestion signals are transmitted through the network along with the data stream. In stage 4, the affected wide-area RDMA network card receives the congestion signal, triggering the congestion counting and rate adjustment mechanisms on the network card side, and the congestion effect begins to propagate from the network layer to the endpoint. In stage 5, the affected training nodes participating in ensemble communication perceive a communication tail delay, meaning that the completion time of the node's rank-level communication sub-event is significantly delayed relative to other participating ranks. In stage 6, the tail delay propagates to the entire training step through the synchronization barrier mechanism of ensemble communication, causing an increase in the total time of the training step, and the local communication tail of the minority ranks is amplified into a global decrease in training performance. In the reverse tracing direction, the device can start from the final observable result of the increased training step time and trace back step by step in the reverse time direction to execute a complete attribution and tracing process. First, the device can detect a significant increase in training step time in stage 6 and, through the aforementioned envelope violation detection and malicious congestion score determination, confirm that the increase in training step time is caused by a malicious congestion event. Next, the device can identify the victimized local communication sub-event node from the training semantic temporal graph, obtain the communication tail delay information of the node, and enter stage 5 to locate the affected set of communication events and the specific victimized rank and victimized network interface card. Then, the device can start from the weighted mapping edge of the victimized local communication sub-event node and trace in the reverse direction to stage 4 to determine the distribution of candidate path segments at the ingress port, and calculate the path segment evidence score based on the congestion evidence strength on each candidate path segment. Subsequently, the device can continue to trace in the reverse direction from the ingress port to stage 3 to identify the switch queues that have congestion notifications and flow control signals, and verify whether the causal order between the burst traffic occurrence time, queue growth time, congestion signal occurrence time, and victimized rank tail occurrence time holds. After that, the device can trace in the reverse direction from the switch queues to stage 2 to determine the upstream queue or wide area link segment, and based on the target shared bottleneck probability between each candidate source and the victimized local communication sub-event, filter out the candidate source most likely to share the bottleneck.Ultimately, in Phase 1, the device can determine the source queue pair, source host, or source user. Based on the burst traffic time location corresponding to each candidate source (i.e., whether the burst traffic occurred before the victim communication tail) and repeated similar anomaly records in historical training steps, it generates attribution confidence. If the attribution confidence exceeds a preset threshold, it outputs the final location result. In practical applications, the reverse tracing direction is opposite to the propagation direction of malicious congestion. That is, the device can start from the final result of increased training step time to verify whether the complete congestion propagation chain of burst → queue growth → congestion signal → victim rank tail → synchronization waiting is valid, and trace back step by step along the propagation chain to ultimately achieve complete attribution tracing from the victim end to the source.

[0253] It should be noted that, referring to Figure 5 , Figure 5 This is the overall flowchart of the congestion detection method in this application. Figure 5 This demonstrates the complete implementation process from data acquisition to final location conclusion output. The process is divided into five levels from top to bottom: data source, semantic modeling, observation modeling, inference and judgment, and result output, along with the core logic of malicious congestion detection and tracing that runs through multiple levels. At the data source level, the device can collect raw data from the training framework and network devices. At the semantic modeling level, the device can collect training semantic data such as training step number, stage, layer, micro-batch, ensemble communication type, participation rank set, message size, and queue pairs from the training framework and ensemble communication library, and extract the computational events and ensemble communication events corresponding to each training node in each training step from the training semantic data. In the time series construction stage, the device can perform cross-source time alignment between training semantic events and network telemetry events according to a time error window, and construct a training semantic time series graph based on the training semantic data. At the observation modeling level, the device can determine the semantic context corresponding to each set of communication events based on the training semantic time sequence diagram, and divide the network observation indicators in the network telemetry data into several envelope buckets based on the preset indicator condition distribution and semantic context. Then, based on the historical sample set in each envelope bucket, a robust statistical method is used to determine the normal value range of the network observation indicators under the corresponding semantic conditions, which serves as the probabilistic semantic network envelope.

[0254] At the reasoning and judgment level, the device can determine the envelope violation score based on the deviation between real-time network observation indicators and the probabilistic semantic network envelope under corresponding semantic conditions. If the envelope violation score exceeds a first preset threshold, a network congestion event is determined to exist. Subsequently, the device can determine the phase locking score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event. Combining these scores with the envelope violation score, a malicious congestion score is determined. If the malicious congestion score exceeds a second preset threshold, a malicious congestion event is determined to exist. After confirming the existence of a malicious congestion event, the device can perform multi-hypothesis diagnosis and candidate source localization. Specifically, the device can calculate the positive evidence score and negative evidence score for each candidate source in the candidate source set, and combine the positive and negative evidence to determine the malicious semantic congestion score for each candidate source. If the malicious semantic congestion score exceeds a preset malicious threshold, the candidate source is determined to be a malicious congestion source. Next, the device can determine the victim local communication sub-event nodes and obtain the distribution of candidate path segments from the training semantic time sequence graph. It determines the path segment evidence score based on the congestion evidence strength on each candidate path segment, and generates attribution confidence based on the target shared bottleneck probability between each candidate source and the victim local communication sub-event, the time location of the burst traffic, and the repeated similar anomaly records in the historical training steps. Then, it determines the final localization granularity based on the level of attribution confidence. Specifically, if the confidence at the queue pair level meets the threshold, the source queue pair is output; if only the confidence at the host level meets the threshold, the source host is output; if only the congested area can be determined, the rack, switch port, or wide area link segment is output; if the evidence is insufficient, "suspicious but cannot be attributed" is output. Finally, at the result output level, the device can output detection results, location results, and evidence results. Among them, the detection results include three states: "benign and explained", "suspicious but not attributed", or "malicious and attributed". The location results include location information of different granularities such as source queue pairs, source hosts, source tenants, racks, switch ports, or wide area link segments. The evidence results include various quantitative evidence such as stage lockout score, envelope violation score, path overlap score, rank skew persistence score, causal order score, attack amplification score, benign explanation strength, malicious semantic congestion score, and attribution confidence.

[0255] It should also be noted that, referring to Figure 6 , Figure 6 This is a system block diagram for the congestion detection method of this application. (See diagram below.) Figure 6As shown, the congestion detection process in this application can be completed collaboratively by five core modules: a training semantic modeling module, a telemetry and event alignment module, a probabilistic semantic network envelope modeling module, a multi-hypothesis diagnosis and candidate source generation module, and a reverse tracing module. In the initial stage of the process, the training semantic modeling module can obtain training semantic data from multiple data sources such as the training framework, performance analyzer, and eBPF events. Then, the training semantic modeling module can extract the semantic context of communication stages and communication events of each set based on the training semantic data, construct a training semantic time sequence diagram, and output the constructed training semantic time sequence diagram to the probabilistic semantic network envelope modeling module. Simultaneously, the telemetry and event alignment module can collect network telemetry data from switches, WAN RDMA network cards, and host agents. Then, the telemetry and event alignment module can align the collected network telemetry data with training semantic events and output the aligned network evidence to the probabilistic semantic network envelope modeling module and the multi-hypothesis diagnosis and candidate source generation module. After receiving the training semantic time sequence diagram from the training semantic modeling module and the aligned network evidence from the telemetry and event alignment module, the probabilistic semantic network envelope modeling module can determine the semantic context corresponding to each set of communication events based on the training semantic time sequence diagram. Based on the preset index condition distribution and the above semantic context, it divides the network observation index in the network telemetry data into several envelope buckets. Then, based on the historical sample set in each envelope bucket, it determines the normal value range of the network observation index under the corresponding semantic conditions, establishes the probabilistic envelope relationship between the training semantics and the network evidence, and then outputs the established probabilistic semantic network envelope to the multi-hypothesis diagnosis and candidate source generation module.

[0256] Furthermore, the multi-hypothesis diagnosis and candidate source generation module can receive aligned network evidence from the telemetry and event alignment module and probabilistic semantic network envelopes from the probabilistic semantic network envelope modeling module. Based on the probabilistic semantic network envelopes, it detects whether malicious congestion events exist in each training step. After confirming the existence of malicious congestion events, it can generate congestion cause hypotheses and filter candidate congestion sources, and output the determined malicious congestion sources and related candidate source information to the reverse tracing module. After receiving candidate source information from the multi-hypothesis diagnosis and candidate source generation module and network evidence from the telemetry and event alignment module, the reverse tracing module can determine the affected local communication sub-event nodes affected by malicious congestion events from the training semantic time sequence graph and obtain the distribution of candidate path segments. It determines the path segment evidence score based on the congestion evidence strength on each candidate path segment, and then infers the congestion propagation chain along the probabilistic path based on the target shared bottleneck probability between each candidate source and the affected local communication sub-event, the burst traffic time location, and repeated similar anomaly records in historical training steps. Specifically, this module can verify whether the time and location of the burst traffic of the candidate source is earlier than the time of the tail of the victim communication, whether there is a shared bottleneck path between each candidate source and the victim communication, and whether the candidate source has repeatedly caused similar anomalies in the historical training steps. Then, it integrates this information to generate attribution confidence and determines the final localization granularity based on the level of attribution confidence, and outputs the confidence-aware source tracing result.

[0257] In this embodiment, by determining the positive and negative evidence scores of each candidate source in the candidate source set, and using target stage locking score, path overlap score, rank skew persistence score, attack amplification score, and causal order score to comprehensively quantify the malicious suspicion of the candidate sources from multiple dimensions, benign anomalies and unreliable factors are eliminated by using benign interpretation strength and uncertainty score. Then, by determining the malicious semantic congestion score of each candidate source based on positive and negative evidence, a comprehensive weighing of the positive and negative evidence for each candidate source is achieved, avoiding misjudgment caused by accidental time overlap. This makes the determination of malicious sources have clear quantitative basis and multi-dimensional evidence support, thereby improving the accuracy of identifying and locating malicious congestion sources.

[0258] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the congestion detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0259] This application also provides a congestion detection device, please refer to... Figure 7 The congestion detection device includes: The data acquisition module 10 is used to acquire training semantic data and network telemetry data of the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task. The network telemetry data is used to describe the network state during the operation of the current distributed training task. The timing graph construction module 20 is used to construct a training semantic timing graph of the current distributed training task based on the training semantic data. The training semantic timing graph is used to describe the relationship between the computation events, the collective communication events and the network resources in each training step of the current distributed training task. The envelope determination module 30 is used to determine the probabilistic semantic network envelope corresponding to the network observation index in the network telemetry data under different semantic conditions based on the training semantic time sequence diagram. The probabilistic semantic network envelope is used to describe the normal value range of the network observation index under each semantic condition. The congestion detection module 40 is used to detect whether there is a malicious congestion event in each training step based on the probabilistic semantic network envelope, and to obtain the network congestion detection result of the current distributed training task.

[0260] The congestion detection device provided in this application, employing the congestion detection method described in the above embodiments, can solve the technical problem that existing congestion detection methods struggle to distinguish between malicious congestion and benign anomalies during training alignment, leading to a high risk of missed and false alarms. Compared with the prior art, the beneficial effects of the congestion detection device provided in this application are the same as those of the congestion detection method provided in the above embodiments, and other technical features in the congestion detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0261] This application provides a congestion detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the congestion detection method in Embodiment 1 above.

[0262] The following is for reference. Figure 8The diagram illustrates a structural schematic suitable for implementing the congestion detection device in the embodiments of this application. The congestion detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The congestion detection device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0263] like Figure 8 As shown, the congestion detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the congestion detection device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the congestion detection device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows congestion detection devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0264] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0265] The congestion detection device provided in this application, employing the congestion detection method in the above embodiments, can solve the technical problem of congestion detection. Compared with the prior art, the beneficial effects of the congestion detection device provided in this application are the same as those of the congestion detection method provided in the above embodiments, and other technical features of the congestion detection device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0266] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0267] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0268] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the congestion detection method in the above embodiments.

[0269] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0270] The aforementioned computer-readable storage medium may be included in the congestion detection device; or it may exist independently and not be assembled into the congestion detection device.

[0271] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the congestion detection device, the congestion detection device performs the following actions: acquires training semantic data and network telemetry data for the current distributed training task. The training semantic data characterizes computational events and aggregated communication events corresponding to multiple training nodes in the current distributed training task, and the network telemetry data describes the network state during the operation of the current distributed training task. Based on the training semantic data, it constructs a training semantic time sequence diagram for the current distributed training task. This diagram describes the relationships between computational events, aggregated communication events, and network resources in each training step of the current distributed training task. Based on the training semantic time sequence diagram, it determines the probabilistic semantic network envelopes corresponding to network observation indicators in the network telemetry data under different semantic conditions. These probabilistic semantic network envelopes describe the normal value range of the network observation indicators under each semantic condition. Based on the probabilistic semantic network envelopes, it detects whether malicious congestion events exist in each training step, thereby obtaining the network congestion detection result for the current distributed training task.

[0272] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0273] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0274] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0275] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the congestion detection method described above. This addresses the technical problem that existing congestion detection methods struggle to distinguish between malicious congestion and benign anomalies during training alignment, leading to a high risk of missed and false alarms. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the congestion detection method provided in the above embodiments, and will not be elaborated upon here.

[0276] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method of congestion detection, characterized by, The method includes: Acquire training semantic data and network telemetry data for the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task. The network telemetry data is used to describe the network state during the operation of the current distributed training task. Based on the training semantic data, a training semantic time sequence diagram of the current distributed training task is constructed. The training semantic time sequence diagram is used to describe the relationship between the computation events, the collective communication events and the network resources in each training step of the current distributed training task. Based on the training semantic time sequence diagram, the probabilistic semantic network envelopes corresponding to different semantic conditions of the network observation indicators in the network telemetry data are determined. The probabilistic semantic network envelopes are used to describe the normal value range of the network observation indicators under each semantic condition. The envelope violation score of each training step is determined based on the degree of deviation between the real-time network observation index in each training step and the probabilistic semantic network envelope under the corresponding semantic conditions, and the presence of network congestion events in each training step is detected based on the envelope violation score. In the presence of the network congestion event, the network congestion detection result of the current distributed training task is obtained by detecting whether there is a malicious congestion event in each training step based on the malicious congestion score corresponding to the network congestion event. The step of determining the probabilistic semantic network envelope corresponding to different semantic conditions of the network observation indicators in the network telemetry data based on the training semantic time sequence graph includes: The semantic context corresponding to each set of communication events in the current distributed training task is determined based on the training semantic time sequence diagram. The semantic context includes at least one of the following: training location information, communication attributes, participating resources, candidate path distribution, message structure, and communication criticality. Based on the preset index condition distribution and the semantic context, the network observation indexes in the network telemetry data are divided into several envelope buckets; Based on the historical sample set, determine the normal value range of the network observation index in each envelope bucket under the corresponding semantic conditions, and define the normal value range as the probabilistic semantic network envelope; The step of detecting whether a malicious congestion event exists in each training step based on the malicious congestion score corresponding to the network congestion event in the presence of the network congestion event includes: If the envelope violation score exceeds a first preset threshold, it is determined that a network congestion event exists in the corresponding training step; Determine the phase lock score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event; The malicious congestion score corresponding to the network congestion event is determined based on the envelope violation score, the phase lock score, the causal order score, the path overlap score, and the attack amplification score. If the malicious congestion score exceeds a second preset threshold, a malicious congestion event is determined to exist in the training step.

2. The method as described in claim 1, characterized in that, The step of constructing the training semantic time sequence graph of the current distributed training task based on the training semantic data includes: Based on the training semantic data, determine the training computation event node, set communication event node, and collaboration node corresponding to each training node in each training step of the current distributed training task; Determine the dependencies between the computation events corresponding to the training computation event nodes, the set communication events corresponding to the set communication event nodes, and the collaborative events corresponding to the collaborative nodes; Based on the aforementioned dependency relationships and preset dependency edge establishment rules, directed dependency edges are established between the training computation event nodes, the set communication event nodes, and the collaborative nodes. The training semantic time sequence graph of the current distributed training task is constructed based on the training computation event nodes, the set communication event nodes, the collaborative nodes, and the directed dependency edges.

3. The method as described in claim 2, characterized in that, The step of constructing the training semantic time sequence graph of the current distributed training task based on the training computation event nodes, the set communication event nodes, the collaborative nodes, and the directed dependency edges includes: Determine the start time and completion time of each of the aforementioned collective communication events; Based on the event start time and the event completion time, each of the set communication events is expanded into several rank-level communication sub-events that are executed by the respective rank; Based on the candidate path distribution, a weighted mapping edge is established between each rank-level communication sub-event and the candidate path segment; Determine the shared bottleneck probability between each pair of the computational event, the rank-level communication sub-event, and the cooperative event; The training semantic time sequence graph of the current distributed training task is constructed based on the training computation event nodes, the rank communication sub-events, the collaborative nodes, the directed dependency edges, the weighted mapping edges, and the shared bottleneck probabilities.

4. The method as described in claim 1, characterized in that, After the step of detecting whether a malicious congestion event exists in each training step based on the malicious congestion score corresponding to the network congestion event, and obtaining the network congestion detection result of the current distributed training task, the method further includes: Determine the positive evidence score and negative evidence score of each candidate source in the candidate source set of the current distributed training task. The positive evidence score includes the target stage locking score, target path overlap score, rank skew persistence score, target attack amplification score, and target causal order score. The negative evidence score includes benign interpretation strength and uncertainty score. The malicious semantic congestion score of each candidate source is determined based on the positive evidence score and the negative evidence score; If the malicious semantic congestion score exceeds a preset malicious threshold, the candidate source is determined to be the malicious congestion event.

5. The method as described in claim 4, characterized in that, After determining that the candidate source is the malicious congestion event, the method further includes: The victimized local communication sub-event nodes affected by the malicious congestion event are determined from the training semantic temporal graph, and the distribution of candidate path segments that have a mapping relationship with the victimized local communication sub-event nodes is obtained. The path segment evidence score corresponding to each candidate path segment is determined based on the congestion evidence strength on each candidate path segment in the candidate path segment distribution. Based on the target sharing bottleneck probability between each candidate source and the victim local communication sub-event corresponding to the victim local communication sub-event node, the burst traffic time location corresponding to each candidate source, and the repeated similar anomaly records of each candidate source in the historical training steps, an attribution confidence score is generated. The malicious congestion location result of the current distributed training task is generated based on the attribution confidence.

6. A congestion detection device, characterized in that, The device includes: The data acquisition module is used to acquire training semantic data and network telemetry data of the current distributed training task. The training semantic data is used to characterize the computation events and aggregate communication events corresponding to multiple training nodes in the current distributed training task, and the network telemetry data is used to describe the network state during the operation of the current distributed training task. The timing graph construction module is used to construct a training semantic timing graph of the current distributed training task based on the training semantic data. The training semantic timing graph is used to describe the relationship between the computation events, the collective communication events and the network resources in each training step of the current distributed training task. An envelope determination module is used to determine the probabilistic semantic network envelope of the network observation index in the network telemetry data under different semantic conditions based on the training semantic time sequence diagram. The probabilistic semantic network envelope is used to describe the normal value range of the network observation index under each semantic condition. The congestion detection module is used to determine the envelope violation score of each training step based on the degree of deviation between the real-time network observation index in each training step and the probabilistic semantic network envelope under the corresponding semantic conditions, and to detect whether there is a network congestion event in each training step based on the envelope violation score. The congestion detection module is further configured to, in the presence of the network congestion event, detect whether there is a malicious congestion event in each training step based on the malicious congestion score corresponding to the network congestion event, and obtain the network congestion detection result of the current distributed training task. The envelope determination module is further configured to determine the semantic context corresponding to each set of communication events in the current distributed training task based on the training semantic time sequence diagram. The semantic context includes at least one of training location information, communication attributes, participating resources, candidate path distribution, message structure, and communication criticality. Based on the preset index condition distribution and the semantic context, the module divides the network observation indicators in the network telemetry data into several envelope buckets. Based on the historical sample set, the module determines the normal value range of the network observation indicators in each envelope bucket under the corresponding semantic conditions, and determines the normal value range as the probabilistic semantic network envelope. The congestion detection module is further configured to: determine if a network congestion event exists in the corresponding training step when the envelope violation score exceeds a first preset threshold; determine the phase lock score, causal order score, path overlap score, and attack amplification score corresponding to the network congestion event; determine the malicious congestion score corresponding to the network congestion event based on the envelope violation score, the phase lock score, the causal order score, the path overlap score, and the attack amplification score; and determine if a malicious congestion event exists in the training step when the malicious congestion score exceeds a second preset threshold.

7. A congestion detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the congestion detection method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the congestion detection method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Message forwarding method based on semantic perception, electronic equipment and storage medium

    CN121967293A

  • Congestion prediction method of data center network, electronic equipment and storage medium

    CN122226629A