RDMA (Remote Direct Memory Access) network performance abnormity diagnosis method under set communication

Abstract

The invention provides an RDMA network performance abnormity diagnosis method under set communication. In a collective communication process, firstly, a monitor located at a server side executes algorithm disassembly in advance, monitors related performance information of a collective communication flow in real time according to steps, and reports the related performance information to an analysis server; the monitors inform the servers of waiting relations by sending Notification packets among the monitors, adaptively execute an appropriate anomaly detection triggering process when performance anomaly occurs, send a polling query data packet to the switches through the servers, trigger telemetry information collection on all the switches related to the anomaly, and report the telemetry information collection to an analysis server in sequence; and finally, the analysis server constructs a waiting relation graph and a network traceability graph, and comprehensively analyzes the waiting relation graph and the network traceability graph to give a diagnosis result. The problems that in a set communication scene, existing analysis modes such as a network traceability graph cannot give the overall performance bottleneck of set communication and cannot give contribution evaluation of flow to overall set communication performance abnormity and the like are solved.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically, to a method for diagnosing performance anomalies in RDMA networks under aggregated communication. Background Technology

[0002] Unified Communication and RDMA Technology: Collective communication is a global communication operation involving all processes in a process group. With the rapid growth in the size of large language models, collective communication has become a crucial factor determining the efficiency of data exchange and parallel computing between GPUs. Collective communication often utilizes RDMA networks for inter-machine interconnection to achieve high throughput and low latency.

[0003] RDMA (Remote Direct Memory Access) technology integrates the network protocol stack into a network interface card (RNIC) specifically designed for RDMA. This allows remote access to the target server's memory, bypassing the CPU and operating system, without requiring any additional data copying. With the evolution of data center network infrastructure, more and more clusters are transitioning from InfiniBand (IB) networks to more flexible high-speed Ethernet, typically using the RoCEv2 (RDMA over Converged Ethernet version 2) protocol for data exchange, especially in areas such as large language model training. For example, Meta's 16,000 GPU cluster uses RoCEv2 for training. To ensure high performance, RoCEv2 typically uses Priority Flow Control (PFC) to achieve lossless networking, combined with congestion control algorithms (such as DCQCN and Swift) to manage flow control and ensure transmission efficiency.

[0004] Complexity of aggregated communication networks: The network complexity of aggregated communication mainly includes two aspects: (1) the multi-stream mode determined by the aggregated communication algorithm; and (2) the complexity of the RDMA mechanism.

[0005] Multi-stream mode: Aggregate communication often employs specific algorithms (such as Ring, Halving, and Doubling) to improve communication efficiency, thus adding more complexity to network anomaly diagnosis. First, unlike single-stream applications, aggregate communication requires the collaboration of multiple streams from multiple nodes. Second, aggregate communication changes dynamically during execution. The corresponding streams differ at different algorithm stages. Third, there are data dependencies between streams. Server B may only initiate stream B to transmit data to other nodes after receiving data from stream A.

[0006] RDMA Mechanism: Unlike traditional TCP networks, RDMA's line-rate start and PFC mechanisms are more prone to network congestion. First, in RoCEv2, RDMA requires PFC flow control to achieve lossless networking and ensure superior transmission performance. Under the PFC mechanism, when the switch queue exceeds a certain threshold, it sends a PAUSE frame to the upstream device. The upstream device then stops sending on that link until it receives a RESUME frame. However, PFC can have a backpressure effect when encountering network congestion, cascading PAUSE frames upstream and causing widespread network congestion. Second, RDMA does not have a slow start process like TCP; instead, it sends at line rate from the beginning. Considering the shallow buffer of commercial RDMA switches, this makes network congestion more frequent and shorter-lived.

[0007] abnormal performance of the aggregated communication network: Network performance anomalies are complex and varied. The two most basic types include: 1) Flow collisions. In large-scale clusters, many tasks share the same infrastructure, making flow collisions inevitable. 2) PFC storms. PFC storms refer to persistent PFC injections caused by hardware errors or other reasons, leading to widespread PFC propagation and resulting in network-wide performance anomalies.

[0008] In real-world network environments, flow collisions and PFC propagation often occur together. Depending on their causes or characteristics, other common anomalies include: 1) Load imbalance. Due to misjudgments in load balancing strategies, such as ECMP (Equal-Cost Multi-Path Routing), aggregated communication or other flows cannot ideally distribute traffic according to the network topology, leading to collisions between different flows. 2) Loops. Due to asynchrony between switches during network updates, some switches may form a forwarding loop. A flow may be forwarded from a switch port, then return to that port after multiple hops. 3) PFC backpressure caused by flow collisions. PFC propagation caused by the offending flow can even affect flows that do not share the propagation path with it, which is a unique characteristic of PFC-related anomaly diagnosis. 4) PFC deadlock. When PFC propagation forms a loop, flows at all loop positions are stopped, resulting in a deadlock.

[0009] The shortcomings of existing network performance anomaly diagnosis systems: Basic network performance diagnostic techniques include anomaly detection, data collection, and diagnostic algorithms.

[0010] A basic technique involves collecting telemetry information across all switches. This method eliminates the need for anomaly detection, and while it ensures accuracy by continuously collecting data across all relevant switches, it introduces significant communication or analysis overhead.

[0011] SpiderMon uses packet accumulation queue latency for anomaly detection. Upon detecting an anomaly, SpiderMon collects telemetry data from switches related to the abnormal flow and designs an algorithm to construct a relationship graph based on the contributions of different flows for diagnosis. However, this diagnostic method is more suitable for flow-level anomaly diagnosis in traditional TCP networks and still has limitations for flow-level diagnosis in RDMA networks. Specifically, due to the unique PFC flow control mechanism in RDMA networks, network congestion at one point may affect the performance of other flows unrelated to the congestion location through PFC propagation, making accurate root cause localization difficult for SpiderMon.

[0012] Hawkeye addresses SpiderMon's shortcomings by employing PFC awareness. It triggers anomaly detection by recognizing RTT timeouts and collects telemetry data from switches associated with the affected flow. Simultaneously, it traces telemetry data along the PFC propagation path to achieve PFC awareness in RDMA networks. However, this approach still has limitations in aggregated communication scenarios. For example, when Hawkeye is used for anomaly diagnosis in aggregated communication, its manually configured and somewhat mechanical anomaly detection triggering mechanism can lead to repeated anomaly detections on multiple communication nodes within a short period, collecting redundant data and introducing significant overhead.

[0013] Although these methods have some advantages, such as the ability to collect telemetry data related to abnormal flow performance and high accuracy, they still have many shortcomings when facing the collective communication scenario. They cannot accurately and efficiently diagnose RDMA network anomalies in collective communication. The following is a summary: (1) First, existing network performance anomaly diagnosis algorithms mainly focus on the single-flow level (e.g., Hawkeye, SpiderMon) in terms of analysis objects, ignoring emerging paradigms such as collective communication, resulting in insufficient analysis capabilities for phased group flows. Unlike the static process at the single-flow level, in the process of collective communication, not only may there be competition and waiting relationships between multiple flows, but the flows in the network may also change over time. For example, such as Figure 1As shown, in the Halving and Doubling algorithm, if the sender is node i and the flow in step j is called FiSj, then the target of the flow changes from F0S1 to F0S2, and the start of F0S2 depends on the completion of F4S1. Existing work lacks accurate capture of this dynamic feature. (2) Secondly, since set communication involves a large number of nodes and flows, reducing overhead while accurately diagnosing is also an important challenge. Summary of the Invention

[0014] The purpose of this disclosure is to provide a method for diagnosing RDMA network performance anomalies in aggregated communication scenarios. This method aims to address the following issues: RDMA network diagnosis in aggregated communication scenarios requires a global view of phased group flows and the ability to capture dynamic changes in inter-flow dependencies during aggregated communication, thereby achieving root cause localization; simple RTT timeout diagnosis triggering mechanisms lead to significant redundant overhead; existing network source graph analysis methods can only assess the contribution of flows to local anomalies, failing to identify overall performance bottlenecks in aggregated communication or assess the contribution of flows to overall aggregated communication performance anomalies.

[0015] In general, a method for diagnosing RDMA network performance anomalies under aggregated communication is provided. In the aggregated communication process, firstly, the monitor located on the server side pre-executes the algorithm decomposition and monitors the relevant performance information of the aggregated communication flow in real time step by step, and reports it to the analysis server. The monitors communicate their waiting relationships with the servers by sending Notification packets. When a performance anomaly occurs, they adaptively execute an appropriate anomaly detection triggering process and send polling query data packets to the switches via the servers to trigger the collection of telemetry information on all switches related to the anomaly, which is then reported to the analysis server in sequence. Finally, the server constructs a waiting relationship graph and a network tracing graph, and a comprehensive analysis is conducted to provide diagnostic results.

[0016] The algorithm decomposition process involves breaking the algorithm down into multiple steps and using these steps to describe different stages of different flows in the set communication. For a flow originating from a specific node, the data block transmitted by the flow or the target address of the flow will change between each step.

[0017] The process of triggering the anomaly detection is as follows: A step-aware, dynamic anomaly detection mechanism is employed to limit unnecessary detection triggers, automatically select appropriate detection parameters, and combine end-side monitoring and network telemetry with collaborative data collection to accurately and efficiently support aggregated communication-level diagnostics. Specifically, First, an end-to-end network combined anomaly data collection system monitors and aggregates communication performance information on the server side. Upon detecting performance anomalies, it adaptively triggers network telemetry to obtain network information recorded in the switches for anomaly diagnosis. During server-side monitoring, each server's monitor collects performance information of the current flow. After each flow execution step is completed, the server reports the five-tuple, the amount of data transmitted, the start time, the end time, and the source address of the waiting flow. When the server detects a performance anomaly, it will send polling packets for telemetry collection through anomaly detection. During telemetry collection, the switches periodically record flow-level telemetry data and port-level telemetry data, and sense the port PFC status. Upon receiving a polling packet, the switch data plane will propagate the polling packet according to the flow path and PFC propagation path. Simultaneously, the switch controller assists in data collection and reports the telemetry information to the analysis server. During the anomaly detection process, the step-aware design is as follows: the number of anomaly detection triggers is determined by configurable parameters, and the triggering opportunities are evenly distributed according to the estimated completion time of the flow during the execution of the aggregated communication. The dynamic configuration of anomaly detection trigger conditions is as follows: Configure the RTT threshold amplification factor and update the RTT threshold setting of the stream before each step of each stream in the aggregated communication; The adaptive design method in anomaly detection is as follows: if a certain step of flow F1 is completed, but the next step of F1 depends on flow F2 is not completed, if F1 still has anomaly detection opportunities, then the server of F1 will send a customized packet to pass the opportunity to the server of F2, thereby increasing the detection opportunities of F2.

[0018] The specific method for the analysis server to construct the waiting relationship graph is as follows: During actual operation, the analysis server arranges the collected data entries according to the flow completion time order and constructs the waiting graph in sequence. Specifically, based on the real-time collected waiting relationship data and timestamps, the analysis server constructs corresponding nodes and edges. For simplicity, after determining that a certain node has an in-degree of 0, the analysis server can recursively prune the nodes with an in-degree of 0. After constructing the complete waiting graph, the analysis server calculates the performance bottleneck of the critical path representation set communication and diagnoses the network root cause corresponding to the network source graph of each critical flow.

[0019] The specific method for constructing the network source graph is as follows: A source graph is constructed based on the packet-level wait relationships between flows and ports, and the graph's point set... It includes stream F, port P, and collection communication stream CF, where and The set of directed edges of a graph The waiting relationships between nodes can be described in the following three categories: (1) : Describes the waiting relationship between a flow and a port, using the queue depth of the flow waiting at the port as the edge weight. Assume the flow... A package Enter the queue and define the flow ahead of it. The number of packages is Then flow right The waiting weight is defined as The weight of the flow to the port is (2) This describes the waiting relationship between a port and a flow. These edges quantify the contribution of a flow to port congestion, using the number of packets contributed by the flow within the port's queue depth as the edge weight. Assuming the port... The queue depth is The number of packets detected within a certain period is Then the weight of the flow at the port is (3) This describes the wait relationship between ports. This type of edge is used to capture the causal relationship of Priority Flow Control (PFC). When Priority Flow Control (PFC) occurs, the downstream switch's outgoing port... Stop the upstream switch output port Therefore, it constitutes arrive Waiting is regarded as wait All packets in the queue, for accurate evaluation arrive Contribution percentage along the path, defined , The weight between them is the traffic ratio. In summary, through , , These three types of edge sets belong to the graph The directed edges and their weights are defined respectively. , , The analyzer constructs a source graph to assess the interaction between flows and ports.

[0020] The specific method of the comprehensive analysis is as follows: diagnose the specific root cause based on the characteristics of different anomaly types, including flow conflict and PFC back pressure; The characteristics of the flow conflict anomaly type are: Convection collisions are identified by examining port nodes in the network source graph. When the source graph shows... When considering the characteristic structure of the flow, it is believed that the flow... With collection communication flow At the port A conflict occurred there.

[0021] The characteristics of the PFC back pressure type are as follows: PFC back pressure is identified by tracing the PFC propagation path. When the source map appears... This indicates that PFC propagation has occurred and affected aggregate communication. Following the PFC propagation path shown in the diagram, the root cause of the PFC can be located.

[0022] The contribution score for the flow conflict is: To evaluate the contribution of other flows to a given set of communication flows: First, obtain the maximally connected subgraph of the network source graph starting from the given set of communication flows. Then all... All flows are included in the scoring object. For each flow, the contribution of the flow to that node is calculated step by step in the direction of the reverse directed edge being waited for. Assume the flow... The neighboring nodes have ,in yes Downstream, Impact rating , In summary, the contribution of a flow to a port is: Known set communication flow Its neighbor set ,as well as For any The flow is calculated using the following formula. right Contributions: in As an indicator function, determine and Is it in Conflict; Evaluate the contribution of other flows to ensemble communication: by analyzing the first... Step-by-step critical flow scoring The weighted summation yields the flow The contribution score to the overall set of communication, assuming the first The execution time of the step is The ideal execution time is Then flow The contribution to the overall ensemble communication is: in Based on theoretical calculations or experience, the main contributors to flow conflicts can be identified by scoring each flow.

[0023] The technical effects to be achieved by the embodiments of the present invention are as follows: (1) An algorithm decomposition and graph description of set communication were designed. In order to capture the dynamic changes of set communication and the data dependencies of the flow, and to obtain a global view, the algorithm of set communication was decomposed at the step level, and a directed graph was constructed using the waiting relationship of the flow to describe the entire set communication process.

[0024] (2) A step-aware dynamic anomaly detection triggering mechanism was designed. In order to be accurate and low-overhead, a step-aware adaptive anomaly detection mechanism was adopted to limit unnecessary detection triggers, automatically select appropriate detection parameters, and combine end-side monitoring and network telemetry to accurately and efficiently support aggregated communication-level diagnosis.

[0025] (3) A multi-level comprehensive analysis and diagnosis mechanism was designed. In order to effectively conduct the root cause analysis of the overall communication, a multi-level graph was constructed to perform bottleneck analysis and root cause location, and to evaluate the main contributors to the flow conflict. Attached Figure Description

[0026] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0027] Figure 1 This diagram illustrates a breakdown of existing collective communication algorithms, with the Ring algorithm on the left and the Halving and Doubling algorithm on the right. Figure 2 This is a schematic diagram illustrating the Vedrfolnir system architecture according to an embodiment of the present disclosure; Figure 3 This is a schematic flowchart illustrating a Vedrfolnir system according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating a Ring Reduce-Scatte according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram illustrating the waiting relationship according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram illustrating an implementation configuration according to an embodiment of the present disclosure. Detailed Implementation

[0028] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0029] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0030] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0031] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0032] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0033] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0034] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0035] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0036] Figure 2 This is a schematic diagram illustrating a method for diagnosing performance anomalies in an RDMA network under aggregated communication according to an embodiment of the present disclosure.

[0037] System Architecture: From the design goals, Vederfolnir (1) in order to capture the dynamic changes of the collective communication process and the data dependencies of the flow, decomposes the collective communication algorithm at the step level and uses the waiting relationship of the flow to construct a directed graph to describe the entire collective communication process; (2) in order to achieve the goal of accuracy and low overhead, adopts a step-aware adaptive anomaly detection mechanism to limit unnecessary detection triggers, and uses the collaborative data collection of end-side monitoring and network telemetry to accurately and efficiently support the collective communication level diagnosis; (3) in order to effectively perform root cause analysis, constructs a multi-level graph to perform bottleneck analysis and root cause location, and evaluates the main contributors of flow conflicts.

[0038] Figure 2 The overall architecture of Vedrafolnir is demonstrated. First, during aggregated communication, the monitor on the server side pre-executes algorithm decomposition and monitors relevant performance information of the aggregated communication flow in real time, reporting it to the analyzer. Second, monitors communicate waiting relationships with servers by sending Notification packets. When performance anomalies occur, Vedrafolnir adaptively executes appropriate anomaly detection triggering procedures. During detection, the server sends polling query packets, triggering the collection of telemetry information on all switches related to the anomaly, which is then reported to the analyzer sequentially. Finally, the analyzer constructs a waiting relationship graph and a network source graph, and provides a comprehensive diagnostic result.

[0039] Figure 3The operation flow of Vederfolnir is shown. It includes software modules: monitor, analysis server, and switch collection program; hardware modules: programmable switch and server. Their functions are as follows: The monitor (1) performs algorithm decomposition before the aggregated communication is executed (2) sets the triggering mode for anomaly detection for each algorithm step according to network topology, FCT estimation, etc. (3) monitors the relevant performance information of the aggregated communication flow in real time according to the steps and reports it to the analysis server (4) the monitors can adaptively allocate anomaly detection triggering opportunities by sending Notification packets, so that the abnormal flow can be detected more. The analysis server is responsible for comprehensively analyzing the collected data, constructing a waiting relationship diagram and a network source diagram, deriving the performance bottleneck of aggregated communication and the specific network root cause, and scoring and evaluating the performance impact of other flows on aggregated communication in flow conflicts. The switch collection program (same as Hawkeye) is responsible for maintaining the switch PFC status, recording port traffic, recording port information and corresponding flow information, and reporting the recorded information to the analysis server after receiving polling packets.

[0040] Algorithm breakdown and graphical description of set communication: The algorithm decomposition and graph description mechanism of aggregate communication—the algorithm decomposes aggregate communication into steps, and uses the waiting relationship of the flow to construct a waiting relationship graph to describe the entire aggregate communication process, which solves the problems of RDMA network diagnosis not having a global view of aggregate communication and difficulty in capturing the dynamic changes of inter-flow dependencies.

[0041] Algorithm Breakdown: Vederfolnir breaks down the algorithm into multiple steps, using these steps to describe the different stages of various streams in set communication. For a stream originating from a specific node, the data blocks transmitted or the destination address of the stream change between each step. For example, in... Figure 1 In the Ring algorithm, when performing set communication, the data block transmitted by the stream (called F0) originating from Node 0 between step 1 and step 2 changes from block A to block C. When using the Halving and Doubing algorithm, F0's target is Node 4 in step 1, and Node 2 in step 2. When one of these two transformations occurs, we say that F0 has completed one step.

[0042] Waiting Graph Definition: After splitting the flow, we construct a waiting graph to describe the entire set communication process. Figure 4 Taking the Ring Reduce-Scatter example shown, Figure 5The corresponding waiting relationship graph is shown. We define (1) Nodes: the start and end of each step of each flow. In particular, the end of the last step of all flows is taken as the source node, and the start of the first step of all flows is taken as the sink node. (2) Directed edges: represent the waiting relationship between nodes. For example, due to data dependency, the start of F2S2 needs to wait for the end of F1S1, which is represented by a blue edge; at the same time, the start of F2S2 also needs to wait for the end of the previous step F2S1, which is represented by an orange edge; the start and end of F2S2 are represented by a dark edge. (3) Edge weight: represents the waiting time. The weight of a light edge is 0, and the weight of a dark edge is the execution time of this step of the flow.

[0043] Anomaly monitoring, detection, and telemetry mechanisms: Step-aware dynamic anomaly detection triggering mechanism – This mechanism limits unnecessary detection triggers, automatically selects appropriate detection parameters, and combines end-side monitoring and network telemetry to collect data in a coordinated manner to accurately and efficiently support aggregated communication-level diagnostics, thereby solving problems such as redundant diagnostic overhead and insufficient accuracy.

[0044] (1) End-to-end network combined abnormal data collection system: The server monitors the communication performance information of the collection. After detecting the performance abnormality, it adaptively triggers network telemetry to obtain the network information recorded in the switch so as to diagnose the abnormality.

[0045] Specifically, in terms of server-side monitoring, to construct a wait relationship graph and analyze performance bottlenecks in aggregated communication, each server's monitor collects performance information for the current flow. After each flow step is completed, the server reports its 5-tuple, the amount of data transmitted, the start time, the end time, and the source address of the flow it was waiting for. When the server detects performance anomalies, it will send polling packets for telemetry collection through anomaly detection.

[0046] For telemetry collection, the same method as in previous work Hawkeye is integrated. The switch periodically records flow-level telemetry data (e.g., flow 5-tuples, flow packet count, queue depth) and port-level telemetry data (e.g., traffic volume between ports, number of packets paused by PFC on each port, dequeue depth), and is aware of port PFC status. Upon receiving polling packets, the switch data plane propagates the polling packets according to the flow path and PFC propagation path, while the switch controller assists in data collection and reports the telemetry information to the analysis server.

[0047] (2) Step-aware design in anomaly detection: To address the issue of redundant overhead and improve accuracy, we made the following improvements to Hawkeye's server-side triggering design: 1) Step-by-step limited triggering. We specify the number of anomaly detection triggers for each step of the aggregated communication through configurable parameters, limiting the theoretical overhead limit while ensuring accuracy. During aggregated communication execution, we evenly distribute triggering opportunities based on the estimated completion time of the flow (which can be calculated based on network topology or set empirically) to ensure the widest possible detection coverage. Specifically, anomaly detection is triggered when three conditions are met: RTT threshold timeout, remaining triggering opportunities, and a certain time interval between two triggers. 2) Dynamically configure anomaly detection triggering conditions. Since the timeout RTT threshold is a configurable parameter in Hawkeye, it is unreasonable to apply it indiscriminately to all flows. We can configure the RTT threshold amplification factor and update the RTT threshold setting of the flow before each step of each flow in aggregated communication.

[0048] (3) Adaptive design in anomaly detection: The main idea is to dynamically allocate the number of triggers, making the triggers more concentrated on critical flows. Since the aggregated communication flows affected by anomalies are slowed down and run for longer, they should be given more anomaly detection trigger opportunities. In the operation of the system of this invention, assuming that a certain step flow F1 is completed, but the next step flow F2 that F1 depends on is not completed, if F1 still has anomaly detection opportunities, the server of F1 will send a customized packet to pass the opportunity to the server of F2, thereby increasing the detection opportunities of F2.

[0049] Multi-level comprehensive analysis and diagnostic mechanism: A multi-level comprehensive analysis and diagnosis mechanism is proposed—constructing a multi-level graph for bottleneck analysis and root cause localization, and evaluating the main contributors to flow conflicts based on calculations, thus solving the problem that existing methods cannot provide an assessment of the overall performance bottlenecks of aggregated communication and the contribution of flows to the overall performance anomalies of aggregated communication.

[0050] Waiting Graph Construction and Bottleneck Analysis: During actual operation, the analysis server arranges the collected data entries according to the flow completion time and constructs a waiting graph accordingly. Based on the real-time collected waiting relationship data and timestamps, the analysis server constructs the corresponding nodes and edges. For simplicity, after determining that a node is not being waited for (i.e., its in-degree is 0), the analysis server can recursively prune nodes with an in-degree of 0. After constructing the complete waiting graph, the analysis server calculates the performance bottleneck of the critical path representation set communication and diagnoses the network root cause based on the network source graph of each critical flow.

[0051] Network source graph construction: Vedrfolnir primarily constructs source graphs based on packet-level wait relationships between flows and ports. The graph's point set... It includes F (stream), P (port), and CF (collective communication stream), among which and The set of directed edges of a graph The waiting relationships between nodes can be mainly divided into the following three categories: (1) This describes the waiting relationship between a flow and a port. The waiting of a flow on a port is essentially congestion caused by multiple flows colliding at the port. We use the queue depth where a flow waits while queuing at a port as the weight of the edge. Assume the flow... A package Enter the queue and define the flow ahead of it. The number of packages is Then flow right The waiting weight is defined as The weight of the flow to the port is (2) This describes the waiting relationship between a port and a flow. We primarily use these edges to quantify the contribution of a flow to port congestion. Similarly, we use the number of packets contributed by the flow to the port queue depth as the weight of the edge. Assume the port... The queue depth is The number of packets detected within a certain period is Then the weight of the flow at the port is... (3) This describes the wait relationship between ports. We use this type of edge to capture PFC causality. When PFC occurs, the downstream switch's outgoing port... It may stop the upstream switch's outgoing port. Therefore, it constitutes arrive Waiting. Since the degree of waiting is measured based on the depth of the waiting queue, in this case, we consider it as... wait All packets in the queue. However, PFC propagation may occur along multiple paths. To accurately evaluate... arrive We define the contribution percentage along the path. , The weight between them is the traffic ratio. In summary, through , , These three types of edge sets belong to the graph The directed edges and their weights are defined respectively. , , The analyzer constructs a source graph to assess the interaction between flows and ports.

[0052] Root Cause Localization: Similar to previous work Hawkeye, Vederfolnir diagnoses specific root causes based on the characteristics of different anomaly types, including flow conflicts, PFC backpressure, etc. Similarly, by adding more feature definitions, Vederfolnir can expand the types of anomalies it can diagnose.

[0053] Diagnosis of the characteristics of convection conflict anomalies: Convection collisions are identified by examining port nodes in the network source graph. When the source graph shows... When considering the characteristic structure of the flow, it is believed that the flow... With collection communication flow At the port A conflict occurred there.

[0054] Diagnosis of PFC backpressure type characteristics: PFC back pressure is identified by tracing the PFC propagation path. When the source map appears... This indicates that PFC propagation has occurred and affected aggregate communication. Following the PFC propagation path shown in the diagram, the root cause of PFC can be located.

[0055] The contribution scores for flow conflict are as follows: To evaluate the contribution of other flows to a given set of communication flows: First, starting from a given set of communication flows (cf), obtain the maximally connected subgraph of the network source graph. Then all... All flows are considered as scoring objects. For each flow, we calculate the flow's contribution to that node level by level, following the direction of the reverse directed edge (i.e., the one being waited for). Assume the flow... The neighboring nodes have ,in yes Downstream. Then Impact rating , In summary, the contribution of a flow to a port is: Now, the set communication flow is known. Its neighbor set ,as well as For any .because and Possibly on the port A direct flow conflict occurs at this point; in this case... right The impact is Instead Therefore, we calculate the flow using the following formula. right Contributions: in As an indicator function, determine and Is it in Conflict.

[0056] Evaluating the contribution of other flows to set communication: Since the performance of set communication is mainly determined by the flows on the critical path in the waiting graph, we evaluate the contribution of other flows to set communication by considering the flow on the critical path in the waiting graph. Step-by-step critical flow scoring The weighted summation yields the flow A score for the contribution to overall set communication. Assume the... The execution time of the step is The ideal execution time is Then flow The contribution to the overall ensemble communication is: in This can be derived from theoretical calculations or experience. By scoring each flow, Vederfolnir identifies the main contributors to flow conflicts.

[0057] Specific implementation examples: Vedrfolnir integrates server-side and network-side components, requiring hardware support such as server-side CPU, GPU, RNIC, and network-side programmable switches. Figure 1 The system architecture of Vedrfolnir has already been demonstrated. Figure 6 This section demonstrates an example of the hardware and software implementation of Vederfolnir. Server GPUs communicate via aggregated communication and use RNIC for RDMA connections. On the server side, C and Python languages ​​are used to write Vederfolnir algorithm decomposition programs, performance monitors, and anomaly detection programs. On the network side, a programmable switch is used, and C and P4 languages ​​are used to write programs related to telemetry data recording and collection control. During deployment, the P4-related programs are first compiled and loaded into the switch chip using the toolchain provided by the programmable switch system, and the data plane state is configured. Then, on the server side, the algorithm decomposition and performance monitoring programs are integrated with the aggregated communication library and automatically enabled during aggregated communication. Finally, the monitoring results are collected and diagnostic algorithms are run on the analysis server. In this way, operators can monitor the aggregated communication performance status in real time through the analysis server and obtain diagnostic results when network performance anomalies occur.

[0058] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for diagnosing performance anomalies in RDMA networks under aggregated communication, characterized in that, During the aggregated communication process, firstly, the monitor located on the server side pre-executes the algorithm decomposition and monitors the relevant performance information of the aggregated communication flow in real time step by step, and reports it to the analysis server; The monitors communicate their waiting relationships with the servers by sending Notification packets. When a performance anomaly occurs, they adaptively execute an appropriate anomaly detection triggering process and send polling query data packets to the switches via the servers to trigger the collection of telemetry information on all switches related to the anomaly, which is then reported to the analysis server in sequence. Finally, the server constructs a waiting relationship graph and a network tracing graph, and a comprehensive analysis is conducted to provide diagnostic results.

2. The method for diagnosing performance anomalies in an RDMA network under aggregated communication as described in claim 1, characterized in that, The algorithm decomposition process involves breaking the algorithm down into multiple steps and using these steps to describe different stages of different flows in the set communication. For a flow originating from a specific node, the data block transmitted by the flow or the target address of the flow will change between each step.

3. The method for diagnosing performance anomalies in RDMA networks under aggregated communication as described in claim 2, characterized in that, The process of triggering the anomaly detection is as follows: A step-aware, dynamic anomaly detection mechanism is employed to limit unnecessary detection triggers, automatically select appropriate detection parameters, and combine end-side monitoring and network telemetry with collaborative data collection to accurately and efficiently support aggregated communication-level diagnostics. Specifically, First, through the end-to-end network combined anomaly data collection system, the server side monitors and collects communication performance information. After detecting performance anomalies, it adaptively triggers network telemetry to obtain network information recorded in the switch in order to diagnose the anomalies. During server-side monitoring, each server's monitor collects performance information of the current flow. After each flow step is completed, the server reports the five-tuple, the amount of data transmitted, the start time, the end time, and the source address of the waiting flow. When the server detects performance anomalies, it will send polling packets for telemetry collection through anomaly detection. During telemetry collection, the switch periodically records flow-level telemetry data and port-level telemetry data, and senses the port PFC status. Upon receiving a polling packet, the switch data plane will propagate the polling packet according to the flow path and PFC propagation path. At the same time, the switch controller assists in data collection and reports the telemetry information to the analysis server. During the anomaly detection process, the step-aware design is as follows: step-by-step limited triggering, the number of anomaly detection triggers for each step of the aggregate communication is specified by configurable parameters, and the triggering opportunities are evenly distributed according to the estimated completion time of the flow during the execution of the aggregate communication. The dynamic configuration of anomaly detection trigger conditions is as follows: Configure the RTT threshold amplification factor and update the RTT threshold setting of the stream before each step of each stream in the aggregated communication; The adaptive design method in anomaly detection is as follows: if a certain step of flow F1 is completed, but the next step of F1 depends on flow F2 is not completed, if F1 still has anomaly detection opportunities, then the server of F1 will send a customized packet to pass the opportunity to the server of F2, thereby increasing the detection opportunities of F2.

4. The method for diagnosing performance anomalies in RDMA networks under aggregated communication as described in claim 3, characterized in that, The specific method for the analysis server to construct the waiting relationship graph is as follows: During actual operation, the analysis server arranges the collected data entries according to the flow completion time order and constructs the waiting graph in sequence. Specifically, based on the real-time collected waiting relationship data and timestamps, the analysis server constructs corresponding nodes and edges. For simplicity, after determining that a certain node has an in-degree of 0, the analysis server can recursively prune the nodes with an in-degree of 0. After constructing the complete waiting graph, the analysis server calculates the performance bottleneck of the critical path representation set communication and diagnoses the network root cause corresponding to the network source graph of each critical flow.

5. The method for diagnosing performance anomalies in an RDMA network under aggregated communication as described in claim 4, characterized in that, The specific method for constructing the network source graph is as follows: A source graph is constructed based on the packet-level wait relationships between flows and ports, and the graph's point set... It includes stream F, port P, and collection communication stream CF, where and The set of directed edges of a graph Describing the waiting relationships between nodes, they are divided into the following three categories: (1) Edge sets : Describes the waiting relationship between a flow and a port, using the queue depth of the flow waiting at the port as the edge weight. Assume the flow... A package Enter the queue and define the flow ahead of it. The number of packages is Then flow right The waiting weight is defined as The weight of the flow to the port is (2) Edge set This describes the waiting relationship between a port and a flow. These edges quantify the contribution of a flow to port congestion, using the number of packets contributed by the flow within the port's queue depth as the edge weight. Assuming the port... The queue depth is The number of packets detected within a certain period is Then the weight of the flow at the port is (3) Edge set This describes the wait relationship between ports. This type of edge is used to capture the causal relationship of Priority Flow Control (PFC). When Priority Flow Control (PFC) occurs, the downstream switch's outgoing port... Stop the upstream switch output port Therefore, it constitutes arrive Waiting is regarded as wait All packets in the queue, for accurate evaluation arrive Contribution percentage along the path, defined , The weight between them is the traffic ratio. .

6. The method for diagnosing performance anomalies in an RDMA network under aggregated communication as described in claim 5, characterized in that, The specific method of the comprehensive analysis is as follows: diagnose the specific root cause based on the characteristics of different anomaly types, including flow conflict and PFC back pressure; The characteristics of the flow conflict anomaly type are: Convection collisions are identified by examining port nodes in the network source graph. When considering the characteristic structure of the flow, it is believed that the flow... With collection communication flow At the port A conflict occurred at the location; The characteristics of the PFC back pressure type are as follows: Identifying PFC back pressure by tracing the PFC propagation path, when the source map appears This indicates that PFC propagation has occurred and affected aggregate communication. Following the PFC propagation path shown in the diagram, the root cause of PFC can be located. The contribution score for the flow conflict is: To evaluate the contribution of other flows to a given set of communication flows: First, starting from a given set of communication flows, obtain the maximally connected subgraph of the network source graph. For the set of communication flows CF, all flows in the graph that meet the criteria are considered to be connected. All flows are included in the scoring object. For each flow, the contribution of the flow to that node is calculated step by step in the direction of the reverse directed edge being waited for. Assume the flow... The neighboring nodes have ,in yes Downstream, Impact rating , The contribution of the flow to the port is: Known set communication flow Its neighbor set ,as well as For any node The flow is calculated using the following formula. right Contributions: in As an indicator function, determine and Is it in Conflict; Evaluate the contribution of other flows to ensemble communication: by assessing the contribution of the first flow to ensemble communication. Step-by-step critical flow scoring The weighted summation yields the flow The contribution score to the overall set of communication, assuming the first The execution time of the step is The ideal execution time is Then flow The contribution to the overall ensemble communication is: in Based on theoretical calculations or experience, the main contributors to flow conflicts can be identified by scoring each flow.

Images