RDMA network performance abnormity diagnosis method based on PFC traceability

Abstract

The invention provides an RDMA network performance abnormity diagnosis method based on PFC traceability. The method comprises the following three steps: step 1, recording passive and PFC-perceived telemetering data, and logging causal relationship data for tracking a PFC propagation path; step 2, carrying out abnormal driving, online causal analysis and data collection, notifying a control plane CPU of the switch by the switch, and collecting and reporting telemetering data in an asynchronous mode; step 3, off-line diagnostic analysis based on traceability: after an analyzer located in the center collects all related telemetry data, an off-line diagnostic algorithm based on traceability is operated, the off-line diagnostic algorithm based on traceability firstly constructs a heterogeneous traceability graph, the traceability graph comprehensively describes details of abnormities, and the details of the abnormities are analyzed; comprising the damage degree of the flow, the causal propagation path of the PFC and the contribution of each flow to congestion; and then, a diagnosis program analyzes on the traceability graph, tracks a causal chain of congestion, and finally locates a root cause of performance abnormality.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically, to a method for diagnosing RDMA network performance anomalies based on PFC tracing. Background Technology

[0002] Remote Direct Memory Access (RDMA) technology aims to achieve high-throughput and low-latency data transmission between network applications. Its core idea is to embed the complex network protocol stack into a dedicated RDMA Network Interface Card (RNIC). Through one-sided operations (such as WRITE and READ) provided by this NIC, application servers can directly and remotely read and write to the memory of another server without the intervention of the remote host's CPU and operating system, avoiding additional data copying. This significantly improves transmission performance while substantially reducing CPU load. Given its excellent performance advantages, RDMA technology has been widely used in high-performance production environments, such as distributed deep learning training and distributed cloud storage. Early RDMA technology was mainly based on the InfiniBand protocol, which was incompatible with mainstream Ethernet, limiting its large-scale deployment. The subsequent emergence of RoCEv2 (RDMA over Converged Ethernet version 2) technology enabled RDMA to be deployed directly on traditional Ethernet, greatly promoting its adoption in modern data centers.

[0003] To ensure reliable and high-performance data transmission for RDMA communication over Ethernet, the RoCEv2 protocol deploys two complementary key mechanisms: hop-by-hop flow control and end-to-end congestion control.

[0004] Hop-by-hop flow control is primarily implemented through the Priority Flow Control (PFC) mechanism, with its core goal of building a functionally "lossless" network environment. The pursuit of losslessness is crucial because RDMA is extremely sensitive to packet loss. Unlike TCP, designed for lossy networks, RDMA's network interface card (NIC) recovery mechanisms (such as Go-Back-N retransmission) are inefficient when packets are lost, leading to severe performance degradation and even a "retransmission storm," which is unacceptable for latency-sensitive applications. Therefore, it is essential to prevent packet loss at its source, rather than relying on retransmissions afterward; PFC is key to achieving this goal. Its working principle is as follows: when the outgoing queue of a switch port in the network exceeds a preset threshold due to congestion, the switch sends a "PAUSE" frame for a specific priority to the upstream device (another switch or NIC). Upon receiving the PAUSE frame, the upstream device immediately suspends sending data of the corresponding priority to this port. When the queue length of the switch port drops below a safe threshold, it sends a signal (logically "RESUME") to notify the upstream device to resume data transmission. The hop-by-hop "pause-resume" mechanism is crucial because of its extremely fast response speed. Faced with microbursts of traffic common in networks, switch buffers can fill up within microseconds. As a link-layer mechanism, PFC's response latency is only the propagation latency of a single-hop link (nanoseconds to microseconds), allowing it to take effect before the buffer overflows. In contrast, end-to-end control mechanisms rely on a full network round-trip time (RTT), making them too slow to handle such transient congestion.

[0005] However, PFC itself is not a perfect solution. Excessive or prolonged triggering can lead to negative consequences such as head-of-line blocking, widespread congestion propagation, and even deadlock. Therefore, end-to-end congestion control mechanisms have been introduced as a supplement and optimization to PFC. A typical example of this type of mechanism is DCQCN (Data Center Quantized Congestion Notification). Its core idea is that switches monitor their own queue status to determine network congestion trends and transmit congestion signals to the final receiver via ECN (Explicit Congestion Notification), which then notifies the original sender. Upon receiving the congestion signal, the sending host proactively adjusts its transmission rate, thereby controlling traffic at its source and preventing the network from entering a state of continuous congestion.

[0006] In modern high-performance RDMA networks, these two mechanisms are deployed together and work in concert at different time scales to ultimately ensure that RDMA achieves efficient transmission over Ethernet.

[0007] While the PFC mechanism successfully constructs lossless networks, it also introduces a series of new and complex network performance anomalies (NPAs). When congestion occurs on a port, the PFC pause mechanism propagates upstream like ripples, and this cascading congestion propagation often causes performance problems to affect flows that did not even directly pass through the initial congestion point. Typical performance problems resulting from this include head-of-line blocking, network unfairness, PFC storms, and even PFC deadlock. These problems make performance anomalies in RDMA networks more frequent and cause more severe performance degradation compared to traditional TCP / IP networks. Although RDMA networks deploy both of these flow control mechanisms, complex network performance anomalies still occur frequently and are extremely difficult to diagnose.

[0008] First, several challenging performance anomalies still exist in RDMA networks. For example, PFC backpressure caused by micro-bursts or incast traffic can cause congestion to propagate through PFC cascading, affecting unrelated "innocent" flows; PFC storms caused by continuous injection of pause frames by terminal hosts can cause widespread network paralysis without any traffic contention; and PFC deadlocks caused when PFC paused links form loops can completely halt all traffic within the loop.

[0009] Secondly, accurately and efficiently diagnosing these NPAs faces two major inherent challenges. First, the complexity of traffic characteristics: RDMA traffic typically features high bandwidth and line-speed startup, easily causing frequent and transient congestion on switches. Second, the complexity of the PFC mechanism: PFC introduces a completely new congestion pattern, where congestion can propagate upstream from downstream nodes independently of local queue conditions. This characteristic physically separates the congestion phenomenon from its root cause, overturning the fundamental assumption of traditional diagnostic tools that rely on "local queue analysis."

[0010] Finally, faced with these challenges, existing network performance anomaly diagnostic tools often struggle to accurately and efficiently pinpoint the root cause of problems. The shortcomings of these tools are mainly reflected in the following three aspects:

[0011] (1) Lack of fine-grained visibility into the impact of PFC: Both diagnostic tools deployed on terminal hosts and traditional monitoring systems deployed on the switch side lack the ability to monitor the impact of flow-level PFC. They cannot effectively distinguish whether the performance degradation of a flow is due to direct queue contention on the path or to a PFC pause propagated from a remote location, thus leading to misjudgment of the cause of congestion.

[0012] (2) Lack of rapid tracing capability for PFC causality: Due to the cascading nature of PFC, locating the root cause requires collecting relevant telemetry data along the PFC propagation path. Existing methods either fail to trace PFC paths across multiple hops because they only focus on the internal workings of a single switch, or incur huge overhead due to the need to collect data from all switches in the network, or ignore situations where the PFC propagation path and the flow path do not completely overlap because they only trace the path of the affected flow, thus missing the true source of congestion.

[0013] (3) Diagnostic algorithms cannot accurately analyze anomalies caused by PFC: Most existing diagnostic algorithms are based on "flow interaction" or "queue sharing" models, which tend to identify flows that share a queue with the victim flow in the same queue as the cause of congestion. However, in RDMA networks, the real root cause may be a burst of traffic a few hops away that does not share any queue with the victim flow, which indirectly causes the performance degradation of the victim flow through the PFC backpressure mechanism. This limitation of the diagnostic paradigm leads to a serious lack of analysis on "why the anomaly occurred" (the real root cause flow) and "how it occurred" (the propagation path of PFC), which easily leads to misjudgment.

[0014] Therefore, designing a novel diagnostic method for RDMA network performance anomalies that can deeply understand and accurately track the causal relationship of PFC has become an urgent technical challenge. Summary of the Invention

[0015] The purpose of this disclosure is to provide a method for diagnosing RDMA network performance anomalies based on PFC tracing, aiming to address the shortcomings of existing RDMA network performance diagnostic tools.

[0016] In general, this paper provides a method for diagnosing RDMA network performance anomalies based on PFC tracing, which includes three steps:

[0017] Step 1: Passive, PFC-aware telemetry data recording: Passively and continuously record telemetry data containing PFC visibility and causal awareness on the switch's data plane, and log causal data for tracking PFC propagation paths.

[0018] Step 2, Anomaly-Driven In-Network Causal Analysis and Data Collection: When the detection agent of the terminal host detects a performance anomaly, it injects a "polling packet" into the network to start the diagnostic process. When the switch receives the polling packet, it analyzes the PFC causal relationship on its data plane and intelligently forwards the polling packet only to neighboring switches related to this anomaly. At the same time, the switch notifies its control plane CPU to perform telemetry data collection and reporting in an asynchronous manner.

[0019] Step 3: Offline Diagnostic Analysis Based on Source Tracing: After collecting all relevant telemetry data, the central analyzer runs an offline diagnostic algorithm based on source tracing. This algorithm first constructs a heterogeneous source tracing graph, which comprehensively describes the details of the anomaly, including the severity of flow damage, the causal propagation path of PFC, and the contribution of each flow to congestion. Subsequently, the diagnostic program analyzes the source tracing graph to trace the causal chain of congestion and ultimately pinpoint the root cause of the performance anomaly.

[0020] The specific method for passively and continuously recording telemetry data containing PFC visibility and causal relationship awareness on the data plane of the switch is as follows: First, a "PFC status register" is set up in the processing pipeline of the data plane for each outgoing port; when the switch receives a PFC PAUSE frame from a downstream device from one of its ingoing ports, the PAUSE frame is specially processed and passed to the outgoing port pipeline. The pause time information carried by the frame is used to update the PFC status register of the corresponding outgoing port. Accordingly, any normal data packet that attempts to dequeue during the "paused" state of the port is identified by the system as a "data packet paused by PFC", and its relevant statistics are also updated.

[0021] The method for constructing causal relationship data for tracking PFC propagation paths is as follows: construct a port-level traffic causal relationship recording structure, and use a two-dimensional traffic counting matrix to record the traffic volume from any ingress port to any egress port in real time.

[0022] The method for recording the telemetry data is as follows: a multi-granularity telemetry data recording and aggregation method based on rounds is designed. The switch divides the time into consecutive rounds with a fixed time period as the unit, and maintains a circular buffer composed of multiple rounds of data in memory.

[0023] The switch uses the high-precision timestamp metadata that comes with each data packet to calculate the round ID to which the data packet belongs at online speed by extracting specific bits of the timestamp, thereby updating the telemetry data to the corresponding position in the circular buffer;

[0024] This structure allows for the simultaneous recording of telemetry data at both the flow level and port level.

[0025] At the flow level, traditional telemetry information such as the five-tuple, total number of packets, and total queuing depth are recorded for each flow. In addition, to achieve PFC awareness, the number of packets paused due to PFC in the flow is also recorded. The entries of the flow are indexed by the hash value of its five-tuple.

[0026] At the port level, the total length of its egress queue and the total number of packets suspended due to PFC are recorded directly for each port, with the port entry directly indexed by its port number.

[0027] The specific method for analyzing the causal relationship of PFC is as follows:

[0028] First, a diagnostic triggering method driven by terminal anomalies is adopted. A lightweight detection agent is deployed on the terminal host. The agent continuously monitors the end-to-end performance indicators of its outgoing flows. When the performance indicator of a flow deteriorates and exceeds a preset threshold, the agent generates a polling data packet and injects it into the network.

[0029] Secondly, a polling packet format and forwarding logic for guiding causal tracing are designed. The header of the polling packet contains the five-tuple information of the victim flow and a key polling flag. Different values ​​of the polling flag indicate that the switch performs different tracing tasks. To ensure that the polling packet itself is not blocked by congestion, it is placed in a high-priority queue that is not affected by PFC for transmission.

[0030] Furthermore, the design incorporates a line-speed causal analysis and forwarding process executed entirely within the switch's data plane. When a switch equipped with a polling packet format and forwarding logic to guide causal tracing receives a polling packet, it performs the following recursive analysis and forwarding operations:

[0031] Step 1, victim flow path tracing: If the flag bit of the received polling data packet is "01", the switch queries the forwarding table based on the victim flow 5-tuple information embedded in the packet and unicasts the polling data packet to the next hop switch of the victim flow.

[0032] Step 2, PFC event detection: While forwarding, the switch checks its internal telemetry data to determine whether the outgoing port of the victim flow is being suspended by PFC.

[0033] Step 3, Tracing Path Extension: If a PFC pause is detected, the switch sets the higher-order flag to "1" in the forwarded polling packet. The modified flag is used to notify the downstream switch to start tracing the PFC causal path.

[0034] Step 4, PFC Causal Path Analysis and Tracing: When a switch receives a polling data packet with the flag "1*", it first multicasts the packet internally to all outgoing ports. Each outgoing port pipeline queries the aforementioned port-level traffic causal relationship record structure to determine whether it is the causal port that caused this PFC backpressure.

[0035] Step 5, Recursive Tracing: Only those ports that are determined to be causally related will continue to forward polling packets upstream. This process is carried out recursively, hop by hop, until the original source of the PFC propagation chain is traced.

[0036] Finally, an asynchronous data collection method assisted by the switch control plane was designed. During the tracing process, each switch that receives the polling data packet will mirror a copy of the polling data packet to its own CPU port while executing the forwarding logic of the data plane. This operation will wake up the collection agent process running on the switch control plane. This process will asynchronously and securely read the complete telemetry data recorded by the data plane, filter out invalid entries, and package the valid data into one or more large data packets, which will eventually be reported to the central analysis server.

[0037] The specific implementation of the source-tracing-based offline diagnostic algorithm is as follows: First, a heterogeneous source-tracing graph is constructed, and the causal chain of congestion is described by a "wait-wait" relationship graph. Then, the abnormality type is identified and the cause of the abnormality is found by analyzing the structural features of the heterogeneous source-tracing graph.

[0038] The method for constructing the heterogeneous source tracing graph is as follows: the heterogeneous source tracing graph is defined as a directed weighted graph, with flows and switch outgoing ports as two different types of nodes. The edges in the graph represent the "waiting" relationships between nodes, and the weight of the edge quantifies the strength or contribution of the waiting relationship. Specifically, three key waiting relationship edges are defined and constructed:

[0039] Port-to-port waiting edge: Used to describe the propagation causality of PFC, if an outgoing port P... i If a device is blocked due to receiving a downstream PFC pause frame, it will "wait" for one or more downstream congested ports P. j Clear its queue, thus constructing a queue from P i Point to P j A directed edge, the weight of which is the product of three factors: P i The number of packets suspended by PFC on the port, from P i Forward to P j Traffic accounts for P i The proportion of total export flow, and P j The port's own queue congestion level;

[0040] Flow-Port Waiting Edge: Used to describe the impact of PFC pause on a single flow, if a flow f i Its path passes through a port P that is suspended by PFC. j Then the flow f i It is in the "waiting" port P j To resume data forwarding, the system constructs a slave node f. i Pointing to port node P j The weight of an edge is defined as f. i During this period in P j The number of data packets that were paused;

[0041] Port-flow waiting edge: Used to describe a direct queue contention relationship occurring within a port. For a port P that is not paused by PFC but is queued due to excessive local traffic, this is relevant. j The system is built from port node P j Points to all flow nodes f that cause queuing within them. i The edges, where the weight of each edge represents the flow f. i The net contribution to the overall congestion of this port is calculated as follows: first, calculate the waiting time f of all other flows in the network. i The total amount, minus f i The total amount of other flows is waiting; ultimately, flows with positive weights are contributors to congestion, while flows with negative weights are victims of congestion.

[0042] The specific method for identifying anomaly types and finding the causes of anomalies by analyzing the structural features of the heterogeneous source map is as follows:

[0043] By traversing a heterogeneous source graph and matching it with various predefined exception signatures, the system automatically identifies the type and root cause of exceptions. Specific types and root causes include:

[0044] PFC backpressure caused by traffic contention: There is a PFC propagation path consisting of one or more port-port waiting edges in the heterogeneous source graph, and the endpoint of the path is a port node with strong port-flow waiting edges pointing to certain specific flows. These flows are the root cause of this event.

[0045] PFC storm: A PFC propagation path exists in the heterogeneous source graph, and the terminal port node of the path does not have any port-flow waiting edges with positive weight;

[0046] PFC deadlock: A loop exists between port-port waiting edges in the heterogeneous source graph;

[0047] Traditional flow contention congestion: If there are no port-to-port waiting edges in the heterogeneous source graph, then analyze the existing port-to-flow waiting edges in the graph to find the root cause flow causing congestion.

[0048] The technical effects to be achieved by the embodiments of the present invention are as follows:

[0049] Hawkeye is a precise and efficient RDMA network performance anomaly diagnostic system based on PFC source tracing. Leveraging the ability of switches to record and collect telemetry data at line speed in the data plane, Hawkeye fully considers two core congestion scenarios in RDMA networks: flow-level queue contention and port-level PFC propagation. Through a novel diagnostic mechanism, it can accurately analyze the complete causal chain of network performance anomalies and pinpoint the root cause of the problem. Hawkeye's accuracy and efficiency stem primarily from its three core technologies: 1) a fine-grained, PFC-aware telemetry mechanism for accurately recording the actual performance impact of PFC on each flow; 2) a PFC causal relationship analysis and tracing mechanism executed within the network data plane for quickly and efficiently identifying all switches related to the anomaly and collecting the necessary telemetry data for diagnosis; and 3) a source tracing graph-based diagnostic algorithm for constructing a heterogeneous "wait-waited" relationship graph, thereby comprehensively analyzing the anomaly occurrence process and accurately identifying the anomaly type and root cause (such as malicious burst flows or problematic terminal hosts). Attached Figure Description

[0050] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0051] Figure 1 This is a schematic diagram illustrating the architecture of an RDMA network performance anomaly diagnosis method based on PFC tracing according to an embodiment of the present disclosure;

[0052] Figure 2 This is a schematic diagram illustrating the port-level traffic causality record structure architecture according to an embodiment of the present disclosure;

[0053] Figure 3 This is a schematic diagram illustrating a telemetry structure architecture according to an embodiment of the present disclosure;

[0054] Figure 4 This is a schematic diagram illustrating the polling packet format architecture according to embodiments of the present disclosure;

[0055] Figure 5 This is a schematic diagram illustrating the architecture of PFC causal analysis according to an embodiment of the present disclosure;

[0056] Figure 6 This is a schematic diagram illustrating the tracing of a typical problem according to embodiments of the present disclosure. Detailed Implementation

[0057] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0058] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0059] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0060] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0061] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0062] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0063] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0064] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0065] Figure 1 This is a schematic diagram illustrating a method for diagnosing RDMA network performance anomalies based on PFC tracing according to an embodiment of the present disclosure.

[0066] To achieve the aforementioned objectives, the present invention employs the following technical framework: Figure 1 As shown.

[0067] The Hawkeye system can passively record telemetry data containing PFC causal relationships at online speed. When an anomaly occurs, it can quickly collect all relevant telemetry information through a network tracing mechanism, and finally use a source-tracing-based diagnostic algorithm to comprehensively analyze the anomaly occurrence process and accurately identify the anomaly type and root cause.

[0068] like Figure 1As shown, the overall workflow of the Hawkeye system includes the following three core steps: (1) Passive, PFC-aware telemetry data recording: The Hawkeye system passively and continuously records telemetry data containing PFC visibility and causal awareness on the switch's data plane. It can not only record the direct impact of PFC on data packets, but also log causal data used to trace the PFC propagation path. This fine-grained flow-level and port-level telemetry information provides a data foundation for subsequent comprehensive diagnosis. (2) Anomaly-driven, on-network causal analysis and data collection: When the detection agent of the terminal host discovers a performance anomaly, it will inject a "polling packet" into the network to start the diagnostic process. When the Hawkeye switch receives the polling packet, its data plane will perform line-speed analysis of PFC causality and intelligently forward the polling packet only to neighboring switches related to this anomaly. At the same time, the switch will notify its control plane CPU to perform telemetry data collection and reporting in an asynchronous manner, thereby efficiently completing data collection without affecting normal data forwarding. (3) Offline Diagnostic Analysis Based on Source Tracing: After collecting all relevant telemetry data, the central analyzer runs an offline diagnostic algorithm based on source tracing. This algorithm first constructs a heterogeneous source tracing graph, which comprehensively describes the details of the anomaly, including the severity of flow damage, the causal propagation path of PFC, and the contribution of each flow to congestion. Subsequently, the diagnostic program analyzes this graph to trace the causal chain of congestion, ultimately pinpointing the root cause of the performance anomaly. The key technologies constituting this invention will be described in detail below.

[0069] (1) Fine-grained PFC sensing and telemetry mechanism

[0070] First, this invention achieves direct visibility and quantified recording of PFC status and PFC causality. To enable switches to perceive port-level PFC status, this invention sets up a "PFC status register" in the data plane processing pipeline for each egress port. When a switch receives a PFC PAUSE frame from a downstream device from one of its ingress ports, the PAUSE frame is specially processed and passed to the egress port pipeline. The pause time information it carries is used to update the corresponding egress port's PFC status register. Accordingly, any normal data packet attempting to dequeue while the port is in a "paused" state will be identified by the system as a "PFC-paused data packet," and its related statistics will be updated.

[0071] Furthermore, to trace the propagation path of PFC congestion, this invention innovatively introduces a port-level traffic causality recording structure. This structure is a two-dimensional traffic counting matrix (Port TrafficMeter) used to record the traffic volume from any ingress port to any egress port in real time. By querying this matrix, the system can accurately determine which ingress ports' traffic collectively constitutes the queue of a suspended egress port, thereby quickly identifying the direct upstream contributors to PFC backpressure. For example, in Figure 2 In the case where both ports P3 and P4 are congested, only traffic is forwarded from the ingress port of P1 to the egress port of P3. Therefore, the system can determine that the direct cause of the P1 upstream device being suspended by PFC is the congestion of P3, while P4 is unrelated to it.

[0072] Secondly, such as Figure 3 As shown, this invention designs a multi-granularity telemetry data recording and aggregation method based on epochs. In order to achieve fine-grained monitoring of network status, the switch divides time into consecutive epochs with a fixed time period (e.g., 1 millisecond) and maintains a ring buffer in memory consisting of data from multiple epochs.

[0073] The switch uses the high-precision timestamp metadata that comes with each data packet to calculate the round ID to which the data packet belongs at online speed by extracting specific bits of the timestamp, thereby updating the telemetry data to the corresponding position in the circular buffer.

[0074] In this structure, the present invention simultaneously records telemetry data at both the flow-level and port-level granularities:

[0075] At the flow level, the system records traditional telemetry information for each flow, such as its 5-tuple, total number of packets, and total queuing depth. In addition, to achieve PFC awareness, the system also specifically records the number of packets paused due to PFC for that flow. Flow entries are indexed by the hash value of their 5-tuple.

[0076] At the port level, to avoid costly aggregation calculations from flow to port in the data plane, the system directly records the total length of its egress queue and the "total number of packets paused due to PFC" for each port. Port entries are directly indexed by their port numbers.

[0077] Through the above design, the telemetry mechanism of this invention can passively and granularly record key network status information, including the impact of PFC, on the switch with extremely high efficiency, providing a solid data foundation for subsequent anomaly detection triggering and accurate diagnosis.

[0078] (2) Causal analysis and tracing of PFC in the network

[0079] First, this invention employs a diagnostic triggering method driven by terminal anomalies. The system deploys a lightweight detection agent on the terminal host, which continuously monitors the end-to-end performance metrics (such as round-trip time, RTT) of its outgoing flows. When the performance metrics of a flow deteriorate and exceed a preset threshold (e.g., RTT exceeds three times the normal value), the agent generates a polling packet and injects it into the network. This terminal host-initiated triggering method has two major advantages compared to switch-side triggering: first, it avoids multiple switches on the PFC path simultaneously detecting anomalies and repeatedly initiating tracing tasks, ensuring that only one complete causal chain tracing is performed for each anomaly event; second, it can utilize richer application-layer performance metrics (such as job completion time) that are only perceptible on the host side as triggering conditions.

[0080] PollingFlag Meaning 00 Uselesstracing 01 (Default) Only trace along victimflow path 10 Only trace along PFC causality 11 Trace both of them

[0081] Table 1. Polling Flag Format

[0082] Secondly, such as Figure 4 As shown in Table 1, this invention designs a polling packet format and forwarding logic for guiding causal tracing. The header of this polling packet contains a five-tuple of victim flow information and a crucial "polling flag." Different values ​​of the flag instruct the switch to perform different tracing tasks. For example, the default value of "01" indicates tracing only along the path of the victim flow; while when the flag changes to "10" or "11," it indicates that the propagation path of PFC needs to be traced simultaneously. To ensure that the polling packet itself is not congested, it is placed in a high-priority queue unaffected by PFC for transmission.

[0083] like Figure 5 As shown, this invention implements a line-speed causal analysis and forwarding process that executes entirely within the switch's data plane. When a switch equipped with the logic of this invention receives a polling data packet, it performs the following recursive analysis and forwarding operations:

[0084] 1. Victim Flow Path Tracing: If the flag bit of the received polling packet is "01", the switch queries the forwarding table based on the victim flow 5-tuple information embedded in the packet and unicasts the polling packet to the next-hop switch of the victim flow.

[0085] 2. PFC Event Detection: While forwarding, the switch checks its internal telemetry data to determine whether the outgoing port of the victim flow is being suspended by PFC.

[0086] 3. Tracing Path Extension: If a PFC pause is detected, the switch sets the higher-order flag bit to "1" (i.e., changes the flag bit to "11") in the forwarded polling packet. This modified flag bit is equivalent to an instruction to notify the downstream switch (i.e., the sender of the PFC PAUSE frame) to start tracing the PFC causal path.

[0087] 4. PFC Causal Path Analysis and Tracing: When a switch receives a polling packet with a flag of "1*" (i.e., 10 or 11), it first multicasts the packet internally to all outgoing ports. Each outgoing port pipeline queries the aforementioned "port-level traffic causality record structure" (i.e., traffic counting matrix) to determine whether it is the "causally related" port causing this PFC backpressure.

[0088] 5. Recursive tracing: Only ports that are determined to be "causally related" will continue to forward polling packets (with the flag set to "10") upstream. This process proceeds recursively, hop by hop, until the initial source of the PFC propagation chain is traced (e.g., a port with only local flow contention and no PFC input, or a port directly connected to the terminal host).

[0089] Finally, this invention employs an asynchronous data collection method assisted by the switch control plane. During the aforementioned tracing process, each switch receiving a polling packet, while executing its data plane forwarding logic, mirrors a copy of that polling packet to its own CPU port. This operation wakes up the collection agent process running on the switch control plane. This process asynchronously and securely reads the complete telemetry data recorded in the data plane, filters out invalid entries (such as records with all zeros), and packages the valid data into one or more large data packets (such as the MTU size), ultimately reporting them to the central analysis server. This approach avoids performing complex data export operations on valuable data plane resources, significantly reducing the impact of data collection on normal data forwarding performance.

[0090] Through this entire mechanism, the present invention can accurately and completely collect remote acquisition data from all switches involved in a performance anomaly event with extremely low bandwidth and processing overhead, laying the foundation for subsequent diagnostic analysis.

[0091] Source-tracing-based diagnostic algorithms

[0092] Once the analysis server has collected all telemetry data related to the anomaly, this invention employs a diagnostic algorithm based on a provenance graph to analyze the data, thereby comprehensively analyzing the causal relationships of the anomaly and accurately locating its root cause. The core of this algorithm is to first construct a heterogeneous "wait-for" relationship graph to describe the causal chain of congestion, and then identify the anomaly type and pinpoint the culprit by analyzing the structural characteristics of this graph.

[0093] The first step is to construct a heterogeneous source graph. This source graph is a directed weighted graph that treats flows and switch output ports as two different types of nodes. The "edges" in the graph represent the "waiting" relationships between nodes, and the "weight" of the edge quantifies the strength or contribution of this waiting relationship. This invention defines and constructs three key waiting relationship edges:

[0094] Port-to-Port waiting edges: Used to describe the propagation causality of PFC. If an outgoing port P... i If it is blocked because it received a downstream PFC pause frame, then it is actually "waiting" for one or more congested downstream ports P. j Clear its queue. Therefore, the system will construct a queue from P. i Point to P j A directed edge. The weight of this edge is the product of three factors: P i The number of packets suspended by PFC on the port, from P i Forward to P j Traffic accounts for P i The proportion of total export flow, and P j The port's own queue congestion level (e.g., average queue depth). This weight precisely quantifies the downstream port P. j For upstream port P i The magnitude of the contribution of PFC back pressure.

[0095] Flow-to-Port waiting edges: These describe the impact of a PFC pause on a single flow. If a flow f i Its path passes through a port P that is suspended by PFC. j Then the flow f i It is in the "waiting" port P j Resume data forwarding. Therefore, the system will construct a slave node f. i Pointing to port node P j The weight of an edge is defined as f. i During this period in P j The number of data packets that are paused.

[0096] Port-to-Flow Waiting Edge: Used to describe direct queue contention relationships occurring within a port. For a port P that is not paused by PFC but is queued due to excessive local traffic, this is a waiting edge. j This port can be considered as each stream in a "waiting" queue being sent out one by one. The system will build a queue from port node P. j Points to all flow nodes f that cause queuing within them. i The edges. The weight of each edge represents the flow f. i The "net contribution" to the overall congestion of this port. This net value is calculated using a sophisticated algorithm: first, the waiting time f for all other flows in the network is calculated. i The total amount, minus f i The total amount of other flows is waiting. Flows with positive weights are contributors to congestion, while flows with negative weights are victims of congestion.

[0097] The second step is anomaly diagnosis based on graph feature matching. For example... Figure 6 As shown, the constructed source graph provides a complete and visual analysis of performance anomalies. The diagnostic algorithm of this invention automatically identifies the type and root cause of the anomaly by traversing this graph and matching it with various predefined anomaly signatures. Typical diagnostic scenarios include:

[0098] PFC backpressure caused by traffic contention: Its source graph is characterized by the existence of one or more port-to-port waiting edges forming a PFC propagation path, and the endpoint of this path (i.e., the initial congestion point) is a port node with strong port-to-flow waiting edges pointing to certain specific flows. These directed flows (usually with bursty characteristics) are the root cause of this event.

[0099] PFC storm: Its source graph features are similar to the previous scenario, with a PFC propagation path also present. However, the key difference is that the terminal port node of the path does not have any port-flow waiting edges with positive weights. This indicates that the congestion on this port is not caused by internal flow contention, but by the continuous injection of PFC frames from external factors (such as network card failure or misconfiguration of the terminal host).

[0100] PFC deadlock: Its most prominent graph feature is the existence of a cycle between port-to-port waiting edges. By further analyzing the connectivity between nodes inside and outside the cycle and the situation of port-to-flow waiting edges, the algorithm can also distinguish whether the deadlock is caused by flow contention inside the cycle or by PFC injection outside the cycle, thus locating different root causes.

[0101] Traditional flow contention congestion: If there are no port-to-port waiting edges in the source graph, it indicates that PFC propagation has not occurred in the network, and the congestion is isolated and represents traditional queue contention. In this case, the root cause flow causing the congestion can be found simply by analyzing the existing port-to-flow waiting edges in the graph.

[0102] Specific application scenarios:

[0103] The Hawkeye system described in this invention requires three components—a terminal server, a switch, and a source tracer—to work together.

[0104] 1. Terminal Server Component: A performance monitoring agent needs to be deployed on each terminal server requiring performance monitoring. This agent, a lightweight process, can be implemented on a smart network interface card (such as an NVIDIA BlueField-3 DPU) or the host operating system. The agent's function is to monitor the end-to-end network performance of critical flows it generates or receives in real time, such as round-trip time (RTT), throughput, or application-layer job completion time. When the agent detects a performance metric deteriorating and exceeding a preset threshold, it immediately generates a polling packet containing the victim flow's identity information (such as a 5-tuple) and sends it into the network, thereby initiating subsequent telemetry data collection and diagnostic processes.

[0105] 2. Switch Components: The switch, which forms the core of network data exchange, needs to provide the data plane telemetry capabilities and control plane interaction capabilities required by this invention. The prototype system of this invention is implemented on a programmable switch. Programmable switches allow developers to customize the packet processing logic of their data plane. P4 (Programming Protocol-independent Packet Processors) is a mainstream high-level language used for programming such switches. In the prototype implementation of this invention, the switch functionality is achieved through the combined programming of the data plane and control plane components:

[0106] 1) Data Plane Program: Programmed using languages ​​such as P4. This program is responsible for executing the core logic of this invention at line rate, mainly including:

[0107] a. Telemetry data logging: Passively and continuously update the flow-level and port-level telemetry data with PFC sensing capabilities as described above, and store it in registers in the data plane.

[0108] b. Polling packet processing: Parse the polling packets received from the network and perform different processing logic based on their content (especially the polling flags).

[0109] c. In-network causal tracing: Based on the internally recorded PFC status and the causal relationship matrix of traffic between ports, it autonomously determines which upstream or downstream switches should be forwarded to the polling data packets, so as to achieve automatic tracing of the complete abnormal causal chain.

[0110] 2) Control Plane Program: Runs on the switch's built-in CPU. When the data plane program receives a polling data packet, it reports a copy of that packet to the control plane. Upon receiving the notification, the control plane program asynchronously reads all relevant telemetry data registers from the data plane hardware via an efficient interface (such as synchronous read operations using DMA transfer). It then performs preliminary data processing (such as filtering null values) and packages the data before sending it to the source analyzer.

[0111] However, it is important to emphasize that building the prototype on a programmable switch is merely one specific technical implementation path, and not a limitation on the scope of protection of this invention. The diagnostic method proposed in this invention can be deployed on any switch hardware capable of providing the required functionality. As long as a switch (whether programmable or conventional) supports the key features required by this invention—for example, the ability to sense the PFC status of ports on the data plane, support inter-port traffic statistics, and export this telemetry data through the control plane—it can be used to deploy the diagnostic system of this invention.

[0112] 3. Source Tracing Analyzer Component: The source tracing analyzer is typically deployed on a separate server. It is responsible for receiving telemetry data fragments reported by various relevant switches. After collecting all the data required for an anomaly, the analyzer executes the source-tracing-based diagnostic algorithm described in this invention. Specifically, it uses the collected data to construct a heterogeneous source graph, identifies the specific type of anomaly (such as PFC deadlock, sudden traffic surge, etc.) through graph analysis and feature matching, and ultimately locates the root cause of the performance anomaly (such as several specific root cause flows or a particular abnormal host), presenting the diagnostic results to the network administrator.

[0113] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for diagnosing RDMA network performance anomaly based on PFC trace, characterized in that, The method comprises three steps. Step 1: passive PFC-aware telemetry data recording: passively and continuously recording telemetry data containing PFC visibility and causality awareness on the data plane of the switch, and logging the causality data for tracking the PFC propagation path; Step 2: abnormality-driven in-network causality analysis and data collection: when the detection agent of the terminal host discovers a performance abnormality, a "polling packet" is injected into the network to start the diagnosis process; when the switch receives the polling packet, the PFC causality is analyzed on the data plane of the switch, and the polling packet is intelligently forwarded only to the neighbor switches related to the present abnormality; at the same time, the switch notifies the control plane CPU to perform telemetry data collection and reporting in an asynchronous manner; Step 3: trace-based offline diagnosis analysis: after collecting all related telemetry data, the central analyzer runs a trace-based offline diagnosis algorithm, which first constructs a heterogeneous trace graph that comprehensively describes the details of the abnormality, including the victim degree of the flow, the causality propagation path of the PFC and the contribution of each flow to the congestion; Then, the diagnosis program traces the causality chain of the congestion by analyzing the trace graph, and finally locates the root cause of the performance abnormality.

2. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 1, wherein, The specific method of passively and continuously recording telemetry data containing PFC visibility and causality awareness on the data plane of the switch is as follows: first, a "PFC state register" is set in the processing pipeline of the data plane for each egress port; when the switch receives a PFC PAUSE frame sent by a downstream device from a certain ingress port, the PAUSE frame is specially processed and transmitted to the egress port pipeline, and the pause time information carried by the PAUSE frame is used to update the PFC state register of the corresponding egress port; accordingly, any normal data packet that attempts to dequeue during the "pause" state of the port is recognized by the system as a "PFC-paused data packet", and the related statistical information is also updated. The method for constructing the causality data for tracking the PFC propagation path is to construct a port-level traffic causality record structure and use a two-dimensional traffic count matrix to record the traffic size from any ingress port to any egress port in real time.

3. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 2, wherein, The telemetry data recording method is to design a round-based multi-granularity telemetry data recording and aggregation method; the switch divides time into consecutive rounds in units of a fixed time period, and maintains a ring buffer composed of multiple round data in memory; The switch uses the high-precision timestamp metadata carried by each data packet to calculate the round ID to which the data packet belongs online at high speed by intercepting specific bits of the timestamp, and updates the telemetry data to the corresponding position in the ring buffer; In this structure, both flow-level and port-level telemetry data are recorded simultaneously: For flow level, record the five-tuple, total packet number, total queue depth, etc. traditional telemetry information for each flow record, in addition to the implementation of PFC awareness, also record the number of packets suspended by PFC for the flow, the entry of the flow is indexed by the hash value of its five-tuple; For port level, directly record the total length of its egress queue and the total number of packets suspended by PFC for each port, the entry of the port is directly indexed by its port number.

4. The RDMA network performance anomaly diagnosis method based on PFC tracing according to claim 3, wherein, The specific method for analyzing PFC causality is: First, a lightweight detection agent is deployed on the terminal host in a terminal exception driven diagnostic trigger mode, which continuously monitors the end-to-end performance indicators of the outgoing flow of the terminal, and when the performance indicators of a certain flow deteriorate and exceed the preset threshold, the agent generates a polling packet and injects it into the network; Second, a polling packet format and forwarding logic for guiding causality tracking is designed, which contains the five-tuple information of the victim flow and a key polling flag in the packet header, different values of the polling flag indicate that the switch performs different tracking tasks, in order to ensure that the polling packet itself is not blocked by congestion, it is placed in a high priority queue that is not affected by PFC for transmission; And a line-speed causality analysis and forwarding process completely executed in the data plane of the switch is designed, when a switch carrying the polling packet format and forwarding logic for guiding causality tracking receives the polling packet, recursive analysis and forwarding operations are performed; Finally, an asynchronous data collection method assisted by the control plane of the switch is designed, during the tracking process, each switch that receives the polling packet, while executing the forwarding logic of the data plane, will mirror a copy of the polling packet to its CPU port, this operation will wake up the collection agent process running on the control plane of the switch, which will asynchronously and safely read the complete telemetry data recorded by the data plane, filter out invalid entries, and package the valid data into one or more large packets, and finally report to the central analysis server.

5. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 4, wherein, The specific steps of the recursive analysis and forwarding operation are: step one, victim flow path tracking: if the received polling packet flag is "01", the switch queries the forwarding table according to the victim flow five-tuple information embedded in the packet, and unicasts the polling packet to the next hop switch of the victim flow; Step two, PFC event detection: while forwarding, the switch checks its internal telemetry data to determine whether the egress port of the victim flow is suffering from PFC suspension; Step three, tracking path expansion: if PFC suspension is detected, the switch sets the higher position of the modified flag to "1" in the forwarded polling packet, and the modified flag is used to notify the downstream switch to start PFC causality path tracking. Step four, PFC causal path analysis and tracking: when a switch receives a polling packet with the flag bit as "1", it first multicasts the packet internally to all egress ports, and each egress port pipeline queries the aforementioned port-level flow causal relationship record structure to determine whether it is a causally related port causing the PFC back pressure; Step five, recursive tracking: only those ports determined to be causally related will continue to forward the polling packet upstream, and this process is recursively performed hop by hop until the original source of the PFC propagation chain is tracked.

6. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 5, wherein, The specific implementation of the offline diagnosis algorithm based on tracing is as follows: first, a heterogeneous trace graph is constructed to describe the causal chain of congestion through a "wait-waited" relationship graph, and then the structure characteristics of the heterogeneous trace graph are analyzed to identify the abnormal type and find the cause of the abnormality.

7. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 6, wherein, The construction method of the heterogeneous trace graph is as follows: the heterogeneous trace graph is defined as a directed weighted graph, and the flow and switch egress port are taken as two different nodes, the edge in the graph represents the "wait" relationship between the nodes, and the weight of the edge quantifies the strength or contribution degree of the wait relationship. Three key wait relationship edges are defined and constructed: Port-Port Wait Edge: used to describe the propagation of the causal relationship of PFC, if an out port P i is blocked by receiving a downstream PFC pause frame, then it waits for some or all of the congested downstream ports P j to empty their queues, thus building a directed edge from P i to P j with a weight that is the product of three factors: the number of packets P i blocked by PFC at P i , the proportion of traffic forwarded from P j to P i , and the degree of queue congestion at P j itself. Flow-port waiting edge: used to describe the impact of PFC suspension on a single flow, if a flow f i passes through a port P j suspended by PFC in its path i , then the flow f j is "waiting" at port P i . When data forwarding is resumed, the system constructs an edge from flow node f j to port node P i with weight defined as the number of packets suspended at P j during this period. Port-flow waiting edge: used to describe the direct queue contention relationship happened within a port, for a port P which is not paused by PFC but has queue up due to local traffic overload j , the system is built from port nodes P j pointing to all flow nodes f i in which queue up is caused, and the weight of each edge represents the net value of the contribution of flow f i to the overall congestion of the port, which is calculated by first calculating the total amount of all other flows waiting f i in the network, and then subtracting the total amount of f i waiting for other flows; The flow with the maximum weight is the contributor of congestion, and the flow with the negative weight is the victim of congestion.

8. The PFC trace-based RDMA network performance anomaly diagnosis method of claim 6, wherein, The specific method of identifying the abnormal type and finding the cause of the abnormality by analyzing the structure characteristics of the heterogeneous trace graph is as follows: By traversing the heterogeneous trace graph and matching the pre-defined multiple abnormal signatures, the type and root of the abnormality are automatically identified, and the specific type and root include: PFC back pressure caused by traffic competition: there is one or more PFC propagation paths composed of port-port wait edges in the heterogeneous trace graph, and the terminal point of the path is a port node with strong port-flow wait edges pointing to some specific flows, and these pointed flows are the root of this event; PFC storm: there is a PFC propagation path in the heterogeneous trace graph, and the terminal port node of the path does not exist any port-flow wait edge with positive weight; PFC deadlock: there is a loop between the port-port wait edges in the heterogeneous trace graph; Traditional flow competition congestion: there is no port-port wait edge in the heterogeneous trace graph, and the port-flow wait edge in the graph is analyzed to find the root flow causing the congestion.

Images