Distributed flow monitoring

Abstract

The present disclosure relates to distributed flow monitoring. One aspect provides a system and method for detecting a root cause of congestion in a network. During operation, the system can monitor flows on respective nodes within a network comprising a plurality of nodes, the nodes comprising a plurality of receive ports and a plurality of transmit ports. In response to detecting a stall event associated with a flow at a receive port, the system can determine whether transmission of the flow is stalled at the node. In response to determining that transmission of the flow is not stalled, the system can determine that the node comprises an original congestion point that caused the flow to stall. In response to determining that transmission of the flow is stalled, the system can transmit a trace flow message associated with the flow to a downstream node, the trace flow message facilitating the downstream node identifying the original congestion point.

Description

Technical Field

[0001] This disclosure relates to congestion management in networks. More specifically, this disclosure relates to implementing distributed flow monitoring to mitigate performance degradation in networks. Background Technology

[0002] Many latency-sensitive applications deployed in modern data centers (e.g., deep learning, real-time big data analytics, high-performance computing (HPC)) rely on Direct Memory Access (DMA) technology to reduce latency. DMA protocols typically require guaranteed data delivery to achieve optimal performance. However, flow control mechanisms implemented in such networks (e.g., Priority-based Flow Control (PFC) in Converged Enhanced Ethernet (CEE) or Credit-based Flow Control (CBFC) in InfiniBand) can lead to congestion propagation, which can degrade network performance. Attached Figure Description

[0003] Figure 1 The illustration shows the formation of a congestion tree and the detection of its root causes according to one aspect of this disclosure.

[0004] Figure 2A A flowchart illustrating an example process for determining the cause of stagnant flow in the receiving direction, according to one aspect of this disclosure, is presented.

[0005] Figure 2B An example process for determining the cause of a stalled flow based on a trace flow message, according to one aspect of this disclosure, is illustrated.

[0006] Figure 3 A flowchart illustrating an example process for determining the cause of stagnant flow in the transmission direction, according to one aspect of this disclosure, is presented.

[0007] Figure 4 An example functional block diagram of a network device according to one aspect of this disclosure is shown.

[0008] Figure 5 The illustration depicts a computer-readable medium (CRM) that facilitates the detection and isolation of the root cause of congestion according to one aspect of this disclosure.

[0009] In these accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0010] In Ethernet networks implementing PFC, when congestion occurs at a switch port, pause frames are sent upstream to stop traffic of the affected priority class. This backpressure propagates recursively, causing upstream switches to buffer or pause their transmissions. Each paused node (e.g., a switch or host) becomes a new congestion point, further propagating the pause upstream and forming a congestion tree rooted at the initial congestion point. In networks employing multipath or adaptive routing, congestion trees can form rapidly, significantly reducing network-wide throughput.

[0011] Once congestion becomes widespread in a network, distinguishing the original congestion point from its victims can be challenging. Packet drops, errors, and timeouts may be observed throughout the network without indication of priority or causation. Timely identification of the root cause of congestion is crucial to preventing systemic performance degradation.

[0012] This disclosure provides a distributed system that monitors multiple potential congestion points in a network to identify root causes. The distributed system may include flow monitoring agents deployed to multiple nodes in the network to detect stagnant flows. Communication between the flow monitoring agents can facilitate the detection and isolation of root causes.

[0013] As used herein, the term "node" can refer to one or more computing devices configured to emulate one or more of the functions described herein. As used herein, "computing device" can be a server, server cluster, storage array, computer equipment, workstation, desktop computer, laptop computer, switch, router, or any other processing device or apparatus that includes processing resources. In an example, a node may include processing resources communicatively coupled to at least one non-transitory computer-readable storage medium including instructions that, when executed by the processing resources, cause the node to perform certain actions and functions as described herein.

[0014] Figure 1 The illustration shows the formation of a congestion tree and the detection of its root causes according to one aspect of this disclosure. Figure 1 The left side of the diagram illustrates the start of congestion in network 100, which includes multiple interconnected switches (e.g., switches 102, 104, 106, and 108) and multiple hosts coupled to the switches (e.g., hosts 112, 114, 116, and 118). In this example, hosts 112, 114, 116, and 118 are coupled to switches 102, 104, 106, and 108, respectively.

[0015] exist Figure 1 In the example shown, a software or hardware failure at host 118 renders it unreachable for incoming traffic from hosts 114 and 116. The attempted communication paths are... Figure 1 The image is shown as a dashed arrow. A failure at host 118 causes congestion at switch 108. Switch 108 is unable to forward packets to the unresponsive host 118, thus applying flow control back to upstream switches 104 and 106, causing packet accumulation in the buffers of these upstream switches. When the buffers at switches 104 and 106 exceed their respective capacity thresholds, each switch applies flow control upstream (e.g., to switch 102). Over time, and as more traffic (e.g., from host 112 or other network endpoints) is injected into the network, a congestion tree may grow and cause widespread congestion throughout network 100.

[0016] Figure 1 The right side of the diagram illustrates a congestion tree originating from a fault on host 118, with dashed arrows indicating links affected by stagnant traffic. As congestion propagates upstream (e.g., to switches 104, 106, and 102), the tree's hierarchy obscures the root cause node (i.e., host 118), making it challenging to distinguish the initial fault from upstream collateral damage. The root causes of congestion can vary in severity and may require different mitigation actions. For example, a poorly designed application might cause long-term incasts, which may eventually clean up on their own. Suboptimal network configurations might lead to unnecessarily high throttling when responding to congestion, which should also clean up on its own. On the other hand, uncorrectable errors or invalid network configurations might cause flow control disruptions, requiring corrective action to restore the network to a healthy state. In many cases, congestion is likely to recur if the underlying problems are not addressed. Identifying the root cause of congestion is crucial.

[0017] According to some aspects of this disclosure, a network can implement a distributed flow monitoring system to monitor flows and detect and isolate the root causes of congestion. The distributed flow monitoring system may include flow monitoring agents running on each switch (and optionally on some hosts) to monitor for signs of stagnant flows. Figure 1 In the example shown, each switch can run a flow monitoring agent. For example, switch 102 runs agent 122, switch 104 runs agent 124, switch 106 runs agent 126, and switch 108 runs agent 128. The flow monitoring agent can run on the controller of each switch. Some or all hosts (e.g., hosts 112-118) can also run the flow monitoring agent.

[0018] A flow monitoring agent running on a switch can observe stagnant flows and communicate with their peers to trace the stagnant flow downstream, thereby identifying the original congestion point causing the flow to stall. The original congestion point is either the destination of the flow or some point between the current switch and the destination, from which upstream progress stalls, but downstream progress does not. The agent can begin monitoring flows before any traffic enters the network and monitors every potential point of failure, including at least every receive (RX) and transmit (TX) port of a network node. Because every point of failure is actively monitored, the primary point of failure (i.e., the root cause of the congestion) should always be the first to be discovered, significantly narrowing the search domain. Furthermore, if a known root cause can be quickly identified, it can be corrected in time before the congestion spreads to the rest of the network.

[0019] Each flow monitoring agent can access certain locally available information to detect stall events. Examples of stall events may include, but are not limited to: hardware failure, hardware timeout events, the number of lost or stalled packets exceeding a predetermined threshold, buffer space utilization exceeding a predetermined threshold, the number of received paused frames (which can indicate persistent back pressure) exceeding a predetermined threshold, and depletion of transmission credits.

[0020] Switches can maintain flow state information at each port. Each flow monitoring agent can be configured to independently monitor the flow in the RX and TX channels of each port by acquiring the flow state information to determine the direction of the affected flow, thus allowing the agent to trace the problem downstream. In some examples, the flow monitoring agent can monitor the receive and transmit buffers to determine whether the flow is stalled on the transmit or receive side (e.g., not making sufficient progress).

[0021] Each flow monitoring agent can communicate with its direct peers (e.g., flow monitoring agents running on their link partners) to trace stagnant flows downstream from one agent to the next. Communication between agents is unaffected by network congestion. For example, agent 122 can communicate with agents 124 or 126 to trace stagnant flows, and this communication is isolated from the stagnant flow it is intended to trace (i.e., it should not be affected by congestion causing the flow stagnation). Depending on some aspects, communication between flow monitoring agents can be implemented via out-of-band channels. Alternatively, each flow monitoring agent can communicate in-band with its peers using separate traffic categories or control messages.

[0022] Flow monitoring agents can report their findings, allowing network operators to diagnose and correct underlying issues. In some scenarios, congestion-causing problems can be automatically corrected without human intervention.

[0023] Figure 2AA flowchart illustrating an example process for determining the cause of stagnant flow in the receive direction, according to one aspect of this disclosure, is presented. During operation, a flow monitoring agent monitors flows on corresponding nodes within a network comprising multiple nodes (operation 202). Network nodes may be switches (e.g., Ethernet switches) with multiple receive and transmit ports, and the flow monitoring agent may run on a switch controller. The flow monitoring agent can monitor flows passing through the switch in both the receive and transmit directions. Each flow may include a series of related packets that share the same source and destination endpoints and may also have other similar characteristics, such as belonging to the same traffic category or being part of a single communication (e.g., in the case of a TCP flow).

[0024] In some examples, interconnected switches (referred to as a switch network or switch architecture) can use flow-based traffic management mechanisms (e.g., flow channels) to provide ordered packet delivery with adaptive routing and apply congestion management to individual traffic flows (i.e., flows). A flow channel is essentially a virtual or logical path through which a continuous flow (e.g., a stream) of related data packets travels between two network nodes. When each packet enters the switch architecture, it is assigned to a new or existing flow channel based on its source, destination, and other quality of service values. Each flow channel can be identified by a unique identifier (e.g., a flow ID). For example, the ingress switch of the architecture can assign a flow ID to packets belonging to the same flow. This flow ID can be a locally valid value specific to the incoming link. When a packet is forwarded to the next-hop switch, the packet enters another link, and a new flow ID can be selected for the next flow channel on the next link. Each link can have its own set of flow channels in each direction, identified by its corresponding flow IDs. As packets of a flow traverse multiple links and switch nodes, the flow IDs corresponding to that flow can form a unique chain of flow channels. As traffic moves through the switch structure, flow channels are constantly being allocated and released.

[0025] Each flow can be represented using a chain of Flow Channel Table (FCT) entries between source and destination. Each switch can maintain an Incoming Flow Channel Table (IFCT) and an Outgoing Flow Channel Table (OFCT) for each port to track traffic entering and leaving the switch. At the switch, the flow ID of an incoming packet can be used to map an entry in the IFCT. When a packet is forwarded to the next-hop switch, the packet enters another link, and a new flow ID can be selected for the next flow channel on the next link. The mapping between incoming flow IDs and outgoing flow IDs can be stored in the OFCT. As data packets belonging to a flow travel forward along its path, a data counter increments in the corresponding FCT entry. As an acknowledgment packet (ACK) travels backward along the path, the ACK counter increments. When both counters are equal, all packets downstream of the entry have been acknowledged (ACKed), and the entry can be released for use by a new flow. A flow monitoring agent can monitor the incrementing of the counters.

[0026] The flow monitoring agent can detect stagnation events associated with flows at the receive port (Operation 204). More specifically, the flow monitoring agent can monitor the status of each flow at the receive port. When a flow channel is implemented, the switch hardware can monitor the FCT (including the IFCT and OFCT for each port) to detect stagnation events. In some examples, a stagnation event can be detected when an FCT entry fails to make progress within a configurable amount of time and reports a timeout via an error flag, a counter, and a timeout bit in the entry itself. In one example, the flow monitoring agent can detect that the counter for the FCT entry corresponding to an active flow has not incremented within a configurable amount of time, indicating that the flow is stagnant.

[0027] The flow monitoring agent can determine whether the transmission of a flow is stalled (operation 206). For example, the flow monitoring agent can determine whether a detected stall event is associated with the transmission of the flow (e.g., a timed-out OFCT entry).

[0028] If the flow transmission is not stalled, the flow monitoring agent can determine that the current node includes the original congestion point (Operation 208). As mentioned earlier, the original congestion point is a point where upstream progress is stalled but downstream progress is not.

[0029] The flow monitoring agent can further identify the original congestion point and optionally perform corresponding corrective actions (Operation 210). The flow monitoring agent can identify the root cause of congestion by analyzing event logs or running diagnostic tests. For example, a node may have a hardware failure or some configuration error. The flow monitoring agent can identify the type of faulty hardware component or configuration error on the switch. The switch controller can automatically correct the identified configuration errors. In another example, an uncorrectable link error may cause one of the control frames to be lost, leading to the formation of a congestion tree. The flow monitoring agent can reset the link.

[0030] The flow monitoring agent can generate notifications indicating the initial congestion point and corrective actions (Operation 212). This notification can be sent to the network management system or network operator, who can then perform further diagnostic and / or corrective actions accordingly. In some examples, the flow monitoring agent can report the initial congestion point by generating Redfish events and adding messages to the switch controller logs. Redfish events are notifications generated by Redfish-compatible systems to inform monitoring applications of changes in system status, configuration, or hardware status. These events are part of the Redfish Events service, which allows network management systems to subscribe to specific types of alerts and receive notifications in real time.

[0031] If the node is a switch (e.g., an Ethernet switch), a flow monitoring agent running on the switch controller can identify multiple potential sending ports on the switch used to send flows downstream and check for TX stalls at any of these ports. In one example, when the flow monitoring agent detects a stalled flow at an RX port, it can determine one or more potential downstream TX ports corresponding to the stalled flow based on flow status (e.g., IFCT / OFCT entries). This list can include a single port or all ports in the switch. An IFCT entry can point to an edge port in the switch, indicating that the end of a non-multicast flow has been reached, or it can correspond to an ordered unicast flow and point to an OFCT entry at a port in the switch. Alternatively, an IFCT entry can correspond to an unordered unicast flow and point to any number of OFCT entries at one or more ports in the switch. IFCT entries can also correspond to a multicast flow and point to any number of OFCT entries and / or egress edge ports in the switch. If no TX stall is detected at any downstream port, the flow monitoring agent can determine that the node does indeed contain the original congestion point.

[0032] In response to the determination of a flow stagnation (meaning the congestion may originate from the current node or somewhere downstream), the flow monitoring agent can send a trace flow message associated with the flow to downstream nodes (Operation 214). The trace flow message helps downstream nodes identify the original congestion point. The downstream node can be a next-hop switch or a host. To send the trace flow message, the flow monitoring agent may first need to determine the downstream node for the stagnant flow. In the case of a flow channel, the flow monitoring agent can trace the stagnant flow downstream based on the corresponding FCT entry. More specifically, tracing a flow through a downstream port may involve following the flow's IFCT entries on one or more ports to OFCT entries, and then following the OFCT entry corresponding to the stagnant flow to the corresponding IFCT entry at the downstream next-hop switch.

[0033] Tracing flow messages can be sent to downstream next-hop switches via congestion-free mechanisms. For example, a flow monitoring agent can send tracing flow messages out-of-band, assign specific traffic categories to them, or send them as control messages (e.g., as credit control frames using credit-based flow control) via in-band channels. Tracing flow messages allow downstream next-hop nodes to identify the original congestion point, which can be the current node, a downstream node, or a different node further downstream.

[0034] When flow channels are implemented on a switch, each OFCT entry is associated with a single IFCT entry on the peering switch. The trace flow message to the peer agent can include the flow ID corresponding to the OFCT, allowing the peer agent to check if the corresponding IFCT entry has timed out (i.e., RX stalling on the peering switch). If RX stalling is detected, tracing continues from the IFCT entry. Otherwise, the peer agent can determine that the upstream node or the link between the two nodes includes the original point of congestion.

[0035] Based on several factors, upon receiving a tracing flow message from an upstream node, the flow monitoring agent can determine whether an RX stall has occurred on the port receiving the message. If not, the flow monitoring agent can determine that the flow has been traced back to the original congestion point, which could be an upstream node or the link between the upstream node and the current node. If an RX stall has occurred on the port, the flow monitoring agent can use... Figure 2A The process shown continues to trace the flow downstream until the root cause of the congestion is determined.

[0036] Figure 2B The illustration depicts a process for determining the cause of a stalled flow based on a trace flow message, according to one aspect of this disclosure. During operation, a node receives a trace flow message associated with the flow at a receive port (operation 222). The trace flow message can be sent from a flow monitoring agent running on an upstream node (e.g., an upstream hop switch) to indicate that a TX stall occurred at the corresponding TX port in the upstream node. Note that a TX stall could be caused by node failure or backpressure from downstream.

[0037] The flow monitoring agent running on the current node can determine whether the flow is stalled at the receive port (Operation 224). For example, the flow monitoring agent can determine whether a detected stall event occurred at the port receiving trace flow messages (e.g., a timed-out IFCT entry).

[0038] If not, the flow monitoring agent can determine that the upstream node or the link between the upstream node and the current node may include the original congestion point (Operation 226). This determination is based on the fact that upstream traffic is stagnant but downstream traffic is not.

[0039] The flow monitoring agent can further identify the original congestion points and optionally perform corresponding corrective actions (Operation 234). The flow monitoring agent can identify the root cause of congestion by analyzing event logs or running diagnostic tests. For example, a node may have a hardware failure or some configuration error. The flow monitoring agent can identify the type of faulty hardware component or configuration error on the switch. The switch controller can automatically correct the identified configuration errors.

[0040] Subsequently, the flow monitoring agent can generate a notification indicating the original congestion point and corrective actions (Operation 236). This notification can be sent to the network management system or network operator, who can then perform further diagnostic and / or corrective actions accordingly. In some examples, the flow monitoring agent can report the original congestion point by generating a Redfish event and adding a message to the switch controller log.

[0041] If a flow stalls at the receive port, the flow monitoring agent can determine whether the flow transmission has stalled (operation 228). For example, the flow monitoring agent can detect a timed-out OFCT entry and determine that the flow transmission has stalled.

[0042] If the flow transmission is not stalled, the flow monitoring agent can determine that the current node includes the original congestion point (Operation 230). This determination is based on the fact that upstream flow is stalled but downstream flow is not.

[0043] The flow monitoring agent can further identify the original congestion points and optionally perform corresponding corrective actions (action 234). For example, the flow monitoring agent can identify the root cause of congestion by analyzing event logs or running diagnostic tests. If a configuration error is detected on the switch, the switch controller can automatically correct the identified configuration error.

[0044] Subsequently, the flow monitoring agent can generate notifications indicating the original congestion point and corrective actions (Operation 236). For example, the flow monitoring agent can report the original congestion point by generating a Redfish event and adding a message to the switch controller log.

[0045] If the flow transmission stalls, the flow monitoring agent can send a tracing flow message to downstream nodes (Operation 232). This process can be repeated along multiple nodes in the downstream direction until the original congestion point is identified.

[0046] Figure 3 A flowchart illustrating a process for determining the cause of a stalled flow in the transmission direction, according to one aspect of this disclosure, is presented. During operation, a flow monitoring agent monitors the flows on the network switch (operation 302). When using flow channels, each flow can be represented using a chain of Flow Channel Table (FCT) entries between the source and destination.

[0047] A flow monitoring agent can detect stagnant events (Operation 304) associated with a flow at a sending port. The network switch can be an Ethernet switch that maintains the flow state at each port. The flow monitoring agent can use the flow state to identify stagnant flows (i.e., flows that are not making progress). In some examples, flow channels are used, and the flow monitoring agent can monitor flows by searching timeout entries in the FCT.

[0048] The flow monitoring agent can determine whether a sending port is coupled to a second network switch or a host's network interface controller (NIC) (Operation 306). The flow monitoring agent can communicate with the port's link partner to determine whether the link partner is a peer switch or a host. Alternatively, the network management system maintains the network topology and can indicate to the flow monitoring agent on the switch whether the sending port is coupled to the switch or a host.

[0049] If the sending port is coupled to a second network switch, the flow monitoring agent can send a trace flow message associated with the stalled flow to the second network switch to allow the second network switch to continue tracing the flow downstream (Operation 308). The trace flow message can be sent via a congestion-free mechanism.

[0050] If the sending port is coupled to a host's NIC, the flow monitoring agent can determine whether the host is implementing flow monitoring (Operation 310). Depending on some aspects, each switch in the network runs a flow monitoring agent, while hosts may or may not implement flow monitoring. The flow monitoring agent on the switch can communicate with the host to determine whether the host is implementing flow monitoring. Alternatively, the network management system maintains the configuration status of all devices in the network (including switches and hosts) and can notify the coupled hosts to the flow monitoring agent on the switch whether they are implementing flow monitoring.

[0051] If the host implements a flow monitoring agent, the flow monitoring agent on the switch can send trace flow messages to the agent on the host (Operation 312). The trace flow messages can be sent via a congestion-free mechanism.

[0052] The host agent can determine whether an RX stall has occurred at the NIC (Operation 314). The host agent can monitor the receive port (e.g., by checking the depth of the receive buffer) to determine whether a stall has occurred at the NIC's receive port.

[0053] If no RX stagnation occurs, the flow monitoring agent on the switch can determine that the original congestion point may include a host, a peer switch, a NIC, or a link between a peer switch and a NIC (Operation 316). If an RX stagnation is observed at the NIC, the host agent can determine that the original congestion point is either the host or the NIC (Operation 318). If the host does not implement a flow monitoring agent, the original congestion point can be the switch, the NIC, a link between the switch and a link, or the host (Operation 316). These determinations are based on the fact that if upstream traffic is stagnant but downstream traffic is not, the device includes the original congestion point.

[0054] The flow monitoring agent on the switch or host can further identify the original congestion point and optionally perform corresponding corrective actions (Operation 320). For example, the flow monitoring agent can identify the root cause of the congestion by analyzing event logs or running diagnostic tests. If a configuration error is detected on the switch, the switch controller can automatically correct the identified configuration error.

[0055] Subsequently, the flow monitoring agent can generate a notification indicating the original congestion point and corrective action (Operation 322). In some examples, the flow monitoring agent can report the original congestion point by generating a Redfish event and adding a message to the switch controller log.

[0056] In some examples, the host agent can detect TX stagnation on the NIC and send a message to the flow monitoring agent running on the peering switch to allow the switch agent to trace the flow downstream.

[0057] Figure 2A , 2B All or any part of the operations shown in 3 can be performed by, for example, a device or a group of devices (e.g., Figure 1 (The host or switch shown) is used to execute this. Although Figure 2A , 2B The example processes in 3 illustrate a specific order for performing certain operations, but these processes are not limited to this order. Operations shown consecutively in the flowchart can be performed in different orders, and can be performed simultaneously, partially simultaneously, or in combination.

[0058] Figure 4 The illustration shows a functional block diagram of a network device according to one aspect of this disclosure. Network device 400 may include any physical device that allows hardware on a computer network to communicate and interact with each other. Examples of network device 400 may include switches, routers, gateways, access points, network interface cards (NICs), host devices, etc. Network device 400 may include multiple communication ports, such as RX port 402 and TX port 404, for communicating with peer network devices.

[0059] Network device 400 may further include one or more processing resources (e.g., processing resource 406), one or more storage devices (e.g., storage device 408), and a flow monitoring system 410. Network device 400 may include... Figure 4 The entities shown are fewer or more units. In some examples, network device 400 may be a switch deployed in the network.

[0060] In the examples described herein, processing resources may include, for example, one or more processors included in a single computing device or distributed across multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry suitable for fetching and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may fetch, decode, and execute instructions stored on a storage medium to perform the function described with respect to the instructions stored on the computer-readable medium. In other examples, the function described with respect to any instructions described herein may be implemented in the form of electronic circuitry, executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located in or away from the computing device executing the instructions but accessible (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media.

[0061] The flow monitoring system 410 may include any number of software units, hardware units, and firmware units that work together to achieve the goal of monitoring flows in order to detect and isolate the root causes of congestion. According to some aspects, the flow monitoring system 410 may include instructions that, when executed by the processing resource 406, cause the processing resource 406 to perform the methods and / or processes described in this disclosure. Specifically, the flow monitoring system 410 may include methods for monitoring flows on the network device 400 (as described above regarding...). Figure 2A The instruction 412 (described in operation 202) can be used to monitor the status of each flow at each RX port and TX port. When using flow channels, instruction 412 may include instructions for checking FCT entries (including IFCT and OFCT entries) at each port in the switch.

[0062] Flow monitoring system 410 may include methods for detecting stagnation events associated with the flow at the receiving port (such as those mentioned above). Figure 2AInstruction 414 (described in operation 204) may include, but is not limited to: hardware timeout events, the number of dropped packets exceeding a predetermined threshold, the number of stalled packets exceeding a predetermined threshold, buffer utilization exceeding a predetermined threshold, backpressure exceeding a predetermined time threshold, or credit exhaustion. When using a streaming channel, instruction 412 may include instructions for detecting timeout events associated with FCT entries.

[0063] Flow monitoring system 410 may include measures to determine, in response to determining that there is no stagnation in the flow transmission at network device 400, that network device 400 includes the original congestion point (as mentioned above). Figure 2A Instruction 416 (described in operations 206 and 208 shown). Furthermore, instruction 412 may include instructions for identifying multiple potential sending ports for transmitting the stream downstream and instructions for determining whether the transmission of the stream has stalled at any potential sending port. When implementing a stream channel, instruction 412 may determine multiple potential downstream TX ports based on the IFCT entries of the RX port.

[0064] The flow monitoring system 410 may include methods for identifying the original congestion point and optionally performing corresponding correction operations (as mentioned above). Figure 2A The instructions 418 (described in operation 210) shown in the diagram. Instructions 418 may include instructions for reading the event log on the network device 400 to identify hardware failures or configuration errors causing congestion. Instructions 418 may also include instructions that can automatically correct configuration errors.

[0065] Flow monitoring system 410 may include notifications for generating information about the original congestion point and correction operations (as mentioned above). Figure 2A The instruction 420 (described in operation 212) is shown in the diagram. In some examples, instruction 420 can report the original congestion point by generating a Redfish event and adding a message to the device log.

[0066] Flow monitoring system 410 may include sending flow-associated tracking messages (as described above) to downstream network devices in response to determining that flow transmission has stalled at network device 400. Figure 2A Instruction 422 (described in operations 206 and 214 shown). Upon receiving a trace flow message, if the downstream network device determines that network device 400 does not include the original congestion point, the downstream network device can further trace the flow downstream.

[0067] The flow monitoring system 410 may include a ratio Figure 4The instructions shown are further examples. For instance, the flow monitoring system 410 may include instructions for detecting a stall event associated with a flow at a sending port, and instructions for determining, in response to observing an RX stall at a NIC, that a host coupled to the sending port or its NIC includes the original congestion point. Additionally, the flow monitoring system 410 may include instructions for receiving a flow tracking message associated with a flow from an upstream network device, and instructions for determining, in response to determining that there is no stall in the flow at a receiving port, that an upstream network device or the link between network device 400 and the upstream network device includes the original congestion point.

[0068] Figure 5 A computer-readable medium (CRM) facilitating the detection and isolation of the root cause of congestion according to one aspect of this disclosure is illustrated. CRM 500 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a “computer-readable storage medium” can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein may be any one or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0069] The CRM 500 can store flows used for monitoring multiple nodes within a network (as mentioned above). Figure 2A Instruction 510 (as described in operation 202) is used to detect stagnation events associated with the stream at the node's receive port (as described above regarding...). Figure 2A Instruction 520 (as described in operation 204); for determining that the node includes the original congestion point in response to determining that the flow transmission has no stall at the node (as mentioned above regarding...). Figure 2A Instruction 530 (as described in operations 206 and 208) is used to identify the original congestion point and optionally perform the corresponding correction operation (as described above regarding...). Figure 2A Instruction 540 (as described in operation 210); used to generate notifications regarding the original congestion point and correction operation (as described above regarding...). Figure 2A Instruction 550 (as described in operation 212 shown); and instruction 550 for sending a stream-associated tracking message (as described above regarding the operation 212) to downstream nodes in response to a determination that the transmission of the stream has stalled at a node. Figure 2A Instruction 560 (described in operations 206 and 214 shown).

[0070] CRM 500 can include more Figure 5 The instructions shown are further examples. For instance, the CRM 500 may include instructions for detecting a stall event associated with a flow at a sending port, and instructions for determining, in response to observing an RX stall at a NIC, that a host coupled to the sending port or its NIC includes the original congestion point. The CRM 500 may further include instructions for receiving a trace flow message associated with the flow from an upstream node, and instructions for determining, in response to determining that there is no stall in the flow at a receiving port, that the link between an upstream network device or node and the upstream node includes the original congestion point.

[0071] Typically, this disclosure provides a solution to the problem of detecting and isolating the root cause of congestion in lossless networks, such as Ethernet networks employing Priority-Based Flow Control (PFC). Distributed systems may include flow monitoring agents deployed on each switch within the network. The flow monitoring agents can independently monitor received and transmitted traffic at each port to detect traffic stagnation in both directions. In response to detecting stagnant flow, the agent can communicate with its peers (e.g., by sending trace flow messages) to trace the flow downstream from one agent to the next-hop agent until the root cause of the congestion is determined. Trace flow messages can be sent using congestion-independent mechanisms. Agents on the root cause node can take corrective actions and report the problem and the actions taken.

[0072] One aspect of this disclosure provides a system and method for detecting the root cause of congestion in a network. During operation, the system can monitor flows on corresponding nodes within a network comprising multiple nodes, each node including multiple receive ports and multiple send ports. In response to detecting a stall event associated with a flow at a receive port, the system can determine whether the flow transmission has stalled at that node. In response to determining that the flow transmission has not stalled, the system can determine that the node contains the original congestion point causing the flow stall. In response to determining that the flow transmission has stalled, the system can send a trace flow message associated with the flow to downstream nodes, the trace flow message facilitating the downstream nodes to identify the original congestion point.

[0073] According to variations of this approach, sending a tracking stream message may include sending the tracking stream message via an out-of-band channel, assigning a special traffic category to the tracking stream message, or sending the tracking stream message as a control message via an in-band channel, thereby preventing the tracking stream message from being affected by congestion.

[0074] Depending on the variations, a stall event may include one or more of the following: hardware timeout; the number of dropped packets exceeds a predetermined threshold; the number of stalled packets exceeds a predetermined threshold; buffer utilization exceeds a predetermined threshold; back pressure continues to exceed a predetermined time threshold; or credit depletion.

[0075] According to a variation of this approach, the node is a network switch, and the system can identify multiple potential sending ports on the network switch used to send flows downstream, and in response to determining that the flow transmission has not stalled at any of the potential sending ports, determine that the network switch includes the original congestion point that caused the flow stall.

[0076] In a further variation, the system can determine whether the sending port is coupled to a second network switch or to the host's network interface controller (NIC) in response to the determination that the flow transmission has stalled at the sending port. If the sending port is coupled to the second network switch, the system can send a flow tracking message to the second network switch.

[0077] According to a further variation, in response to a first NIC on a first host that does not implement a flow monitoring agent, the system can determine that the node, the first NIC, the first host, or the link between the first NIC and the node is the original congestion point. In response to a second NIC on a second host that implements a flow monitoring agent, the system can determine whether the flow is stalled at a receive port on the second NIC. In response to determining that the flow is stalled at a receive port on the second NIC, the system can determine that the second host or the second NIC includes the original congestion point. In response to determining that the flow is not stalled at a receive port on the second NIC, the system can determine that the node, the second NIC, the second host, or the link between the second NIC and the node includes the original congestion point.

[0078] According to a variation of this, the system can perform correction operations corresponding to the identified original congestion points and generate notifications about the identified original congestion points and correction operations.

[0079] According to a variation of this approach, in response to receiving a tracking flow message associated with the second flow from an upstream node at the second receiving port, the system can determine whether the second flow is stalled at the second receiving port. In response to determining that the second flow is not stalled at the second receiving port, the system can determine that the upstream node or the link between upstream nodes includes the original congestion point that caused the second flow to stall.

[0080] Depending on the variation, the system can maintain flow channel tables for receive and transmit ports on the nodes. Flow monitoring can include tracking entries in the flow channel tables.

[0081] One aspect of this disclosure provides a network device. The network device may include: a plurality of receive ports and transmit ports for coupling the network device to a plurality of network devices in a network; at least one processing resource; and a non-transitory computer-readable storage medium storing instructions that, when executed by the processing resource, cause the processing resource to perform a method. The method may include: monitoring flows on the network device; determining, in response to detecting a stall event associated with a flow at a receive port, whether the transmission of the flow has stalled at a node; determining, in response to determining that the transmission of the flow has not stalled, that the network device includes an original congestion point causing the flow stall; and, in response to determining that the flow transmission has stalled, sending a flow tracking message associated with the flow to a downstream network device, the flow tracking message facilitating the downstream network device to identify the original congestion point.

[0082] One aspect of this disclosure provides a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method. The method may include: monitoring a flow on a network device; determining, in response to detecting a stall event associated with a flow at a receiving port, whether the transmission of the flow has stalled at a node; determining, in response to determining that the transmission of the flow has not stalled, that the node includes an original congestion point causing the flow stall; and, in response to determining that the transmission of the flow has stalled, sending a flow tracking message associated with the flow to a downstream node, the flow tracking message facilitating the downstream node to identify the original congestion point.

[0083] Within this disclosure, the term "switch" is used in a general sense and can refer to any standalone or structured switch operating at any network layer. "Switch" should not be construed as limiting aspects of the invention to Layer 2 networks. Any device that can forward traffic to an external device or another switch can be called a "switch." Any physical or virtual device (e.g., a virtual machine or switch operating on a computing device) that can forward traffic to an end device can be called a "switch." Examples of "switch" include, but are not limited to, Layer 2 switches, Layer 3 routers, routing switches, components of Gen-Z networks, or structured switches comprising multiple similar or heterogeneous smaller physical and / or virtual switches.

[0084] The methods and processes described in the Detailed Description section can be embodied in code and / or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored within the computer-readable storage medium.

[0085] The methods and processes described above may be included in hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code at specific times, and other programmable logic devices now known or developed later. When a hardware module or device is activated, it executes the methods and processes included therein.

[0086] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0087] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method, the method comprising: Monitoring flows on corresponding nodes within a network comprising multiple nodes, wherein the nodes include multiple receive ports and multiple send ports; In response to detecting a stall event associated with a stream at the receiving port, determine whether the transmission of the stream has stalled at the node; In response to determining that the flow transmission is not stalled, the node is determined to include the original congestion point that caused the flow to stall; as well as In response to determining that the flow has stalled, a trace flow message associated with the flow is sent to a downstream node, the trace flow message enabling the downstream node to identify the original congestion point.

2. The computer-implemented method as described in claim 1, wherein, Sending the tracking stream message includes: The tracking stream message is transmitted via an out-of-band channel; Assign a specific traffic category to the tracking stream message; or The tracking stream message is transmitted as a control message via an in-band channel; This prevents the tracking stream messages from being affected by the congestion.

3. The computer-implemented method as described in claim 1, wherein, The stagnation event includes one or more of the following: Hardware timeout events; The number of dropped packets exceeded a predetermined threshold; The number of stalled groups exceeds a predetermined threshold; The buffer utilization rate exceeds the predetermined threshold; The back pressure continues to exceed the predetermined time threshold; or Credit depletion.

4. The computer-implemented method as described in claim 1, wherein, The node is a network switch, and the method further includes: Identify multiple potential sending ports on the network switch used to send the flow downstream; and In response to determining that the transmission of the flow did not stall at any of the potential sending ports, the network switch is determined to include the original congestion point that caused the flow to stall.

5. The computer-implemented method of claim 4, further comprising: In response to determining that the transmission of the stream has stalled at the sending port, it is determined whether the sending port is coupled to a second network switch or a host network interface controller (NIC); as well as In response to the sending port being coupled to the second network switch, the tracking flow message is sent to the second network switch.

6. The computer-implemented method of claim 5, further comprising: In response to the sending port being coupled to a first NIC of a first host that does not implement a flow monitoring agent, it is determined that the node, the first NIC, the first host, or the link between the first NIC and the node includes the original congestion point; In response to the sending port being coupled to a second NIC of a second host implementing a flow monitoring agent, it is determined whether the flow is stalled at a receiving port on the second NIC; In response to determining that the flow is stalled at the receive port on the second NIC, it is determined that the second host or the second NIC includes the original congestion point; as well as In response to determining that the flow is not stalled at the receive port on the second NIC, it is determined that the node, the second NIC, the second host, or the link between the second NIC and the node includes the original congestion point.

7. The computer-implemented method of claim 1, further comprising: Perform the correction operation corresponding to the identified original congestion point; as well as Generate a notification regarding the identified original congestion point and the correction operation.

8. The computer-implemented method of claim 1, further comprising: In response to receiving a tracking stream message associated with the second stream from the upstream node at the second receiving port, determine whether the second stream is stalled at the second receiving port; In response to determining that the second flow is not stalled at the second receiving port, it is determined that the upstream node or the link between the node and the upstream node includes the original congestion point that caused the second flow to stall.

9. The computer-implemented method of claim 1, further comprising maintaining flow channel tables for the receive port and the transmit port on the node, wherein, Monitoring the flow includes tracking entries in the flow channel table.

10. A network device, comprising: Multiple receiving ports and transmitting ports are used to couple the network device to multiple network devices in the network; At least one processing resource; as well as A non-transitory computer-readable storage medium storing instructions that, when executed by the processing resource, cause the processing resource to perform a method, the method comprising: Monitor the flow on the network device; In response to detecting a stall event associated with a stream at the receiving port, determine whether the transmission of the stream has stalled at the node; In response to determining that the transmission of the flow has not stalled, the network device includes the original congestion point that caused the flow to stall; and In response to determining that the flow has stalled, a trace flow message associated with the flow is sent to a downstream network device, the trace flow message enabling the downstream network device to identify the original congestion point.

11. The network device as claimed in claim 10, wherein, The stagnation event includes one or more of the following: Hardware timeout events; The number of dropped packets exceeded a predetermined threshold; The number of stalled groups exceeds a predetermined threshold; The buffer utilization rate exceeds the predetermined threshold; The back pressure continues to exceed the predetermined time threshold; or Credit depletion.

12. The network device as claimed in claim 10, wherein, The network device is a switch, and the method includes: Identify multiple potential sending ports on the switch used to send the flow downstream; and In response to determining that the transmission of the flow did not stall at any of the potential sending ports, the switch is determined to include the original congestion point that caused the flow to stall.

13. The network device as claimed in claim 12, wherein, The method further includes: In response to determining that the transmission of the stream has stalled at the sending port, it is determined whether the sending port is coupled to a second network switch or a host network interface controller (NIC); In response to the sending port being coupled to the second network switch, the tracking flow message is sent to the second network switch; In response to the sending port being coupled to a first NIC of a first host that does not implement a flow monitoring agent, it is determined that the node, the first NIC, the first host, or the link between the first NIC and the node includes the original congestion point; In response to the sending port being coupled to a second NIC of a second host implementing a flow monitoring agent, it is determined whether the flow is stalled at a receiving port on the second NIC; In response to determining that the flow is stalled at the receive port on the second NIC, it is determined that the second host or the second NIC includes the original congestion point; and In response to determining that the flow is not stalled at the receive port on the second NIC, it is determined that the node, the second NIC, the second host, or the link between the second NIC and the node includes the original congestion point.

14. The network device as claimed in claim 10, wherein, The method further includes: Perform correction operations corresponding to the identified original congestion points; and Generate a notification regarding the identified original congestion point and the correction operation.

15. The network device as claimed in claim 10, wherein, The method further includes: In response to receiving a tracking flow message associated with the second flow from an upstream network device at the second receiving port, determine whether the second flow is stalled at the second receiving port; In response to determining that the second flow is not stalled at the second receiving port, it is determined that the upstream node or the link between the node and the upstream node includes a second original congestion point that caused the second flow to stall.

16. The network device of claim 10, wherein, The method further includes maintaining flow channel tables for the receiving port and the transmitting port, wherein monitoring the flow includes tracking entries in the flow channel tables.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform a method, the method comprising: Monitoring flows on corresponding nodes within a network comprising multiple nodes, wherein the nodes include multiple receive ports and multiple send ports; In response to detecting a stall event associated with a stream at the receiving port, determine whether the transmission of the stream has stalled at the node; In response to determining that the flow transmission is not stalled, the node is determined to include the original congestion point that caused the flow to stall; as well as In response to determining that the flow has stalled, a trace flow message associated with the flow is sent to a downstream node, the trace flow message enabling the downstream node to identify the original congestion point.

18. The non-transitory computer-readable storage medium of claim 17, wherein, The node is a network switch, and the method includes: Identify multiple potential sending ports on the network switch used to send the flow downstream; and In response to determining that the transmission of the flow did not stall at any of the potential sending ports, the network switch is determined to include the original congestion point that caused the flow to stall.

19. The non-transitory computer-readable storage medium of claim 18, wherein, The method further includes: In response to determining that the transmission of the stream has stalled at the sending port, it is determined whether the sending port is coupled to a second network switch or a host network interface controller (NIC); In response to the sending port being coupled to the second network switch, the tracking flow message is sent to the second network switch; In response to the sending port being coupled to a first NIC of a first host that does not implement a flow monitoring agent, it is determined that the node, the first NIC, the first host, or the link between the first NIC and the node includes the original congestion point; In response to the sending port being coupled to a second NIC of a second host implementing a flow monitoring agent, it is determined whether the flow is stalled at a receiving port on the second NIC; In response to determining that the flow is stalled at the receive port on the second NIC, it is determined that the second host or the second NIC includes the original congestion point; and In response to determining that the flow is not stalled at the receive port on the second NIC, it is determined that the node, the second NIC, the second host, or the link between the second NIC and the node includes the original congestion point.

20. The non-transitory computer-readable storage medium of claim 17, wherein, The method further includes: In response to receiving a tracking flow message associated with the second flow from an upstream network device at the second receiving port, determine whether the second flow is stalled at the second receiving port; In response to determining that the second flow is not stalled at the second receiving port, it is determined that the upstream node or the link between the node and the upstream node includes a second original congestion point that caused the second flow to stall.

Images