Network interface controller and method

Abstract

A network interface controller (NIC) capable of facilitating fine-grained flow control (FGFC) is provided. The NIC may be equipped with a network interface, an FGFC logic block, and a traffic management logic block. During operation, the network interface may determine that a control frame from a switch is associated with FGFC. The network interface may then identify a data flow indicated in the control frame for applying FGFC. The FGFC logic block may insert information from the control frame into an entry in a data structure stored in the NIC. The traffic management logic block may identify an entry in the data structure based on one or more fields of a packet belonging to the flow. The traffic management logic block may then determine whether to allow forwarding of the packet based on the information in the entry.

Description

Technical Field

[0001] The present disclosure relates generally to the field of networking technology. More particularly, the present disclosure relates to systems and methods for facilitating a network interface controller (NIC) with fine-grained flow control (FGFC) support. Background Art

[0002] As network-enabled devices and applications become increasingly ubiquitous, diverse traffic types and increasing network loads continue to place greater performance demands on the underlying network architecture. For example, applications such as high-performance computing (HPC), media streaming, and the Internet of Things (IoT) generate diverse traffic types with unique characteristics. Consequently, in addition to conventional network performance metrics such as bandwidth and latency, network architectures continue to face challenges such as scalability, versatility, and efficiency. Summary of the Invention

[0003] A network interface controller (NIC) capable of facilitating fine-grained flow control (FGFC) is provided. The NIC may be equipped with a network interface, an FGFC logic block, and a traffic management logic block. During operation, the network interface may determine that a control frame from a switch is associated with FGFC. The network interface may then identify a data flow indicated in the control frame for applying FGFC. The FGFC logic block may insert information from the control frame into an entry in a data structure stored in the NIC. The traffic management logic block may identify an entry in the data structure based on one or more fields of a packet belonging to the flow. The traffic management logic block may then determine whether to allow forwarding of the packet based on the information in the entry. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An exemplary network is shown.

[0005] Figure 2A An exemplary NIC chip having multiple NICs is shown.

[0006] Figure 2B An exemplary architecture of a NIC is shown.

[0007] Figure 2C An exemplary FGFC selection process in the NIC is shown.

[0008] Figure 3A An exemplary FGFC control process in a NIC is shown.

[0009] Figure 3B An exemplary data packet forwarding process associated with FGFC in a NIC is shown.

[0010] Figure 4AA flow chart showing the Ethernet-based FGFC process in a NIC is shown.

[0011] Figure 4B A flow chart of the FGFC process based on the application programming interface (API) in the NIC is shown.

[0012] Figure 4C A flow chart illustrating exemplary packet processing for facilitating FGFC in a NIC is shown.

[0013] Figure 5 An exemplary computer system equipped with a NIC with FGFC support is shown.

[0014] In the drawings, like reference numerals denote like drawing elements. DETAILED DESCRIPTION

[0015] Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown.

[0016] Overview

[0017] The present disclosure describes systems and methods for facilitating fine-grained flow control (FGFC) in network interface controllers. NICs allow hosts to communicate with data-driven networks. By maintaining state information for individual packet flows, the network can adapt to dynamic data traffic with fast, efficient congestion control. More specifically, packets injected into a network switch can be classified into flows, which can be mapped to their layer-2, layer-3, or other protocol-specific header information. Each flow can be marked with a unique identifier that is local to the switch's input port and provided with a flow-specific input buffer, allowing each flow to be individually flow controlled. Furthermore, packets for the corresponding flow can be acknowledged upon arrival at the network's exit point, and acknowledgement packets can be sent back to the flow's entry point along the same data path in the opposite direction. As a result, each switch can obtain state information for the active packet flows it is forwarding and can perform highly responsive flow-specific flow control. This flow control can allow the network to operate at higher capacity while providing versatile traffic engineering capabilities.

[0018] The embodiments described herein address the problem of flow-level congestion management by (i) identifying a flow causing congestion in a NIC, and (ii) throttling the forwarding rate of packets belonging to the flow at the NIC.

[0019] Network congestion in a network (e.g., a switch fabric) can exhaust the packet buffers of switches within the network. Using existing techniques, a switch facing congestion can instruct upstream switches to pause or slow down the rate at which packets are injected into a specific traffic class. However, this class-level congestion control approach can affect all data flows within that class. For example, traffic from multiple applications can belong to the same traffic class. Consequently, packets that are not causing congestion may be adversely affected by this congestion control strategy.

[0020] To address this issue, a congested switch can transmit flow-specific congestion notifications to a link partner, which can be a NIC on a host device. Congestion notifications can generate "back pressure" on a sequence of packets belonging to the flow that caused the congestion (e.g., an Internet Protocol (IP)-level flow or an application-level flow), rather than throttling traffic from all applications and services in a traffic class. By identifying congestion at the flow level, the switch can allow the NIC to facilitate fine-grained flow control (FGFC).

[0021] In some embodiments, upon detecting congestion, the switch can identify a sequence of packets that has caused the congestion. Such a sequence of packets can be referred to as a flow. The switch can then provide this information to a link partner (such as a NIC) by sending a "shutdown" control frame (which can be referred to as an XOFF frame). Upon receiving the XOFF frame, the NIC can avoid sending packets for this flow and buffer the packets within the NIC. The NIC then relies on the switch to manage the flow. Based on the congestion associated with the flow, the switch can send control frames to the NIC, which can be referred to as credit frames.

[0022] Upon receiving the credit frame, the NIC can forward more packets belonging to the flow to the switch based on the corresponding amount indicated by the credit frame. This allows the NIC to limit the number of packets for the flow while facilitating regular forwarding for other flows. If congestion is relieved, the switch can send an "on" control frame, which can be called an XON frame. Upon receiving the XON frame, the NIC releases traffic from the FGFC and initiates regular forwarding for packets belonging to the flow.

[0023] One embodiment of the present invention provides a NIC. The NIC may be equipped with a network interface, an FGFC logic block, and a traffic management logic block. During operation, the network interface may determine that a control frame from a switch is associated with FGFC. The network interface may then identify a data flow indicated in the control frame for application of FGFC. The FGFC logic block may insert information from the control frame into an entry in a data structure stored in the NIC. The traffic management logic block may identify an entry in the data structure based on one or more fields of a packet belonging to the flow. The traffic management logic block may then determine whether to allow forwarding of the packet based on the information in the entry.

[0024] In a variation on this embodiment, the network interface may determine whether to process a control frame at the network interface based on the type of the control frame.

[0025] In further variations, the network interface may provide information from one or more fields of the control frame to the traffic management logic block based on the type of the control frame.

[0026] In a variation on this embodiment, the network interface may generate an event for a flow based on the duration value and credit value from information in the control frame.The event may be an internal control message that may indicate whether to start or terminate the FGFC for the flow.

[0027] In a variation on this embodiment, the FGFC logic block may insert information into the entry by: (i) determining a duration value for applying FGFC to the flow based on information in the control frame, and (ii) updating a duration counter in the entry based on the duration value.

[0028] In a variation on this embodiment, the FGFC logic block may insert information into the entry by: (i) determining credit information based on information in the control frame, the credit information indicating the amount of data for the flow that can be forwarded, and (ii) updating a duration counter in the entry based on the duration value.

[0029] In another variation, the traffic management logic block may distribute the data packet to a plurality of message slicing units (MCUs). The traffic management logic block may then arbitrate among the plurality of MCUs to select an MCU for forwarding the data packet based on the credit value in the entry.

[0030] In a variation on this embodiment, the FGFC logic block may insert information into an entry by: (i) determining whether one or more fields match an existing entry in a data structure, (ii) if no match is found, determining a new entry in the data structure, and (iii) inserting information from the one or more fields into the new entry.

[0031] In another variation, the FGFC logic block may determine whether the data structure is available for the new entry. If the data structure is not available, the FGFC logic block may discard the control frame.

[0032] In a variation on this embodiment, an entry may include one or more of: an identifier for the flow which may be an index of the entry, a validity flag indicating whether the entry is valid, a duration counter indicating a duration value for applying the FGFC to the flow, a credit value indicating an amount of data of the flow that can be forwarded, and an event queue identifier.

[0033] In variations on this embodiment, the FGFC logic block may be associated with a network interface or traffic management logic block.

[0034] In this disclosure, Figure 1 The description is associated with the network architecture and combined with Figure 2A The description that follows provides more details about the architecture and operation associated with a NIC that supports FGFC.

[0035] In this disclosure, packet traffic may also be referred to as a "packet flow," or simply a "flow." The data path traversed by a flow, along with its configuration information maintained by the switch, may be referred to as a "flow channel." Furthermore, the terms "buffer" and "queue" are used interchangeably in this disclosure.

[0036] Figure 1An exemplary network is shown. In this example, a network of switches 100 (which may also be referred to as a "switch fabric") may include switches 102, 104, 106, 108, and 110. Each switch may have a unique address or ID within the switch fabric 100. Various types of devices and networks may be coupled to the switch fabric. For example, a storage array 112 may be coupled to the switch fabric 100 via switch 110; an InfiniBand (IB)-based HPC network 114 may be coupled to the switch fabric 100 via switch 108; multiple end hosts such as host 116 may be coupled to the switch fabric 100 via switch 104; and an IP / Ethernet network 118 may be coupled to the switch fabric 100 via switch 102. In general, a switch may have edge ports and fabric ports. An edge port may be coupled to a device external to the fabric. A fabric port may be coupled to another switch within the fabric via a fabric link. Typically, traffic may be injected into the switch fabric 100 via an ingress port of an edge switch and leave the switch fabric 100 via an egress port of another (or the same) edge switch. An ingress link can couple the NIC of an edge device (e.g., an HPC end host) to an ingress edge port of an edge switch. The switch fabric 100 can then transport the traffic to an egress edge switch, which in turn can deliver the traffic to a destination edge device via another NIC.

[0037] Exemplary NIC Architecture

[0038] Figure 2A An exemplary NIC chip with multiple NICs is shown. Figure 1 In the example shown in FIG. 1 , the NIC chip 200 may be a custom application-specific integrated circuit (ASIC) designed for the host 116 to work with the switch fabric 100. In this example, the chip 200 may provide two independent NICs 202 and 204. The respective NICs of the chip 200 may be equipped with a host interface (HI) (e.g., an interface for connecting to a host processor) and a high-speed network interface (HNI) for connecting to the host processor. Figure 1 For example, the NIC 202 may include the HI 210 and the HNI 220, and the NIC 204 may include the HI 211 and the HNI 221.

[0039] In some embodiments, HI 210 may be a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) interface. HI 210 may be coupled to a host via host connection 201, which may include N (e.g., in some chips, N may be 16) PCIe Gen 4 lanes capable of operating at a signaling rate of up to 25 Gbps per lane. HNI 210 may facilitate a high-speed network connection 203 that may be connected to a host computer. Figure 1 The HNI 210 can communicate over links in the switch fabric 100. The HNI 210 can operate at an aggregate rate of 100 Gbps or 200 Gbps using M (e.g., M can be 4 in some chips) full-duplex serial channels. Each of the M channels can operate at 25 Gbps or 50 Gbps based on non-return-to-zero (NRZ) modulation or pulse amplitude modulation 4 (PAM4), respectively. The HNI 220 can support the Institute of Electrical and Electronics Engineers (IEEE) 802.3 Ethernet-based protocol and an enhanced frame format that supports small messages at higher rates.

[0040] NIC 202 can support one or more of the following: point-to-point messaging based on the Message Passing Interface (MPI), remote memory access (RMA) operations, offloading and scheduling of bulk data collection operations, and Ethernet packet processing. When the host sends an MPI message, NIC 202 can match the corresponding message type. In addition, NIC 202 can implement the urgent protocol and the commitment protocol for MPI, thereby offloading the corresponding operations from the host.

[0041] In addition, the RMA operations supported by NIC 202 may include PUT, GET, and atomic memory operations (AMO). NIC 202 may provide reliable transport. For example, if NIC 202 is a source NIC, NIC 202 may provide a retry mechanism for idempotent operations. In addition, connection-based error detection and retry mechanisms may be used for ordered operations that may manipulate the target state. The hardware of NIC 202 may maintain the state required for the retry mechanism. In this way, NIC 202 may eliminate the burden from the host (e.g., software). The policy for specifying the retry mechanism may be specified by the host through driver software, thereby ensuring flexibility in NIC 202.

[0042] Additionally, the NIC 202 can facilitate triggering of operations, general mechanisms for offloading, and the progression of dependent operation sequences (such as bulk data collection). Figure 1The switch fabric 100 provides an application programming interface (API) (e.g., libfabric API) for fabric communication services to applications running on the host 116. The NIC 202 may also support low-level network programming interfaces, such as the Portals API. In addition, the NIC 202 may provide efficient Ethernet packet processing, which may include efficient transmission when the NIC 202 is a sender, flow steering when the NIC 202 is a target, and checksum calculation. Furthermore, the NIC 202 may support virtualization (e.g., using containers or virtual machines).

[0043] Figure 2B An exemplary architecture of a NIC is shown. In NIC 202, the port macros of HNI 220 can facilitate low-level Ethernet operations such as the physical coding sublayer (PCS) and media access control (MAC). In addition, NIC 202 can support link layer retries (LLR). Incoming packets can be parsed by parser 228 and stored in buffer 229. Buffer 229 can be a PFC buffer provided to buffer a threshold amount (e.g., 1 microsecond) of delay bandwidth. HNI 220 can also include a control send unit 224 and a control receive unit 226 for managing outgoing and incoming packets, respectively.

[0044] NIC 202 may include a command queue (CQ) unit 230. CQ unit 230 may be responsible for acquiring and issuing host-side commands. CQ unit 230 may include a command queue 232 and a scheduler 234. Command queue 232 may include two independent groups of queues for initiator commands (PUT, GET, etc.) and target commands (Append, Search, etc.), respectively. Command queue 232 may be implemented as a circular buffer maintained in the memory of NIC 202. Applications running on the host may write directly to command queue 232. Scheduler 234 may include two separate schedulers for initiator commands and target commands, respectively. Based on a hash function, initiator commands are classified into flow queues 236. One of the flow queues 236 may be assigned to a unique flow. In addition, CQ unit 230 may also include a trigger operation module (or logic block) 238 responsible for queuing and dispatching triggered commands.

[0045] The outbound transport engine (OXE) 240 can pull commands from the stream queue 236 to process them for dispatch. The OXE 240 may include an address translation request unit (ATRU) 244 that can send address translation requests to the address translation unit (ATU) 212. The ATU 212 can provide virtual-to-physical address translation on behalf of different engines, such as the OXE 240, the inbound transport engine (IXE) 250, and the event engine (EE) 216. The ATU 212 can maintain a large translation cache 214. The ATU 212 can perform the translation itself or use a host-based address translation service (ATS). The OXE 240 may also include a message splitting unit (MCU) 246 that can split large messages into packets corresponding to the size of the maximum transmission unit (MTU). The MCU 246 may include multiple MCU modules. When an MCU module becomes available, it can obtain the next command from the assigned stream queue. The received data can be written to the data buffer 242. The MCU module may then send the packet header, corresponding traffic class, and packet size to the traffic shaper 248. The shaper 248 may determine which requests presented by the MCU 246 may proceed to the network.

[0046] The selected packets may then be sent to Packet and Connection Tracking (PCT) 270. PCT 270 may store the packets in a queue 274. PCT 270 may also maintain state information for outbound commands and update the state information when responses are returned. PCT 270 may also maintain packet state information (e.g., to allow responses to be matched with requests), message state information (e.g., to track the progress of multi-packet messages), initiator completion state information, and retry state information (e.g., to maintain information required to retry a command if a request or response is lost). If no response is returned within a threshold time, the corresponding command may be stored in a retry buffer 272. PCT 270 may facilitate connection management for initiator and target commands based on a source table 276 and a target table 278, respectively. For example, PCT 270 may update its source table 276 to track the necessary state for reliable packet delivery and message completion notification. PCT 270 may forward the outbound packets to HNI 220, which stores the packets in an outbound queue 222.

[0047] NIC 202 may also include an IXE 250 that provides packet processing if NIC 202 is a target or destination. IXE 250 may obtain input packets from HNI 220. Parser 256 may parse incoming packets and pass corresponding packet information to list processing engine (LPE) 264 or message status table (MST) 266 for matching. LPE 264 may match incoming messages to buffers. LPE 264 may determine the buffer and start address to be used for each message. LPE 264 may also manage a pool of list entries 262 for representing buffers and unexpected messages. MST 266 may store matching results and information required to generate target-side completion events. MST 266 may be used by unrestricted operations, including multi-packet PUT commands and single-packet and multi-packet GET commands.

[0048] Subsequently, the parser 256 may store the data packet in the data packet buffer 254. The IXE 250 may obtain a matching result for conflict checking. The DMA write and AMO module 252 may then issue updates generated by the write and AMO operations to the memory. If the data packet includes a command to generate a target-side memory read operation (e.g., a GET response), the data packet may be passed to the OXE 240. The NIC 202 may also include an EE 216 that may receive requests to generate event notifications from other modules or units in the NIC 202. The event notification may specify the generation of a fill event or a count event. The EE 216 may manage an event queue located within the host processor memory, to which it writes complete events. The EE 216 may forward the count event to the CQ unit 230.

[0049] Congestion Management in NIC

[0050] Figure 2C 2 shows an exemplary FGFC selection process in a NIC. The NIC 202 can use control frames to control the flow of data packets at a granular level. During operation, upon receiving an FGFC control frame 280, the NIC 202 can determine the type of the frame based on one or more header fields of the frame 280. The frame 280 can be an Ethernet frame having multiple header fields, such as a destination MAC (DMAC) address, a source MAC (SMAC) address, an organizationally unique identifier (OUI) extended Ethernet type, a protocol identifier (PID), an FGFC frame identifier (FID), an FGFC type, a pause period value (e.g., expressed as an Ethernet pause amount), an FGFC credit value, an FGFC identifier, an IPv4 source IP (SIP) address, and an IPv6 SIP address.

[0051] The FGFC identifier may include one or more of the following: a virtual network identifier (VNI), a VLAN ID, an IPv4 flow label, and an IPv6 flow label. The FGFC FID may include a predetermined value associated with the corresponding FGFC frame. The PID may be represented based on the OUI, which may indicate that the link partner is from a supported vendor and may support the same protocol. Instead of assigning a traffic class for flow control, the NIC 202 may identify flows based on the VNI, which may be based on a hash over the source IP address and multiple fields of the packet, such as the protocol type, source and destination IP addresses, source and destination ports, etc. The VNI may be added by the NIC 202 if it is the source NIC and may be removed by the NIC 202 if it is the destination NIC. The VNI may be checked by the ingress and egress switches of the switch fabric.

[0052] NIC 202 can facilitate either Ethernet-based or API-based FGFC. For example, if the link partner of NIC 202 supports the portal API, NIC 202 can provide API-based FGFC to the link partner. On the other hand, if the link partner supports Ethernet-based communication, NIC 202 can provide Ethernet-based FGFC. Upon receiving frame 280, HNI 220 can examine multiple fields of frame 280 (such as the DMAC address, Ethernet type, PID, and FID) to determine whether frame 280 is an FGFC frame. In some embodiments, HNI 220 can maintain a set of control and status registers (CSRs) to store the desired information segments and match the fields with the corresponding CSRs. For example, the DMAC address field should match a CSR that can store the MAC address of NIC 202.

[0053] If HNI 220 determines that frame 280 is an FGFC frame, HNI 220 examines the FGFC type field of frame 280. The FGFC type can identify whether the FGFC frame is based on an API (such as a portal API), Ethernet, IPv4, or IPv6 protocols. HNI 220 can maintain a CSR for each of these types. If the FGFC type of frame 280 does not match any of the types, HNI 220 can issue an error message and discard frame 280. If the FGFC type indicates API-based FGFC, HNI 220 can provide the pause period, FGFC credit value, and the lower portion of the identifier of frame 280 (e.g., the lower 16 bits) to OXE 240 for further processing. On the other hand, if the FGFC type indicates Ethernet, IPv4, or IPv6, HNI 220 can determine that frame 280 is an Ethernet-based FGFC frame. In some embodiments, HNI 220 can then process frame 280 at HNI 220. The NIC 202 may also process the frame 280 at any other element of the NIC 202. For example, Figure 2B The OXE 240 or CQ unit 230 in the MCU can process the FGFC control frame. In addition, the MCU module can generate data packets and stop the corresponding command queue.

[0054] Figure 3A An exemplary FGFC control process in a NIC is shown. Since the HNI 220 is an interface that forms a link with the switch fabric, the NIC 202 receives an FGFC control frame 300 at the HNI 220. If the HNI 220 determines that the frame 300 is an Ethernet-based FGFC frame, the HNI 220 may process the frame 300 using a set of address CSRs 310, an FGFC buffer 320, and an output queue 222. The CSRs 310 may include a set of CSRs (e.g., 4 CSRs) for each of the IPv4 and IPv6 addresses of the NIC 202. The HNI 220 may match the IPv4 or IPv6 source address of the frame 300 with the value stored in the corresponding CSR. As shown in FIG. Figure 2B As described, each of the addresses can be associated with an EQ identifier that identifies a corresponding event queue (EQ). Additionally, CSR 310 can include a programmable CSR for an EQ identifier. If a field of frame 300 does not match a value stored in a corresponding CSR, HNI 220 can discard the frame.

[0055] FGFC cache 320 may have multiple entries, each of which may store information associated with a flow. For example, FGFC cache 320 may include cache entry 322, which may include information associated with the flow, such as a validity field (e.g., a flag), a type field, a tag for the source IP address, an identifier field, an EQ identifier field, and a pause counter. The validity field may indicate whether entry 322 is valid. The type field may indicate the FGFC type used for entry 322. The source IP address tag may indicate the type of source IP address used for entry 322. For example, the tag may contain integer values ​​from 0 to 3, each indicating a type of IP address. A value of 0 may indicate a Layer 2 frame. The identifier field may store a 32-bit identifier from frame 300 associated with the tag. The EQ identifier field may store an EQ identifier obtained from a matching address. In addition, a pause counter may be periodically decremented based on an Ethernet pause standard. The pause counter may be loaded from an FGFC frame and decremented over time based on a pause amount.

[0056] If HNI 220 is able to successfully match the address of frame 300 with the address stored in CSR 310, HNI 220 can determine whether cache 320 is enabled. If cache 320 is disabled, each frame that matches the address in CSR 310 can generate an event (e.g., Figure 2B The event may be an internal control message used for communication between elements of the NIC 202. On the other hand, if the cache 320 is enabled, the type, tag, and identifier for the source IP address in the frame 300 are checked against the information in the corresponding entry in the cache 320.

[0057] If the field of frame 300 matches a valid entry, and frame 300 has a timeout period of zero, HNI 220 may invalidate this entry in cache 320 (e.g., by modifying the valid field). HNI 220 may then forward the event (e.g., to Figure 2B EE 216 in the frame 300). This event may indicate an XON for the EQ identifier of the entry and include the credit value specified in the frame 300. On the other hand, if the fields of the frame 300 match a valid entry and the frame 300 has a non-zero pause period value, the HNI 220 may update the pause counter based on the pause period value in the frame 300. The HNI 220 may then forward an XOFF event, which may include the non-zero credit value specified in the frame 300. However, if the credit value is zero, the HNI 220 may update the cache 320 without forwarding the event.

[0058] If the fields of frame 300 do not match a valid entry, HNI 220 may determine whether frame 300 includes a non-zero pause period value and whether cache 320 has availability for a new entry (e.g., whether a cache line is available). If cache 320 has availability and frame 300 includes a non-zero pause period value, HNI 220 may create an entry in cache 320 with the pause counter set to the pause period value in frame 320. HNI 220 may also forward an XOFF event, which may include a credit value specified in frame 300. On the other hand, if cache 320 does not have availability and frame 300 includes a non-zero pause period value, HNI 220 may discard frame 300 without creating an event. If frame 300 includes a zero pause period value, HNI 220 may forward an XON event, which may include a credit value specified in frame 300.

[0059] If the entry in the cache 320 has a pause counter value lower than the pause amount, then the HNI 220 can set a flag indicating that the HNI 220 should create an XON event for the entry. The HNI 220 can apply a round-robin arbitration process to select the entry. Subsequently, the HNI 220 can invalidate the entry and forward the event. The event can indicate the XON state of the EQ identifier for the entry. However, if a subsequent FGFC frame arrives before selecting the entry through arbitration, the HNI 220 can update the pause counter in the entry and remove the request for arbitration of the entry. The EQ identifier from the entry can be used to locate the target event queue. In some embodiments, when there is no Ethernet-based FGFC frame that matches the entry and there is availability in the queue 222, the HNI 220 can perform arbitration based on the clock of the NIC 202.

[0060] If EE 216 is backed up, queue 222 allows HNI 220 to process a smaller number of FGFC frames. Events forwarded from the previous state can be inserted into queue 222. If queue 222 is full, the generated events can be discarded. Corresponding entries in queue 222 (such as entry 324) can include a return code, a type field, a label for the source IP address, an identifier field, credit information, an XOFF indicator, an EQ identifier, and an event type. The return code can be set to a constant indicating a valid return. The type field can indicate whether frame 300 corresponds to Ethernet, IPv4, or IPv6. The label for the source IP address can indicate the type of IP address of the source address of frame 300. The corresponding values ​​for the identifier and credit fields can be obtained from the corresponding fields in frame 300. The XOFF indicator can indicate whether an XOFF event should be generated. The EQ identifier field can store the EQ identifier obtained from the matching address. In addition, the event type field can be set to Ethernet. If a cache timeout occurs for an XON event, the corresponding values ​​for the type, label, identifier, and EQ identifier fields can be obtained from cache 320. Additionally, for a cache timeout event, the value of the credit field may be set to zero.

[0061] On the other hand, if the FGFC type indicates API-based FGFC, the HNI 220 may provide information 350 associated with the frame 300 to the OXE 240 for further processing. The information 350 may include a pause period value, an FGFC credit value, and the lower portion (e.g., the lower 16 bits) of the identifier of the frame 300. The OXE 240 may then store the information 350 in the FGFC table 330. The NIC 202 may use the table 330 to throttle packets belonging to flows subject to FGFC. The table 330 may include multiple entries. A corresponding entry of the table 330 (such as entry 332) may include a VNI field, a valid field (e.g., a flag), a credit field, and a pause counter. These fields may include 16 bits, 1 bit, 24 bits, and 32 bits, respectively.

[0062] OXE 240 can match the VNI field with the incoming FGFC data packet and determine the MCU module that is allowed to send more data packets from MCU 246. The valid field can indicate whether the VNI is valid. The credit field can store the sum of the credit values ​​received in the FGFC frame (such as frame 300). In some embodiments, each credit allows the MCU module to forward one byte. If the value of the credit field becomes negative, table 330 may lack credits to send a data packet. The credit field can be associated with a maximum value (i.e., the maximum value to which the credit can be incremented). The pause counter can correspond to an Ethernet pause. The upper 16 bits can be loaded from frame 300. The lower 16 bits can represent a fraction that can be decremented over time based on the pause amount.

[0063] Upon classifying frame 300 as an API-based FGFC frame, if table 330 is enabled, HNI 220 may pass frame 300 to OXE 240 for processing. If frame 300 matches a valid entry for the VNI in frame 300 and frame 300 has a pause period value of zero, OXE 240 may mark the entry as invalid. Otherwise, if frame 300 matches a valid entry for the VNI in frame 300 and frame 300 has a non-zero pause period value, OXE 240 may increment the credit value in the entry based on the credit indicated in frame 300 and update the pause counter based on the pause value of frame 300. If frame 300 does not match a valid entry and table 330 has availability (e.g., a row in table 300 is available), OXE 240 may create an entry in table 330 by inserting the VNI, credit value, and pause value from frame 300 into the entry. The initial credit may be reduced by a credit adjustment constant. In some embodiments, the default value of this constant may be determined as (MTU+maximum header size+FCS). Here, FCS indicates a frame check sequence. If the frame 300 does not match a valid entry and the table 330 is not available, the OXE 240 may drop the frame 300.

[0064] Figure 3B FIG2 illustrates an exemplary data packet forwarding process associated with FGFC in a NIC. During operation, a host device of NIC 202 may send a data packet 360 belonging to a flow subject to FGFC via host interface 210. In NIC 202, data packet 360 may be forwarded to OXE 240. When data packet 360 arrives at OXE 240, it may be assigned to one of multiple MCU modules 302, 304, and 306 in MCU 246. OXE 240 may use arbitration module 340 to select an MCU module to forward the data packet. In other words, arbitration provided by arbitration module 340 (e.g., based on a round-robin technique or priority values) may schedule data packet forwarding from the MCU modules. Assume that data packet 360 has been assigned to MCU module 306. If arbitration module 340 selects MCU module 306, MCU module 306 may check whether the VNI in data packet 360 matches an entry in table 330.

[0065] If no entry matches the packet 360, the OXE 240 may allow the packet 360 to proceed and may place it in the output buffer 242. If an entry exists and the credit in the entry is not negative, the OXE 240 may allow the packet 360 to proceed and deduct an amount of credit from the credit field of the matching entry. The amount of credit may be determined as:

[0066] [(byte_len+extra_bytes+2 round_pos -1)&~(2 round_pos -1)]

[0067] However, if the entry exists and the credit is negative, the OXE 240 may set the FGFC flag for the MCU module 306 and drop the packet 360 (eg, by deselecting the MCU module 306 during the arbitration process).

[0068] Because the FGFC flag of the MCU module 306 is set, the arbitration module 340 can remove the MCU module 306 from the arbitration. The OXE 240 can save the index of the corresponding entry (i.e., the entry of the matching data packet 360) of the table 330. The OXE 240 can then monitor the entry based on the index. If the entry becomes invalid or the credit value in the entry increases to a non-negative value, the OXE 240 can clear the FGFC flag of the MCU module 306. When the FGFC flag is cleared, the arbitration module 340 can include the MCU module 306 in the arbitration process. In addition, when FGFC is applied to an MCU module, in addition to selecting an MCU module based on credit during the arbitration process, the MCU module can also be in "ordered" mode. Therefore, the MCU module can forward packets based on their order until the MCU module undergoes FGFC.

[0069] Figure 4A A flow chart of an Ethernet-based FGFC process in a NIC is shown. During operation, the HNI of the NIC may obtain an Ethernet-based FGFC frame (operation 402). The HNI may then check whether the frame matches an entry in the FGFC cache and has a zero pause value (operation 404). If the frame matches an entry in the FGFC cache and has a zero pause value, the HNI may mark the entry as invalid and forward an XON event with credits from the frame (operation 414). Otherwise, the HNI may check whether the frame matches an entry in the FGFC cache and has a non-zero pause value (operation 406). If the frame matches an entry in the FGFC cache and has a non-zero pause value, the HNI may process the frame based on the credit value. The HNI may update the pause counter in the entry based on the non-zero pause value from the frame, and if the frame has a non-zero credit value in the frame, forward an XOFF event (indicated by brackets) with non-zero credits from the frame (operation 416). On the other hand, if the frame has a zero credit value, the HNI may update the pause counter in the entry based on the non-zero pause value from the frame without forwarding the XOFF event (operation 416).

[0070] If the frame does not match an entry in the FGFC cache (operations 404 and 406), the HNI may check whether the cache is available (operation 408). If the cache is available, the HNI may create an entry with a pause counter based on the non-zero pause value from the frame and forward an XOFF event with credits from the frame (operation 418). If the cache is not available, the HNI may check whether the frame has a non-zero pause value (operation 410). If the frame has a non-zero pause value, the HNI may forward an XON event with credits from the frame (operation 420). On the other hand, if the frame does not have a non-zero pause value (i.e., has a zero pause value), the HNI may defer the frame (operation 412) (e.g., may wait for more credits to arrive).

[0071] Figure 4B A flow chart of an API-based FGFC process in a NIC is shown. During operation, the OXE of the NIC may obtain an API-based FGFC frame (operation 432). The OXE may then check whether the frame matches an entry in the FGFC table and has a zero pause value (operation 434). If the frame matches an entry in the FGFC table and has a zero pause value, the OXE may mark the entry as invalid (operation 442). Otherwise, the OXE may check whether the frame matches an entry in the FGFC table and has a non-zero pause value (operation 436). If the frame matches an entry in the FGFC table and has a non-zero pause value, the OXE may update the pause counter in the entry based on the non-zero pause value from the frame and increment the credit value in the entry with the non-zero credit from the frame (operation 444).

[0072] If the frame does not match an entry in the FGFC table (operations 434 and 436), the OXE may check the table for availability (operation 438). If the table is available, the OXE may create an entry in the FGFC table with a pause counter and a credit value, and subtract a default credit value (operation 446). The pause counter may be based on a non-zero pause value, and the credit value may be based on credits from the frame. If the cache is not available, the OXE may discard the frame (operation 440).

[0073] Figure 4C A flowchart is shown for facilitating exemplary packet processing for FGFC in a NIC. During operation, the NIC's OXE may assign a packet associated with FGFC to a corresponding MCU module (operation 452) and select an MCU module based on arbitration (454). The OXE may then check whether the packet matches an entry in the FGFC table (operation 456). If the packet matches an entry in the FGFC table, the OXE may allow the packet to proceed (operation 464).

[0074] Otherwise, the OXE may check whether the credit in the entry is negative (operation 458). If the credit in the entry is not negative, the OXE may allow the packet to proceed and deduct a certain amount of credit from the credit of the entry (operation 466). On the other hand, if the credit in the entry is negative, the OXE may discard the packet and set the FGFC flag for the MCU module (operation 460). The OXE may then remove the MCU module from arbitration and monitor for matching entries (operation 462).

[0075] Exemplary Computer System

[0076] Figure 5 An exemplary computer system equipped with a NIC with FGFC support is shown. Computer system 550 includes a processor 552, a memory device 554, and a storage device 556. Memory device 554 may include a volatile memory device (e.g., a dual in-line memory module (DIMM)). Additionally, computer system 550 may be coupled to a keyboard 562, a pointing device 564, and a display device 566. Storage device 556 may store an operating system 570. Applications 572 may operate on operating system 570.

[0077] Computer system 550 may be equipped with a host interface coupled to NIC 520, which facilitates efficient data request management. NIC 520 may provide computer system 550 with one or more Host-Interfaces (HNIs), such as HNI 540. NIC 520 may be coupled to switch 502 via HNI 540. Upon receiving an FGFC control frame from switch 502, HNI 540 may determine whether the frame is Ethernet-based or API-based. If the frame is Ethernet-based, HNI 540 may compare the source IP address with the local address stored in register 532. Upon detecting a match, HNI 540 may process the frame based on the entry in FGLC cache 534 and the frame's contents. HNI 540 may also include a queue 536, which may store events that cannot be accommodated in the event engine of NIC 520. If the frame is an API-based frame, the HNI 540 may provide header information to the OXE logic block 530 of the NIC 520 and transmit the frame to the OXE logic block 530 .

[0078] The OXE logic block 530 may store the information in an entry in the FGLC table 536. The OXE logic block 530 may then process the frame based on the entry in the FGLC table 536 and the contents of the frame. Upon receiving a packet belonging to a flow subject to FGLC from the computer system 550 via the HI of the NIC 520, the OXE logic block 530 may assign the packet to the MCU logic block 532. The arbitration logic block 534 may select the MCU logic block 532 based on the arbitration policy. If the MCU logic block 532 is selected, the OXE logic block 530 may process the packet based on the matching entry in the FGLC table 536 and the contents of the packet.

[0079] In summary, the present disclosure describes a NIC that facilitates fine-grained flow control (FGFC). The NIC can be equipped with a network interface, an FGFC logic block, and a traffic management logic block. During operation, the network interface can determine that a control frame from a remote switch is for applying FGFC. The network interface can then identify a data flow indicated in the control frame for applying FGFC. The FGFC logic block can insert information from the control frame into an entry in a data structure stored in the NIC. The traffic management logic block can identify an entry in the data structure based on one or more fields of a packet belonging to the flow. The traffic management logic block can then determine whether to allow forwarding of the packet based on the information in the entry.

[0080] The methods and processes described above can be performed by hardware logic blocks, modules, logic blocks, or devices. Hardware logic blocks, modules, or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute code blocks at specific times, and other programmable logic devices now known or later developed. When the hardware logic blocks, modules, or devices are activated, they execute the methods and processes contained therein.

[0081] The methods and processes described herein may also be implemented as code or data, which may be stored in a storage device or computer-readable storage medium. When a processor reads and executes the stored code or data, the processor may perform the methods and processes.

[0082] The foregoing descriptions of the embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the invention to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. Additionally, the above disclosure is not intended to limit the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A network interface controller (NIC), comprising: A host interface, the host interface being used to connect to a host; a network interface configured to communicate with a link in a switch fabric, wherein the switch fabric includes a plurality of switches, and wherein the network interface is further configured to: determining that a control frame from a switch in the switch fabric is associated with fine grained flow control (FGFC); and identifying a data flow in the switch fabric indicated in the control frame for applying the FGFC; An FGFC logic block is configured to insert information from the control frame into an entry of a data structure stored in the network interface controller, wherein inserting the information into the entry comprises: determining a duration value for applying the FGFC to the data stream based on the information in the control frame, and updating a duration counter in the entry based on the duration value; and Traffic management logic blocks for: identifying the entry in the data structure based on one or more fields of a packet belonging to the data flow; and determining whether to allow forwarding of the data packet to the switch in the switch fabric based on the information in the entry, wherein the entry includes a credit value indicating an amount of data of the data flow that can be forwarded, and if the control frame matches the entry and has a non-zero pause value, the traffic management logic block increments the credit value in the entry with the non-zero credit from the control frame. 2 . The network interface controller of claim 1 , wherein the network interface is further configured to determine whether to process the control frame at the network interface based on a type of the control frame.

3. The network interface controller of claim 2, wherein the network interface is further configured to provide information from one or more fields of the control frame to the traffic management logic block based on a type of the control frame.

4. The network interface controller of claim 1 , wherein the network interface is further configured to generate an event for the data flow based on a duration value and a credit value from the information in the control frame, wherein the event is a control message indicating whether to initiate or terminate the FGFC for the data flow.

5. The network interface controller of claim 1 , wherein inserting the information into the entry comprises: determining credit information from the information in the control frame, wherein the credit information indicates an amount of data of the data flow that can be forwarded; as well as The credit value in the entry is updated based on the determined credit information.

6. The network interface controller of claim 5, wherein the traffic management logic block is configured to: Distributing the data packet to an MCU among a plurality of message segmentation units MCU; Arbitration is performed among the plurality of MCUs to select an MCU for forwarding the data packet based on the credit value in the entry.

7. The network interface controller of claim 1 , wherein inserting the information into the entry comprises: determining whether one or more fields match an existing entry in the data structure; In response to not finding a match, determining a new entry in the data structure; as well as Information from the one or more fields is inserted into the new entry.

8. The network interface controller according to claim 7, wherein the FGFC logic block is further configured to: determining whether the data structure has availability for the new entry; and In response to the data structure not being available, the control frame is discarded.

9. The network interface controller of claim 1 , wherein the entries include one or more of the following: an identifier of the data stream, wherein the entries are indexed based on the identifier; a validity flag indicating whether the entry is valid; a duration counter indicating a duration value for applying FGFC to the data flow; and The event queue identifier.

10. A method for facilitating fine-grained flow control, comprising: Determining, by a network interface controller NIC according to any one of claims 1 to 9, that a control frame from a switch in the switch fabric is associated with a fine-grained flow control FGFC; identifying a data flow in the switch fabric indicated in the control frame for applying the FGFC; Inserting information from the control frame into an entry of a data structure stored in the NIC, wherein inserting the information into the entry comprises: determining a duration value for applying the FGFC to the data stream based on the information in the control frame, and updating a duration counter in the entry based on the duration value; identifying the entry in the data structure based on one or more fields of a packet belonging to the data flow; and determining whether to allow forwarding the data packet to the switch in the switch fabric based on information in the entry, wherein the entry includes a credit value indicating an amount of data of the data flow that can be forwarded, and if the control frame matches the entry and has a non-zero pause value, the traffic management logic block increments the credit value in the entry with the non-zero credit from the control frame.

11. The method of claim 10, further comprising determining a type of the control frame, wherein the type indicates how to process the control frame.

12. The method of claim 11, further comprising forwarding information from one or more fields of the control frame within the NIC based on a type of the control frame.

13. The method of claim 10, further comprising generating an event for the data flow based on a duration value and a credit value from the information in the control frame, wherein the event is a control message indicating whether to initiate or terminate the FGFC for the data flow.

14. The method of claim 10, wherein inserting the information into the entry comprises: determining credit information from the information in the control frame, wherein the credit information indicates an amount of data of the data flow that can be forwarded; as well as The credit value in the entry is updated based on the determined credit information.

15. The method according to claim 10, further comprising: Distributing the data packet to an MCU among a plurality of message segmentation units MCU; Arbitration is performed among the plurality of MCUs to select an MCU for forwarding the data packet based on the credit value in the entry.

16. The method of claim 10, wherein inserting the information into the entry comprises: determining whether one or more fields match an existing entry in the data structure; In response to not finding a match, determining a new entry in the data structure; as well as Information from the one or more fields is inserted into the new entry.

17. The method according to claim 16, further comprising: determining whether the data structure has availability for the new entry; as well as In response to the data structure not being available, the control frame is discarded.

18. The method of claim 10, wherein the entries include one or more of the following: an identifier of the data stream, wherein the entries are indexed based on the identifier; a validity flag indicating whether the entry is valid; a duration counter indicating a duration value for applying FGFC to the data flow; a credit value indicating an amount of data of the data flow that can be forwarded; and The event queue identifier.

Images