Programmable congestion control communication scheme

Abstract

The network adapter includes a receive (Rx) pipe, a transmit (Tx) pipe, and a congestion management circuit. The Rx pipe is configured to receive packets sent by the peer network adapter over the network and process the received packets. The Tx pipe is configured to send packets to the peer network adapter over the network. The congestion management circuit is configured to receive congestion control (CC) events derived from at least some of the packets exchanged with the peer network adapter from the Tx pipe and from the Rx pipe, to exchange user-programmable congestion control packets with the peer network adapter, and to alleviate congestion affecting one or more packets in response to CC events and user-programmable congestion control packets.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims a continuation-in-part of U.S. patent application No. 16 / 683,302 filed on November 14, 2019, the disclosure of which is incorporated herein by reference. Technical Field

[0003] The present invention relates generally to packet communication networks and, in particular, to methods and systems for congestion control in such networks. Background Art

[0004] Congestion management of packet traffic in communication systems is important because poor congestion control may significantly affect system performance.

[0005] There are some congestion control techniques used in the industry, such as "Explicit rate congestion control with binary notifications" by Almeida and Belo, 10th IEEE Symposium on Local and Metropolitan Area Networks, November 21-24, 1999, describes a rate-based source adaptation algorithm for packet-switched networks, where binary notifications are sent to the source reflecting the positive or negative difference between the source rate and the estimated fair rate, and based on these notifications, the source increases or decreases the sending rate.

[0006] Internet Engineering Task Force (IETF) RFC 3168, entitled "The Addition of Explicit Congestion Notification (ECN) to IP", September 2001, specifies the incorporation of Explicit Congestion Notification (ECN) into TCP and IP, including the use of two bits of ECN in the IP header. Summary of the invention

[0007] Embodiments of the invention described herein provide a network adapter, including a receive (Rx) pipe, a transmit (Tx) pipe, and a congestion management circuit. The Rx pipe is configured to receive packets sent by a peer network adapter over a network and process the received packets. The Tx pipe is configured to send packets to the peer network adapter over the network. The congestion management circuit is configured to receive congestion control (CC) events from the Tx pipe and from the Rx pipe originating from at least some packets exchanged with the peer network adapter, exchange user-programmable congestion control packets with the peer network adapter, and mitigate congestion affecting one or more packets in response to the CC events and the user-programmable congestion control packets.

[0008] In some embodiments, the congestion affects packets sent to the peer network adapter, and wherein the congestion management circuit is configured to throttle the transmission rate of packets sent by the Tx pipe according to the CC event and the user-programmable congestion control packet. In other embodiments, the congestion affects packets received from the peer network adapter, and wherein the congestion management circuit is configured to generate a congestion notification based on the CC event and the user-programmable congestion control packet, and send the congestion notification to the peer network adapter.

[0009] In one embodiment, the congestion management circuit supports an application programming interface (API) for receiving a user-defined congestion control (CC) software code and alleviating the congestion by running the CC software code. In another embodiment, the programmable congestion control packet has a format of a management datagram (MAD) packet.

[0010] In some embodiments, the congestion management circuit includes a hardware congestion control circuit configured to pre-process the CC event, and a user-programmable congestion control processor configured to alleviate the congestion in response to the pre-processed CC event. The hardware congestion control circuit is configured to merge two or more CC events and provide the merged CC event to the congestion control processor.

[0011] According to one embodiment of the present invention, a communication method is also provided. The method includes using a receiving (Rx) pipe of a network adapter to receive a packet sent by a peer network adapter through a network, and processing the received packet. Using a sending (Tx) pipe of the network adapter, a packet is sent to the peer network adapter through the network. Using a congestion management circuit in the network adapter, a congestion control (CC) event originating from at least some packets exchanged with the peer network adapter is received from the Tx pipe and the Rx pipe. User-programmable congestion control packets are exchanged with the peer network adapter. In response to CC events and user-programmable congestion control packets, congestion affecting one or more packets is alleviated.

[0012] According to an embodiment of the present invention, a network adapter is also provided, comprising a receive (Rx) pipe, a transmit (Tx) pipe and a management circuit. The Rx pipe is configured to receive packets sent by a peer network adapter through a network and process the received packets. The Tx pipe is configured to send packets to the peer network adapter through the network. The management circuit is configured to receive events from at least some packets exchanged with the peer network adapter from the Tx pipe and the Rx pipe, exchange user-programmable control packets with the peer network adapter, and perform control operations in response to the events and the user-programmable control packets.

[0013] According to one embodiment of the present invention, a communication method is also provided, including using a receive (Rx) pipe of a network adapter to receive packets sent by a peer network adapter through a network, and processing the received packets. Using a transmit (Tx) pipe of the network adapter, packets are sent to the peer network adapter through the network. Using management circuitry in the network adapter, events originating from at least some packets exchanged with the peer network adapter are received from the Tx pipe and the Rx pipe. User-programmable control packets are exchanged with the peer network adapter. Control operations are performed in response to events and user-programmable control packets.

[0014] The present invention will be more fully understood through the following detailed description of its embodiments in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram schematically illustrating a system using Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) with congestion relief according to an embodiment of the present invention;

[0016] Figure 2 is a block diagram schematically showing the structure of a network adapter according to an embodiment of the present invention;

[0017] Figure 3 is a table schematically showing the structure of a congestion control management datagram (CC-MAD) according to an embodiment of the present invention;

[0018] Figure 4A is a flow chart schematically illustrating a congestion control process in a network adapter configured as a reaction point (RP) according to an embodiment of the present invention; and

[0019] Figure 4B is a flowchart schematically showing a congestion control process in a network adapter configured as a notification point (NP) according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] Overview

[0021] The Transmission Control Protocol (TCP) was introduced in 1974, but its congestion control (CC) algorithms have continued to evolve. The CC algorithm changes the rate at which packets are sent from a source (e.g., a network adapter in a source node) in response to congestion notifications received by the source. Congestion notifications are typically added to the packet stream (either as a separate packet, or as additional information in an existing packet). When packets reach their destination (e.g., a network adapter in a destination node), the destination sends congestion notifications back to the source, which may reduce the rate at which the responsive notifications are sent (or increase the rate if no congestion notifications were received).

[0022] Many CC algorithms have been (and still are) proposed that are concerned with sending rate setting in response to congestion notifications (see, e.g., U.S. Pat. No. 7,000,025). CC algorithms may be, for example, flow-oriented and may independently control the rate of each packet flow in response to congestion notifications for packets on the same flow.

[0023] Congestion control (CC) algorithms can be executed by software running on the host processor or by dedicated hardware. Implementing software on the host has some major disadvantages. First, host software implementations usually exhibit long delays caused by the context switches required to process congestion notifications; second, host software implementations consume a lot of CPU resources; and finally, algorithms executed by host software require communication between the network adapter and the host over the bus. On the other hand, software implementations are usually flexible, allowing easy adaptation to new CC algorithms and changes in network configurations, while hardware implementations tend to be rigid and difficult to modify.

[0024] An efficient congestion control algorithm may need to measure the round trip delay of the network, which is from a sending network adapter to a receiving network adapter and then back. This measurement is typically done using round trip time (RTT) measurement packets that are initiated by a sending network adapter and routed back by a receiving network adapter, where each network adapter adds a timestamp related to the time at which the network adapter received and / or sent the RTT packet.

[0025] Some congestion control algorithms may require the use of additional control / notification packets to transfer additional data between the sending and receiving network adapters.

[0026] Embodiments according to the present invention disclosed herein provide flexible and fast network adapters with programmable congestion management. Congestion management is available for sending and receiving network adapters that communicate using congestion notifications, proprietary programmable congestion control packets, and RTT measurement packets.

[0027] The following description refers to a network adapter. In some embodiments, the network adapter may include an Ethernet network interface controller (NIC); in other embodiments, the network adapter may include a wireless broadband host channel adapter (HCA); in other embodiments, any other suitable network adapter may be used.

[0028] Hereinafter we refer to network adapters that receive data (and receive congestion notifications) as Notification Points (NPs), and network adapters that send data (and may need to react to congestion notifications) as Reaction Points (RPs). It is understood that both NPs and RPs can send and receive packets, and that the terms NP and RP refer only to specific congestion-controlled packet flows.

[0029] In each of the NP and RP, the network adapter generally includes one or more ingress ports that receive packets from the network and send the packets to further input processing (e.g., parsing, queuing, and steering); and one or more egress ports that receive processed (e.g., steering and queuing) packets and send the packets to the network. In the following description, we refer to the aggregation of ingress ports and further input processing as a receive pipeline, and the aggregation of egress packet processing and egress ports as a send pipeline.

[0030] In some embodiments, the network adapter also includes congestion management circuitry to which the receive and transmit pipelines send congestion-related events. Congestion-related events may include Tx events for each predefined burst of transmitted packets, and Rx events for the following received packets: positive and negative acknowledgement (ACK and NACK) packets; congestion notification packets (packets sent back by the peer network adapter in response to transmitted packets), RTT measurement packets, and user-defined programmable congestion control packets.

[0031] In an embodiment, the congestion management circuit executes a congestion control algorithm and can issue a response to the peer network adapter. In the RP, the congestion management circuit is further configured to throttle the sending rate of the network adapter, reduce the rate to alleviate congestion, and increase the sending rate to improve performance when congestion is light (or non-existent).

[0032] Some congestion control algorithms may require the transmission of information between the NP and the RP. One example is the RTT packet; however, other requirements may arise to allow for a variety of congestion control algorithms (e.g., sending credits in a credit-based congestion control algorithm). Embodiments of the present invention allow for the sending and receiving of generic user-programmable congestion control messages between the NP and the RP. In an embodiment, the user-programmable congestion control packets have a message datagram (MAD) format as defined by InfiniBand (the congestion control MAD is referred to as CC-MAD). In other embodiments, congestion control packets of different formats may be used instead of or in addition to the MAD.

[0033] In an embodiment, the congestion management circuit includes a congestion control circuit (CCC) and a congestion control reduced instruction set computer (CC-RISC), wherein the CC-RISC is configured to run a programmable congestion control algorithm, and the CCC is configured to receive congestion control inputs, including congestion events and CC packets, pre-process (e.g., queue and merge) the congestion control inputs, and send the inputs to the CC-RISC. In an embodiment, the receive and transmit pipelines may send some or all congestion control inputs directly to the CC-RISC. In another embodiment, some congestion control algorithms may be implemented by circuits other than the CC-RISC, such as the CCC or the receive / transmit pipelines.

[0034] In some embodiments, CC-RISC may include a commercially available RISC core, such as RISC-V, ARM, or any other suitable processor. In other embodiments, multiple processors of the same or different types may be used.

[0035] In some of the example embodiments below, we will refer to Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE); however, the techniques disclosed herein are not limited to RoCE and may be used in any suitable network configuration, such as non-volatile memory on fabric (NVM-F).

[0036] Other aspects of programmable congestion control are addressed in U.S. patent application Ser. No. 16 / 683,302, filed on Nov. 14, 2019, referenced above.

[0037] System Description

[0038] Figure 1 1 is a block diagram schematically illustrating a RoCE communication system 100 with congestion relief according to an embodiment of the present invention. A transmitting network adapter 102 sends a packet to a receiving network adapter 106 via a network 104. Both the transmitting network adapter and the receiving network adapter are configured to send and receive packets; the terms "send" and "receive" hereinabove refer to the direction of congestion relief. Figure 1 In the example implementation shown, the transmitting network adapter 102 and the receiving network adapter 106 may be identical (albeit configured differently).

[0039] Each of the transmitting network adapter 102 and the receiving network adapter 106 includes a transmit (TX) pipe 110 that queues and arbitrates packets sent by the network adapter, a receive (RX) pipe 112 that receives incoming packets from the network, and a congestion management unit 114 .

[0040] The transmit pipeline 110 of the transmitting network adapter 102 queues and arbitrates egress packets and transmits the packets over the network 104. The egress packets may originate, for example, from a CPU (not shown) coupled to the network adapter, or from congestion management 114.

[0041] according to Figure 1 In the illustrated example embodiment, the network includes a switch 116 that can send packets sent by a network adapter with an explicit congestion indication (ECN) mark when congested.

[0042] The receiving network adapter 106 sends return packets back to the sending network adapter, including packets for congestion control, such as CNP packets, ACK / NACK packets, RTT measurement packets, and programmable CC packets (described below). When the receiving network adapter receives a packet with an ECN indication, the receiving network adapter sends the CNP packet back to the sending network adapter.

[0043] according to Figure 1 In the illustrated example implementation, a network adapter can generate programmable CC packets to facilitate various programmable congestion control algorithms (e.g., credit-based). Figure 1 In the example embodiment shown, such a programmable CC packet is an Infiniband Management Datagram (MAD), and will be referred to hereinafter as CC-MAD. In other embodiments, other CC packets may be used instead of or in addition to CC-MAD packets.

[0044] Congestion management 114 is configured to execute congestion control algorithms, initiate transmission of congestion control packets and alleviate congestion in the RoCE transmission path.

[0045] Congestion management 114 receives Tx events when transmit pipeline 110 transmits bursts of packets, and receives Rx events when receive pipeline 112 receives congestion notification packets. Received congestion notification packets may include, for example, ACKs and NACKs received in response to transmitted packets, CNP packets generated by a receiving network adapter in response to receiving ECN marked packets, RTT measurement packets, and congestion control packets.

[0046] Congestion management 114 is configured to:

[0047] i) receiving an event related to the transmission of an egress packet (or a burst of egress packets);

[0048] ii) receiving an event related to receiving a congestion control packet;

[0049] iii) controlling the sending channel to send RTT packets;

[0050] iv) receiving the returned RTT packet from the receiving pipe;

[0051] v) Construct, send (through the send pipe), and receive (through the receive pipe) congestion control packets;

[0052] vi) running a congestion control algorithm; and,

[0053] vii) In the receiving network adapter - adjust the sending rate as dictated by the congestion control algorithm (in some embodiments, congestion management runs the congestion control algorithm separately for each packet flow and determines the sending rate separately for each flow).

[0054] Therefore, according to Figure 1 In the example embodiment shown, the congestion control circuit handles congestion events and runs a congestion control algorithm. In order to alleviate congestion in the RoCE network protocol (or other suitable protocols), the network adapter 102 includes a CC management circuit that monitors egress traffic, initiates and receives congestion control packets and notifications, sends and receives congestion control packets, and runs a suitable congestion control algorithm; if the network adapter is a sending endpoint, the congestion control management will also throttle the sending rate.

[0055] As will be appreciated, the configuration of the RoCE architecture 100 is an example configuration depicted purely for conceptual clarity. Other suitable configurations may be used in alternative embodiments of the present invention. For example, instead of (or in addition to) RoCE, the architecture may be TCP and / or fused non-volatile memory (NVM) storage (e.g., hyper-converged NVM-f).

[0056] Figure 2 is a schematic diagram showing a network adapter 102 ( Figure 1 ) is a block diagram of the structure of . The network adapter 102 can be a sending network adapter or a receiving network adapter. In the context of RoCE, the sending network adapter is a reaction point (RP) and the receiving network adapter is a notification point (NP).

[0057] The network adapter 102 includes a send pipeline 110, a receive pipeline 112, and a congestion management 114 (see above). Figure 1All of these are briefly described). The transmit pipeline 110 includes a transmit WQE (SXW) circuit 202, which is configured to exchange work queue elements (WQEs) and completion queue elements (CQEs) with a CPU (not shown) coupled to a network adapter; and generate egress packets. The transmit pipeline also includes a send packet (SXP) circuit 204, which is configured to queue, arbitrate, and send packets over the network.

[0058] The receive pipeline 112 includes a parsing and steering (PSA) circuit 206 configured to parse and steer ingress packets, and a receive circuit (RXT) 208 configured to process ingress packets and forward the ingress packets to a CPU coupled to a network adapter (typically, but not necessarily, the same CPU is coupled to the transmit pipeline and the receive pipeline).

[0059] Congestion management 114 includes congestion control circuitry 210 and a congestion control reduced instruction set computer (CC-RISC) 212, which includes one or more CPUs (e.g., RISC-V or ARM) and is configured to execute congestion control algorithms using dedicated low-overhead software (hereinafter referred to as firmware or FW).

[0060] Note that when the NP network adapter routes the RTT message back (after adding a timestamp), the NP bypasses congestion management 114 and routes the RTT packet directly from the receive pipe to the send pipe; each pipe adds a timestamp to the packet.

[0061] In some embodiments, CCC 210 is configured to receive congestion control events and packets from receive and transmit pipes, to consolidate, queue and send these events to CC-RISC, to receive responses from CC-RISC and to send CC-RISC messages to the transmit pipe.

[0062] The division between hardware and firmware congestion control tasks described above is cited as an example. In alternative embodiments, any suitable division of congestion control tasks between hardware (e.g., CCC 210) and firmware (e.g., code running on CC-RISC 212) may be used. In embodiments, some congestion control tasks may be performed by an external CPU.

[0063] SXW 202 sends Tx event notifications to CCC 210. Such events may include an indication of the transmission of a burst of packets. In some embodiments, SXW sends notifications corresponding to the transmission of bursts of packets of the same flow; in other embodiments, only notifications related to a predefined subset (e.g., packets leaving through a predefined port) are sent. An address vector (AV) may be attached to the Tx event notification to identify the packet flow. In an alternative embodiment, SXW 202 may send some or all Tx events directly to CC-RISC 212.

[0064] RXT 208 sends CC Rx event notifications and ingress CC packets to CCC 210. In some embodiments, RXT, for example, sends only some notifications based on the ingress port where the notification was received. CC Rx events include received ACK packets, NACK packets, RTT measurement packets, and (if the network adapter is a receiving network adapter) ECN notifications. In some embodiments, RXT sends ACK and NACK indications instead of full ACK / NACK packets; in other embodiments, RXT, for example, sends only some notifications to CCC based on the ingress port where the notification was received. In other embodiments, RXT 208 sends Rx events and / or CC packets directly to CC-RISC 212.

[0065] The CCC 210 is configured to receive Tx events and corresponding AV indications from the transmit pipe; and, Rx events and CC packets from the receive pipe. The CCC may merge some or all events and send the merged set of events and any CC packets to the CC-RISC.

[0066] CC-RISC 212 receives CC events and CC packets from CCC 210 or directly from SXW 202 and RXT 208. CC-RISC is configured to run a congestion control algorithm and, in response to the algorithm running, output indications to SXP 204 via CCC 210 (in some embodiments, CC-RISC may send some or all output indications directly to SXP 204).

[0067] In some embodiments, when the network adapter is configured as an RP, the indications output by CC-RISC 212 to the pipeline include: i) indications sent by CC-RISC via CCC: sending RTT indications (to measure round trip delay) and CC packets; and, ii) indications sent by CC-RISC directly to SXW 204: Tx rate, for each packet flow (in some embodiments, CC-RISC sends increasing rate or decreasing rate indications; in other embodiments, CC-RISC always sends updated rates for all flows).

[0068] In an embodiment, when the network adapter is configured as an NP, the indication output by CC-RISC 212 includes CC packets, and the network adapter may send back CC packets in response to some ingress packets.

[0069] The interface between CC-RISC 212 and CCC 210 may include merged CC events sent by CCC to CC-RISC, and event responses (including ACK, NACK, RTT) sent by CC-RISC to CCC. Typically, CCC sends event descriptors to CC_RISC instead of events, which are condensed representations of the corresponding events. The interface between CCC and CC-RISC also includes CC packets—CC packets that CCC forwards to CC-RISC that CCC receives from RXT; and CC-RISC may generate CC packets and send CC packets to SXP via CCC.

[0070] In various embodiments, variations of the above CCC-CC-RISC interface may be used. For example, in some embodiments, the CCC may send Tx rate increment and decrement indications to the CCC; the CCC may store the current Tx rates of all flows, increase or decrease the rate value in response to the indication of the CC-RISC, and send the updated rates of all flows to the SXW.

[0071] To achieve reliable round-trip time measurements, network adapter 112, when configured as an NP, should route RTT packets from input to output with minimal delay. Figure 2 In the example implementation shown, PSA 106 routes the ingress RTT packet directly to SXP 204, which sends the RTT packet back to the network adapter that sent the ingress RTT. PSA 206 and SXP 204 typically add ingress and egress timestamps, respectively, to the RTT packets sent by the NP network adapter.

[0072] In some embodiments of the present invention, in order to avoid congestion in the congestion control event processing, the congestion management is configured to discard (drop) some less important CC events.

[0073] Understandably, Figure 2The structure of the network adapter 102 shown in is cited as an example. The network adapter according to the disclosed technology is not limited to the above description. In alternative embodiments, for example, some congestion control algorithms can be executed by hardware (e.g., by CCC 210); in an embodiment, CCC 210 is configured to use hardware to execute all congestion control algorithms, and CC-RISC 212 may not be required. In other embodiments, any suitable division of congestion control tasks between hardware (e.g., CCC 210) and firmware (e.g., CC-RISC 212) can be used, including, for example, dynamic division based on RISC-V payload.

[0074] In some embodiments, more events may be monitored, such as an Rx event on each received packet.In an embodiment, the CCC or a portion thereof may be embedded within the transmit pipeline and / or within the receive pipeline.

[0075] CC grouping and CC-MAD grouping

[0076] In an embodiment according to the present invention, peer NP and RP network adapters can exchange user programmable congestion control (CC) messages. Such CC messages can include CC management datagrams (CC-MAD) and other types of packets defined by programmers (the following messages will be referred to as non-MAD CC packets).

[0077] CC-MAD is a Management Datagram (MAD) defined by InfiniBand Specification Volume 1 Version 1.4 Chapter 13.4. According to an embodiment of the present invention, CC-MAD is encapsulated in a CC-MAD packet (and is therefore limited to 256 bytes) and processed by Queue Pair 1 (QP 1), which may be assigned a high priority in the Tx pipe. In some embodiments, congestion management sends CC-MAD directly to SXP 204 ( Figure 2 ), bypassing the queue pair mechanism. In NP, PSA 206 ( Figure 2 ) detected MAD may be given higher priority than other packets; in some embodiments, such packets are routed directly to CC management.

[0078] According to an embodiment, a user of a network adapter may define several CC-MAD types and insert special opcodes in the CC-MAD payload to signal congestion control events between peer network adapters.

[0079] Figure 3 is a table schematically showing the structure of CC-MAD 300 according to an embodiment of the present invention. CC-MAD contains 256 bytes. Figure 3 Each line in the byte array depicts the contents of four bytes (32 bits).

[0080] The first six entries of CC-MAD ( Figure 3 The first six lines in the ; bytes 23:0) include the MAD field defined by InfiniBand. The Base Version (byte 3) value is 0x01. The Class (byte 2), Class Version (byte 1), and Attribute ID (bytes 18, 19) are firmware configurable (FW CR).

[0081] The next two entries of the CC-MAD (bytes 31:24) define a firmware-configurable 64-bit key. The key changes every preset period (e.g., one second) according to a preset algorithm (e.g., by advancing a feedback shift register (FSR)). The purpose of the key is to prevent a hacker from trying to overwhelm the network adapter by sending a large number of CC-MADs—the network adapter will reject any MAD packets that do not match the current key. The key check is typically done by hardware in the Rx pipeline with little or no impact on performance.

[0082] The next entry (bytes 35:32) is an 8-bit firmware configurable capability mask (unused bits are 0), which can be used to set the capability parameters of the packet (the use of the capability field in the CC MAD packet is beyond the scope of this invention). Next, byte 39 defines the congestion control message type, which can be set and defined according to the type of CC-MAD, and bytes 38:36 define the message size in double words (because the message size is limited to 256 bytes (64 double words), at most 6 bits of this 24-bit field will be non-zero).

[0083] Bytes [255:44] are the payload of CC-MAD, which can be set by FW according to the needs of the selected congestion control algorithm. Since the length of the packet may vary, the last unused byte is usually (but not necessarily) 0.

[0084] It is understood that the following are cited by way of example: Figure 2 The structure of CC-MAD 300 is shown. CC-MAD according to the disclosed technology is not limited to the above description. In alternative embodiments, CC-MAD may include more than 256 bytes; the content and order of the various fields may be different. Security means other than identification keys may be used, including, for example, public key authentication.

[0085] In some embodiments, other types of control packets may be used to perform other types of control operations instead of or in addition to CC-MAD. However, the Tx and Rx pipelines will assign a lower priority to such control packets (QP-1, assigned to CC-MAD, which will usually receive a high priority), and the security of such packets will not be checked by hardware.

[0086] API

[0087] In some embodiments, the CCC to CC-RISC interface includes an application programming interface (API). In an example embodiment, the API includes the following parameters:

[0088]

[0089]

[0090] flow chart

[0091] Figure 4A and Figure 4B The congestion management flow for RP and NP configurations is shown. The flow chart is presented for conceptual clarity and does not necessarily represent a sequence of events; Figure 4A , Figure 4B As described herein, congestion management 114 performs the flowchart, but in embodiments, the various steps of the flowchart may be performed by other components of the network adapter, including receive pipe 112, transmit pipe 110, and a CPU (not shown) coupled to the network adapter.

[0092] Figure 4A 4 is a flow chart schematically illustrating a congestion control flow 400 in a network adapter configured as a reaction point (RP) according to an embodiment of the present invention. The flow chart is provided by the congestion management 114 ( Figure 2 ) is performed using hardware (e.g., CCC210) and / or using firmware (e.g., CC-RISC 212).

[0093] The flow begins at a Get Tx Events step 402 where the congestion management receives any pending Tx events from the output pipe, such as notifications that a burst of packets (corresponding to the same flow) has been sent. An address vector may be received with each Tx event, indicating the destination of the burst of packets.

[0094] Next, congestion management receives any pending Rx events, including ACK and NACK packets, from the Rx pipe in Get Rx Events step 404. In Get Rx-RTT Events 406, congestion management receives any returned RTT packets (that the RP network adapter has sent to the NP network adapter) from the receive pipe, which include timestamps and facilitate the calculation of the round trip time.

[0095] Next, in a Get CC-MAD step 408, the congestion management receives from the receive pipe any CC-MADs (typically initiated by the NP network adapter) that the receive pipe may have received from the network.

[0096] After step 408, congestion management has received all current CC inputs and now proceeds to the allow CC algorithm step 410 (in some embodiments, in response to the event, congestion management runs the CC algorithm after all events). CC algorithms may vary and include a variety of existing and yet to be developed algorithms; the algorithms may change dynamically in response to changing network conditions and various packet flows.

[0097] In the next three steps, CC management may send the output calculated in step 410. CC management may output a new Tx rate for each current packet flow in an output rate step 412. In some implementations, CC management only outputs the Tx rate for flows whose rates have been changed.

[0098] In CC algorithm step 410, CC management may decide to measure the round trip time between the RP and the NP and / or send a CC-MAD packet to the NP. In send RTT step 414, CC management may send an RTT packet to the NP; and in send CC-MAD step 416, CC management may send a CC-MAD. After step 416, congestion management reenters step 402.

[0099] Figure 4B 4 is a flowchart schematically illustrating a congestion control flow 450 in a network adapter configured as a notification point (NP) according to an embodiment of the present invention. Similar to flowchart 400, flowchart 450 is provided by congestion management 114 ( Figure 2 ) is performed using hardware and / or firmware.

[0100] The process begins at Get Rx Events step 452, where the congestion management receives Rx events from the receive pipeline, such as receiving packets from the RP. Some packets may have an Explicit Congestion Notification (ECN) set (this bit may have been set by a switch in the network, such as switch 116, Figure 1 Additionally (or alternatively), in step 452, if ingress packets become congested at the RXT (ie, the rate at which ingress packets are processed is slower than the rate at which packets are received), congestion management may obtain congestion notifications from the RXT 208.

[0101] Next, CC Management acquires any CC-MAD packets that the RP may have sent in Get CC-MAD step 454. After step 454, CC Management proceeds to Run CC Algorithm step 456 to process the Get Rx event. In step 45, CC Management may prepare a response and may prepare a CC-MAD packet.

[0102] In the notify congestion step 458, CC management may send an explicit congestion notification packet (CNP) to the RP, or notify congestion in other ways; in the send CC-MAD step 462, CC management may send a CC-MAD (prepared in step 456) to the RP. After step 462, CC management re-enters step 452.

[0103] according to Figure 4B In the illustrated example implementation, CC management runs a parallel RTT processing flow in an NP configuration. The RTT processing flow begins with a check RTT step 464, where congestion management checks whether a new RTT is received from the RP. If a new RTT is not received, congestion management loops through step 464 until an RTT is received. When an RTT is received, congestion management proceeds to a process RTT step 466, where congestion management adds a timestamp to the RTT and sends the modified RTT packet back to the RP. After step 466, congestion management re-enters step 464.

[0104] It should be understood that Figure 4A and Figure 4B The flowcharts shown in and described above are cited as examples. In alternative embodiments, other suitable flowcharts may be used. The steps of the flowcharts may be performed in a different order; some or all steps may be performed simultaneously. Figure 4A , Figure 4B In the example implementation shown in , events are queued in the Rx pipe and the Tx pipe, which forward the events to CC management at steps such as 402, 404, 452. In alternative implementations, when such events are received, the Rx pipe and / or the Tx pipe pass the events to CC management; and CC management can queue the events until needed. In other implementations, the Rx pipe and / or the Tx pipe can combine event groups and send the combined events to CC management. In an implementation, step 466 can be completed by dedicated circuitry of CC management.

[0105] Figure 1 , Figure 2 and Figure 3 The configuration of the network adapter shown in, including its components and subcomponents, is an example configuration depicted purely for conceptual clarity. Any other suitable configuration can be used in alternative embodiments. The different network adapter elements can be implemented using suitable hardware, such as in one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), using hardware, using software, or using a combination of hardware and software elements.

[0106] CC-RISC 212 typically includes one or more programmable processors that are programmed with software to perform the functions described herein. The software may be downloaded to the processor in electronic form, such as over a network, or it may be provided and / or stored in a non-transitory tangible medium, such as magnetic, optical or electronic memory, alternatively or additionally.

[0107] Although the embodiments described herein primarily address congestion control in communication networks, the methods and systems described herein may also be used in other applications, such as in adaptive routing. In general, management circuitry in a network adapter may be configured to receive events from at least some packets exchanged with a peer network adapter from a Tx pipe and from an Rx pipe, exchange appropriate user-programmable control packets with the peer network adapter, and perform control operations in response to the events and the user-programmable control packets. Any suitable control operation may be performed in this manner.

[0108] It should therefore be understood that the above embodiments are cited as examples, and the present invention is not limited to the contents specifically shown and described above. On the contrary, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as changes and modifications that would occur to a person skilled in the art after reading the foregoing description and that are not disclosed in the prior art. The documents incorporated by reference into this patent application should be considered as an integral part of this application, unless any term defined in these incorporated documents in a manner that conflicts with the explicit or implicit definition in this specification, only the definition in this specification should be considered.

Claims

1. A network adapter, comprising: a receiving (Rx) pipe configured to receive packets sent by the peer network adapter through the network and process the received packets; a transmit (Tx) pipe configured to transmit packets to the peer network adapter over the network; as well as A congestion management circuit, the congestion management circuit being configured to: receiving user-defined congestion control (CC) software code that executes a CC algorithm selected from a plurality of different CC algorithms, wherein the selected CC algorithm specifies user-programmable congestion control packets to be exchanged between the congestion management circuit and a peer congestion management circuit in the peer network adapter; receiving CC events from the Tx pipe and the Rx pipe resulting from at least some of the packets exchanged with the peer network adapter; exchanging the user-programmable congestion control packets specified in the selected CC algorithm with the peer congestion management circuit in the peer network adapter; and relieving congestion affecting one or more of the packets in response to the CC event and the user programmable congestion control packet by executing the selected CC algorithm.

2. The network adapter of claim 1 , wherein the congestion affects the packets sent to the peer network adapter, and wherein the congestion management circuit is configured to throttle a transmission rate of the packets sent by the Tx pipe based on the CC events and the user-programmable congestion control packets.

3. The network adapter of claim 1 , wherein the congestion affects the packets received from the peer network adapter, and wherein the congestion management circuit is configured to generate a congestion notification based on the CC event and the user-programmable congestion control packet and send the congestion notification to the peer network adapter.

4. The network adapter of claim 1, wherein the congestion management circuit supports an application programming interface (API) for receiving the user-defined congestion control (CC) software code.

5. The network adapter of claim 1, wherein the programmable congestion control packet has a format of a management datagram (MAD) packet.

6. The network adapter of claim 1, wherein the congestion management circuit comprises a hardware congestion control circuit configured to pre-process the CC event, and a user-programmable congestion control processor configured to alleviate the congestion in response to the pre-processed CC event. 7 . The network adapter of claim 6 , wherein the hardware congestion control circuit is configured to merge two or more of the CC events and provide the merged CC event to the congestion control processor.

8. A communication method, comprising: Using the receive (Rx) pipe of the network adapter, receiving packets sent by the peer network adapter through the network, and processing the received packets; sending a packet to the peer network adapter over the network using a transmit (Tx) pipe of the network adapter; using congestion management circuitry in the network adapter; receiving user-defined congestion control (CC) software code that executes a CC algorithm selected from a plurality of different CC algorithms, wherein the selected CC algorithm specifies user-programmable congestion control packets to be exchanged between the congestion management circuit and a peer congestion management circuit in the peer network adapter; receiving CC events from the Tx pipe and the Rx pipe resulting from at least some of the packets exchanged with the peer network adapter; exchanging the user-programmable congestion control packets specified in the selected CC algorithm with the peer congestion management circuit in the peer network adapter; and Congestion affecting one or more of the packets is alleviated in response to the CC event and the user programmable congestion control packet by executing the selected CC algorithm.

9. The method of claim 8, wherein the congestion affects the packets sent to the peer network adapter, and wherein alleviating the congestion comprises throttling a transmission rate of the packets sent by the Tx pipe based on the CC events and the user-programmable congestion control packets.

10. The method of claim 8, wherein the congestion affects the packets received from the peer network adapter, and wherein alleviating the congestion comprises generating a congestion notification based on the CC event and the user-programmable congestion control packet, and sending the congestion notification to the peer network adapter.

11. The method of claim 8, and comprising receiving the congestion control (CC) software code in the congestion management circuit using an application programming interface (API), wherein alleviating the congestion comprises running the CC software code.

12. The method of claim 8, wherein the programmable congestion control packet has a format of a management datagram (MAD) packet.

13. The method of claim 8, wherein alleviating the congestion comprises pre-processing the CC events using hardware congestion control circuitry, and alleviating the congestion in response to the pre-processed CC events via a user programmable congestion control processor.

14. The method of claim 13, wherein pre-processing the CC event comprises merging two or more of the CC events and providing the merged CC event to the congestion control processor.

15. A network adapter, comprising: a receiving (Rx) pipe configured to receive packets sent by the peer network adapter through the network and process the received packets; a transmit (Tx) pipe configured to send packets to the peer network adapter over the network; and A management circuit, the management circuit being configured to: receiving user-defined software code that executes a control algorithm selected from a plurality of different control algorithms, wherein the selected control algorithm specifies user-programmable control packets to be exchanged between the management circuit and a peer management circuit in the peer network adapter; receiving events from the Tx pipe and the Rx pipe resulting from at least some of the packets exchanged with the peer network adapter; exchanging user programmable control packets specified in the selected control algorithm with peer management circuitry in the peer network adapter, and performing control operations in response to the events and the user programmable control packets by executing the selected control algorithm.

16. A communication method, comprising: Using the receive (Rx) pipe of the network adapter, receiving packets sent by the peer network adapter through the network, and processing the received packets; sending a packet to the peer network adapter over the network using a transmit (Tx) pipe of the network adapter; as well as Using management circuitry in the network adapter; receiving user-defined software code that executes a control algorithm selected from a plurality of different control algorithms, wherein the selected control algorithm specifies user-programmable control packets to be exchanged between the management circuit and a peer management circuit in the peer network adapter; receiving events from the Tx pipe and the Rx pipe resulting from at least some of the packets exchanged with the peer network adapter; exchanging user programmable control packets specified in the selected control algorithm with peer management circuitry in the peer network adapter; and A control operation is performed in response to the event and the user programmable control packet by executing a selected control algorithm.

Images