Congestion notification in multi-queue environment

Abstract

Examples described herein relate to a network interface device. In some examples, a network interface device includes: a host interface; direct memory access (DMA) circuitry; a network interface; and a circuit arrangement. The circuitry may be configured to: based on telemetry data received from the at least one switch: select a next hop network interface device from the plurality of network interface devices based on the received telemetry data.

Description

[0001] Related applications

[0002] This application claims priority under 35 U.S.C. §365(c) to U.S. Application No. 18 / 239,467, filed on August 29, 2023, which in turn claims priority to U.S. Provisional Application No. 63 / 433,736, filed on December 19, 2022. The entire contents of these applications are incorporated herein by reference in their entirety. Background Art

[0003] Data centers provide processing, storage, and networking resources. For example, cars, smartphones, laptops, tablets, or Internet of Things (IoT) devices can utilize data centers to perform data analysis, data storage, or data retrieval. Devices in data centers are connected using high-speed networking equipment such as network interfaces, switches, or routers.

[0004] Networking devices utilize congestion control (CC) to attempt to reduce congestion by limiting the sending rate of flows and limiting outstanding unacknowledged packets. Some programmable switches can detect and propagate per-hop congestion status to network interface devices. For example, CC schemes such as High Precision Congestion Control (e.g., Li et al., “HPCC: High Precision Congestion Control,” SIGCOMM (2019)) configure switches to propagate per-hop congestion status to the sending network interface device and one or more switches. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An example of allocating bandwidth to different traffic classes is shown.

[0006] Figure 2 An example system is depicted.

[0007] Figure 3 Depicts an example process.

[0008] Figure 4 An example network interface device is depicted.

[0009] Figures 5A to 5C An example network interface device is depicted.

[0010] Figure 6 An example system is depicted. DETAILED DESCRIPTION

[0011] Figure 1An example of allocating bandwidth to different traffic classes 0 through 2 is shown. The scheduler 110 of the switch 100 can allocate bandwidth to packets from queues for traffic classes 0 through 2. Separate queues can be assigned to traffic classes to buffer packets of the traffic classes. In this example, queues 0 through 2 are assigned to packets of corresponding traffic classes (TCs) 0 through 2. If a TC is not using its allocated bandwidth, the bandwidth of that TC can be allocated to another TC. For example, if TC1 is allocated more than 20% of the egress port bandwidth but is using 20% ​​of the available egress port bandwidth, the scheduler 110 can allocate the unused bandwidth to another TC (e.g., TC0 or TC2).

[0012] By using queues for different TCs, the switch 100 can provide differentiated quality of service (QoS) and send packets associated with different TCs via egress ports. However, some CC schemes assume that only packets of a single business class are sent via one egress port. For example, the switch 100 provides congestion telemetry data to the transmitter 150. The congestion telemetry data can represent congestion for each egress port rather than for each queue. For example, utilization of HPCC involves calculating congestion based on queuing in the switch 100 and comparing the drain rate of the queue (e.g., the rate at which packets are sent out from the queue to the egress port) with the target egress port bandwidth (BW). In the case where multiple queues for different business classes provide packets to the egress port, the drain rate of one queue may not reach the target egress port BW because the target egress port BW is shared by multiple queues and the utilization determined based on HPCC may not accurately indicate the utilization of the target egress port BW.

[0013] Transmitter 150 can receive congestion telemetry data generated by switch 100 and can adjust the send rate of packet flows based on the received congestion telemetry data. However, because congestion telemetry data is per-port, transmitter 150 may adjust its send rate without considering that other flows are also utilizing the bandwidth of the egress port. In some cases, transmitter 150 may over-allocate send bandwidth to packets of flows reporting congestion telemetry data.

[0014] Various examples described herein can provide congestion telemetry data for a CC based on taking into account queue depth information of one or more other queues utilizing the same egress or output port, and potentially based on an arbiter configuration that allocates egress bandwidth from an egress port based on uniform or uneven weighting of different queues and associated traffic classes. According to various examples, a network interface device can calculate a utilization (U) for a traffic class and send congestion telemetry data including U for that traffic class as feedback to a traffic sender network interface device. For example, a switch can determine per-queue utilization based on one or more of the following: outgoing or egress queue length (qlen) for class i; i )(where i is an integer, and different values ​​of i are assigned to different service classes), the transmission rate of class i (txRate i ), or egress bandwidth or line or port egress rate (B). Receipt of congestion telemetry data may enable the traffic sender network interface device to maintain, reduce, or increase the send rate of packets of class i. Thus, for queues associated with different traffic classes and potentially different assigned bandwidth allocations (e.g., weights), when packets from multiple queues are sent out at an egress port, the network interface device (e.g., a switch) can measure the utilization of each traffic class and indicate this utilization to the traffic sender network interface device in the telemetry congestion data.

[0015] In some examples, the network interface device may have access to data regarding the length of the egress queue and may calculate the utilization (U) of traffic class i as shown in equation (1):

[0016]

[0017] in,

[0018] qlen i It can represent the queue length,

[0019] Bandwidth Delay Product (BDP) can be based on the transmission rate (txRate i ) to determine,

[0020] T may represent a basic round trip time (RTT) and may be calculated based on the time difference between a packet transmission and a packet reception indication, and

[0021] txRate i It can represent the sending rate of the egress queue of category i of the switch.

[0022] In some examples, a switch can determine an egress queue depth, an estimated per-class send rate, and an egress port bandwidth, and (1) indicate a utilization value that will reduce the send rate of traffic class i associated with the queue based on meeting or exceeding the queue depth level, or (2) indicate a utilization value that will cause the send rate of class i associated with the queue to be increased based on not meeting and not exceeding the queue depth level. In some examples, a switch can determine the utilization of a queue based on a queue occupancy level and an arbiter weight (or queue class weight) configured by a control plane of the switch, and (1) indicate a utilization value that will reduce the send rate of class i associated with the queue based on meeting or exceeding the queue depth level, or (2) indicate a utilization value that will cause the send rate of class i associated with the queue to be increased based on not meeting and not exceeding the queue depth level.

[0023] For example, when a traffic class shares egress port bandwidth with another traffic class, if packets in a queue for the traffic class meet or exceed a first level, the CC scheme can reduce the send rate of the traffic class and potentially reduce memory utilization for such queues at the switch. When a first traffic class shares egress port bandwidth with a second traffic class, and the first traffic class is utilizing less egress port bandwidth than allocated to the first traffic class, and the queue associated with the second traffic class is less than the first level, the switch can instruct one or more senders of packets of the second traffic class to increase the send rate so that the second traffic class utilizes the increased level of egress port bandwidth. In some examples, the second traffic class is subject to a CC based on HPCC, Poseidon (e.g., W. Wang, M. Moshref, Y. Li, G. Kumar, TENg, Cardwell, and N. Dukkipati, "Poseidon: Efficient, Robust, and Practical Data Center CC with Deployable INT" (2023)), or other CC.

[0024] Figure 2 An example system is depicted. At the request of a process or driver executed by a host system (not shown), transmitter 200 can send one or more packets to receiver 230 via one or more switches (e.g., switches 205, 210, and / or 220). The example host system at least refers to Figure 6Packet may be used herein to refer to various formatted collections of bits that can be sent across a network, such as an Ethernet frame, an IP packet, a Transmission Control Protocol (TCP) segment, a User Datagram Protocol (UDP) datagram, etc. References to the L2, L3, L4, and L7 layers (Layer 2, Layer 3, Layer 4, and Layer 7) are references to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer of the Open Systems Interconnection (OSI) layer model, respectively.

[0025] A flow can include a sequence of packets transmitted between two endpoints, typically representing a single session using a known protocol. Thus, a flow can be identified by a set of defined tuples, and for routing purposes, a flow is identified by two tuples identifying the endpoints (e.g., source and destination addresses). For content-based services (e.g., load balancers, firewalls, intrusion detection systems, etc.), flows can be distinguished at a finer granularity by using N-tuples (e.g., source address, destination address, IP protocol, transport layer source port, and destination port). Packets in a flow are expected to have the same set of tuples in the packet header. The packet flow to be controlled can be identified by a combination of a tuple (e.g., Ethernet type field, source and / or destination IP address, source and / or destination User Datagram Protocol (UDP) port, source / destination TCP port, or any other header field) and a unique source and destination queue pair (QP) number or identifier. A packet can be used herein to refer to various formatted sets of bits that can be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc.

[0026] References to flows may alternatively or additionally refer to tunnels (e.g., Multiprotocol Label Switching (MPLS) Label Distribution Protocol (LDP), Segment Routing on the IPv6 Data Plane (SRv6) source routing, VXLAN tunnel services, GENEVE tunnel services, network slicing based on Virtual Local Area Networks (VLANs), the techniques described in Mudigonda, Jayaram, et al., “SPAIN: Off-the-Shelf Commodity Data Center Ethernet for Multipathing over Arbitrary Topologies,” NSDI. Vol. 10.2010 (hereinafter referred to as “SPAIN”), etc.

[0027] In some examples, the transmitter network interface device 200, switch 205, switch 210, switch 220, and / or receiver network interface device 230 may include one or more of the following: a network interface controller (NIC), a NIC supporting remote direct memory access (RDMA), a SmartNIC, a router, a switch, a forwarding unit, an infrastructure processing unit (IPU), a data processing unit (DPU), or an edge processing unit (EPU). The edge processing unit (EPU) may include a network interface device that utilizes a processor and an accelerator (e.g., a digital signal processor (DSP) for virtualized radio access network (vRAN), cryptographic operations, compression / decompression, etc.), a signal processor, or a wireless-specific accelerator). In some examples, the transmitter network interface device 200, switch 205, switch 210, switch 220, and / or receiver network interface device 230 may be implemented as one or more of the following: one or more processors; one or more programmable packet processing pipelines; one or more accelerators; one or more application-specific integrated circuits (ASICs); one or more field programmable gate arrays (FPGAs); one or more memory devices; one or more storage devices; and the like. In some examples, the transmitter network interface device 200, the switch 205, the switch 210, the switch 220, and / or the receiver network interface device 230 may include a reference Figure 4 、 5A , 5B, 5C and / or 6 described circuit devices, firmware and / or software.

[0028] For example, switch 205, switch 210, and / or switch 220 may include an instance of packet processing circuitry 212. In this example, switch 210 may utilize packet processing circuitry 212 to generate utilization data or congestion telemetry data and transmit the utilization data or congestion telemetry data to transmitter 200 or receiver 230, and receiver 230 may transmit the utilization data or congestion telemetry data to transmitter 200. However, switch 205 and / or switch 220 may perform operations similar to those of packet processing circuitry 212 to generate utilization data or congestion telemetry data and transmit the utilization data or congestion telemetry data to transmitter 200 or receiver 230, and receiver 230 may transmit the utilization data or congestion telemetry data to transmitter 200. In some examples, packet processing circuitry 212 may determine the utilization data or congestion telemetry data based on an estimated transmit rate for a traffic class or a transmit rate for the traffic class determined based on a weight applicable to the traffic class, as described herein. In some examples, utilization data or congestion telemetry data can be measured for a traffic class that shares egress port bandwidth with at least one other traffic class. In some examples, the traffic class and / or at least one other traffic class can be subject to congestion control based on HPCC, Poseidon, etc.

[0029] For example, queues 214 may include queues for categories 0 to i (where i is an integer) and may be allocated in a memory (not shown) of switch 210 or in one or more memory devices (not shown) attached to switch 210 via an interface. Congestion monitoring 216 may include a txRate that may access queue category i. i and qlen i The export pipeline, where txRate i It can represent the sending rate of category i, and qlen i may represent the egress or exit queue length for class i. The egress scheduler 218 may schedule the egress of packets based on the allocated bandwidth or weights applied to the queues of classes 0 through i.

[0030] For example, the estimated sending rate for a traffic class may be determined as follows: In some examples, congestion monitoring 216 may determine utilization data based on equation (2) below.

[0031]

[0032] Among them, qlen th It can represent the threshold level that the switch uses to limit the txRate of senders of category i so that the total egress memory usage of category i traffic remains within qlen th or lower than qlen thIn some examples, qlen can be specified as the number of packets or the number of bytes. In some examples, qlen th Can be set to 1 packet so that any packet queuing will result in a slowdown in the send rate.

[0033] If qlen i >qlen th , then U can be set to more than 1, which can cause the sender (e.g., transmitter 200) to slow down the transmission of packets of class i because the sending rate can be inversely proportional to the value of U. Otherwise, if qlen i ≤qlen th , the packet processing circuitry 212 may perform a binary search (eg, the current txRate i and B (the average of the port rate) and set U < 1 to cause one or more senders of packets of class i (e.g., transmitter 200 and / or other transmitters) to increase the sending rate of packets of class i. The binary search can estimate the sending rate of packets of class i based on utilization because it uses the queue utilization for only one class (e.g., class i).

[0034] The operation of the packet processing circuit device 212 to estimate the transmission rate of the service class may be as follows: lines 2 to 8, which will be described later.

[0035]

[0036] Line 2 calculates U for received data packets, not for acknowledgements (ACKs) of packet reception. Lines 3-6 allow the switch to provide txRate i and qlen i . Line 7 checks whether the U value received in the packet from another switch or network interface device is less than the calculated U, and if so, line 8 updates the packet congestion telemetry data information from the other switch or network interface device with the higher U value. The return ACK packet or other packet from the receiver 230 can carry the U in the congestion telemetry data to the transmitter 200, and the transmitter 200 can adjust its txRate for category i based on HPCC or other schemes. In HPCC, the service sender network interface device can calculate U by utilizing an ACK embedded in in-band network telemetry (INT) collected at one or more switches between the receiver 230 and the transmitter 200. Other ways of transmitting congestion telemetry data are described herein.

[0037] For example, the sending rate for a traffic class based on the weight applicable to the traffic class may be determined as follows: For an egress port for sending packets associated with queue class i, the packet processing circuitry 212 may have access to an assigned weight W i and queue occupancy or queue length (qlen i or Q i ). For example, the egress pipeline in the packet processing circuitry 212 may access this information via a control plane interface. The congestion monitoring 216 may determine U for class i based on the total weights of the different queue classes saved during initialization or switch configuration. For example, the total weights of the different queue classes may be determined using equation (3):

[0038] W total =∑W i …#(3)

[0039] Among them, W total ≤100 (at most 100% of B).

[0040] For an incoming data packet at queue i, Q j , where i≠j, and the congestion monitoring 216 can calculate the unused weight (W) using equations (4) and (5) as follows unused ) and activity weight (W active ):

[0041] W unused =∑W j ,if Q j =0…#(4)

[0042] W active =∑W j ,if Q j ≠0…#(5)

[0043] in,

[0044] W i can be expressed as a percentage of the total egress port bandwidth allocated to class i,

[0045] W j can be expressed as the percentage of the total egress port bandwidth allocated to any other class j≠i,

[0046] W unused can represent the percentage of total egress port bandwidth that is unused, and

[0047] W active Can represent the percentage of total egress port bandwidth used.

[0048] The assigned weight W for queue i can be determined using equation (6) alloc :

[0049]

[0050] Therefore, W alloc provides the percentage of line rate B used by packets from queue category i. The packet processing circuitry 212 may determine U for queue i based on equation (7):

[0051]

[0052] An example of the operation of the packet processing circuitry 212 to determine the transmission rate for a traffic class based on the weight applicable to the traffic class may be as shown in lines 1-20.

[0053]

[0054]

[0055] Lines 1-3 may execute equation (3) to determine the total weight percentages for the different categories. Line 5 may check whether it is a data packet or an ACK packet. Lines 6-13 may execute equations (4) and (5), while line 14 may execute equation (6). Lines 15-18 may execute equation (7). Lines 19-20 may check whether the U value received in a packet from another switch or network interface device is less than the newly calculated U, and if so, update the packet congestion telemetry data information with the higher U value.

[0056] Receiver 230 or a switch (e.g., 220, 210, or 205) may send congestion telemetry data to another switch or transmitter 200 based on one or more of the following: in-band network telemetry (INT) (e.g., P4.org Application Working Group "In-band Network Telemetry (INT) Data Plane Specification" Version 2.1 (2020)), round-trip time (RTT) probes, acknowledgment (ACK) of packet receipt messages sent to transmitter 200, Internet Engineering Task Force (IETF) draft-lapukhov-lane-probe-01 "Data Plane Probing for In-band Telemetry Collection" (2016), IETF draft-ietf-ippm-ioam-data-09 "In-Place Operations, Administration, and Maintenance (IOAM)" (March 8, 2020). In-Place Operations, Administration, and Maintenance (IOAM) may record operational and telemetry information in a packet as it traverses a path between two devices in a network. IOAM describes data fields and associated data types used for in-place OAM. The in-situ OAM data field may be encapsulated into various protocols, such as Network Services Header (NSH) (eg, IETF RFC 8300 (2020)), Segment Routing, Geneve, IPv6 (via extension headers), or IPv4.

[0057] In some examples, a switch (e.g., 220, 210, or 205) may send the congestion telemetry data to another switch for forwarding to transmitter 200 so that transmitter 200 may receive the congestion telemetry data before, or instead of, receiving an ACK with the congestion telemetry data from receiver 230.

[0058] In some examples, InfiniBand can be used TM The InfiniBand Architecture Specification Volume 1 (IBTA) specifies a reserved field for the Congestion Notification Packet (CNP) to send congestion telemetry data. See, for example, InfiniBand TM Architecture Specification Volume 1 (2007) and its revisions, variations, or updates. A congested point (eg, an endpoint or a switch device) may directly or indirectly send a congestion notification packet (CNP) to the transmitter 200 to reduce its sending rate and alleviate congestion.

[0059] For class i, calculating U at the switch can also enable a single highest U value to be sent in a single packet on the path between the receiver and sender nodes. For class i, the highest U along the path between transmitter 200 and receiver 230 can represent the utilization of the bottleneck link, thereby signaling the lowest possible txRate on the entire path. In addition, the U value for class i can be collected periodically (e.g., every RTT) by an in-band flow analyzer (IFA) (e.g., an IFA from Broadcom), further reducing the overhead of sending congestion information. Note that congestion telemetry data for multiple different traffic classes can be sent to transmitter 200 by one or more of switches 205, 210 and / or 220.

[0060] The transmitter 200 or its host can receive the received U i To adjust the txRate of category i i . The congestion control 202 may schedule packet transmissions from multiple class queues to network links based on weighting, weighted differential round-robin (WDRR), weighted round-robin, etc. The congestion control 202 may select a next-hop network interface device from a plurality of network interface devices based on received congestion telemetry data. For example, the congestion control 202 may select a next-hop network interface device (e.g., a switch) among the plurality of network interface devices that is associated with a lowest received utilization (U) value or is otherwise identified as being associated with a path that experiences an amount of queuing or packet drops that is equal to or less than a configured level. For example, the congestion control 202 may select a next-hop network interface device (e.g., a switch) or a path among the plurality of network interface devices or paths by increasing a weight assigned to a queue or an egress port that is used to provide packets to a path associated with the selected next-hop network interface device or the selected path. The selected next-hop network interface device or the selected path may be associated with a particular queue or a particular egress port.

[0061] In some examples, congestion control 202 can utilize congestion telemetry data to determine the bandwidth of the physical path and select physical links to aggregate based on link aggregation. Link aggregation can be based on IEEE 802.3ad (2020) and include port aggregation to assign multiple physical links to a single logical link (e.g., Link Aggregation Control Protocol (LACP) based on IEEE 802.3ad (2020)).

[0062] In some examples, congestion control 202 can utilize congestion telemetry data to select a link or path to send packets based on Multipath TCP (MPTCP) (e.g., Internet Engineering Task Force (IETF) Request for Comments 8684 (2020)), Multipath extensions for Quick UDP Internet Connections (QUIC) (e.g., QUIC Working Group, draft-ietf-quic-multipath-05 (July 2023)), Datagram Congestion Control Protocol (DCCP) extensions for Multipath Operation with Multiple Addresses (e.g., Multipath Extensions to RFC 4340 (2006)), Stream Control Transmission Protocol (SCTP) (e.g., IETF RFC 9260 (2022)), etc.

[0063] For some applications, the underlying transport layer is the Transmission Control Protocol (TCP). A number of different congestion control (CC) schemes can be used for TCP. Explicit Congestion Notification (ECN), defined in RFC 3168 (2001), allows end-to-end notification of network congestion, whereby the receiver of packets from transmitter 200 (e.g., receiver 230) sends a congestion indication back to the sender (e.g., transmitter 200). The packet sender can reduce its packet sending rate in response to receiving ECN. If the detection and response to congestion is slow or delayed, the use of ECN may result in packet drops. For TCP, congestion control 202 can apply congestion control based on heuristics based on congestion metrics such as network delay or the number of packet drops.

[0064] Congestion control 202 can apply other TCP congestion control schemes, including Google's Swift, Amazon's SRD, and Data Center TCP (DCTCP) (e.g., described in RFC 8257 (2017)). DCTCP is a TCP congestion control scheme whereby when the buffer reaches a threshold, packets are marked with ECN, and the end host receives the mark and sends the marked packets to the transmitter. Transmitter 200 can adjust the transmitter's sending rate by adjusting the congestion window (CWND) size to adjust the number of sent packets for which no receipt confirmation has been received. In response to ECN, transmitter 200 can reduce the CWND size to reduce the number of sent packets for which no receipt confirmation has been received. Swift, SRD, DCTCP, and other CC schemes adjust the CWND size based on indirect congestion metrics such as packet drops or network delay.

[0065] Figure 3An example process is depicted. The process can be performed by a switch or a network interface device. At 302, congestion telemetry data can be determined based on an operating mode. An orchestrator (e.g., Kubernetes), a system administrator, or a device manufacturer can configure the operating mode for the switch or network interface device. In a first operating mode, the congestion telemetry data can include a utilization that can be determined based on an estimated send rate for a business class, as described herein. In a second operating mode, the congestion telemetry data can include a utilization that can be determined based on a send rate weighted for a business class, as described herein. For the first and second operating modes, if a queue associated with a business class stores one or more packets, the utilization can be set so that the packet send rate for the business class is reduced. If a queue associated with a business class does not store packets or stores a number of packets below a configured level, the utilization can be set so that the packet send rate for the business class is increased.

[0066] Congestion telemetry data can be determined for multiple queues or traffic classes. In some examples, a single queue can be assigned to at least one traffic class. In some examples, multiple queues can be assigned to groups of at least one traffic class. In some examples, a single queue can be assigned to groups of multiple traffic classes.

[0067] At 304, congestion telemetry data can be sent to a transmitter of packets for at least one traffic class. For example, the congestion telemetry data can be sent using an ACK packet, a header field, a payload, an INT, an IFA, etc. Based on receiving the congestion telemetry data, the transmitter can adjust its sending rate or congestion window size, as described herein.

[0068] Figure 4 An example network interface device or packet processing device is depicted. In some examples, the circuitry of the network interface device can be used by a network interface or another network interface for packet transmission and packet reception, and by at least reference to Figure 5A 、 5B and / or the switch circuitry described in 5C, as described herein. In some examples, the packet processing device 400 can be implemented as a network interface controller, a network interface card, a host fabric interface (HFI), or a host bus adapter (HBA), and such examples can be interchangeable. The packet processing device 400 can be coupled to one or more servers using a bus, PCIe, CXL, or double data rate (DDR). The packet processing device 400 can be implemented as part of a system on a chip (SoC) that includes one or more processors, or included on a multi-chip package that also includes one or more processors.

[0069] Some examples of the packet processing device 400 are part of an infrastructure processing unit (IPU) or a data processing unit (DPU), or are used by an IPU or DPU. An xPU may refer to at least an IPU, a DPU, a GPU, a GPGPU, or other processing unit (e.g., an accelerator device). An IPU or DPU may include a network interface having one or more programmable or fixed-function processors to perform offloading of operations that could otherwise be performed by a CPU. An IPU or DPU may include one or more memory devices. In some examples, an IPU or DPU may perform virtual switch operations, manage storage transactions (e.g., compression, encryption, virtualization), and manage operations performed on other IPUs, DPUs, servers, or devices.

[0070] The network interface 400 may include a transceiver 402, a processor 404, a transmit queue 406, a receive queue 408, a memory 410, a host interface 412, and a DMA engine 452. The transceiver 402 is capable of receiving and transmitting packets according to an applicable protocol, such as Ethernet as described in IEEE 802.3, although other protocols may also be used. The transceiver 402 may receive packets from the network and transmit packets to the network via a network medium (not depicted). The transceiver 402 may include a PHY circuit device 414 and a media access control (MAC) circuit device 416. The PHY circuit device 414 may include encoding and decoding circuit devices (not shown) to encode and decode data packets according to applicable physical layer specifications or standards. The MAC circuit device 416 may be configured to assemble data to be transmitted into packets, the packets including destination and source addresses as well as network control information and error detection hash values.

[0071] Processor 404 and / or system-on-chip 450 may include one or more of a processor, a core, a graphics processing unit (GPU), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other programmable hardware device that allows programming of network interface 400. For example, an "intelligent network interface" may use processor 404 to provide packet processing capabilities in the network interface.

[0072] The processor 404 and / or the system on chip 450 may include one or more packet processing pipelines that may be configured to use information stored in a ternary content addressable memory (TCAM) table or an exact match table in some embodiments to perform match-actions on received packets to identify packet processing rules and next hops. For example, a match-action table or circuitry may be used whereby a hash of a portion of a packet is used as an index to find an entry. The packet processing pipeline may perform one or more of the following: packet parsing (parser), exact match-action (e.g., small exact match (SEM) engine or large exact match (LEM)), wildcard match-action (WCM), longest prefix match block (LPM), hash block (e.g., receive side scaling (RSS)), packet modifier (modifier), or traffic manager (e.g., send rate metering or shaping). For example, the packet processing pipeline may implement access control lists (ACLs) or packet discarding due to queue overflow.

[0073] Configuration of the operation of the processor 404 and / or the system-on-chip 450 (including its data plane) may be programmed based on one or more of: a protocol-independent packet processor (P4), software for open networking in the cloud (SONiC), Network Programming Language (NPL), DOCA TM , Infrastructure Programmer Development Kit (IPDK), etc.

[0074] As described herein, the processor 404 , system on chip 450 , or other circuitry may be configured to adjust a packet sending rate or congestion window based on received congestion telemetry data.

[0075] The packet distributor 424 can use the timeslot allocation or RSS described herein to provide distribution of received packets for processing by multiple CPUs or cores. When the packet distributor 424 uses RSS, the packet distributor 424 can calculate a hash or make another determination based on the contents of the received packet to determine which CPU or core will process the packet.

[0076] Interrupt coalescing 422 may perform interrupt moderation, whereby the network interface interrupt coalescing 422 waits for multiple packets to arrive or for a timeout to expire before generating an interrupt to the host system to process the received packet. Receive segment coalescing (RSC) may be performed by the network interface 400, whereby portions of an incoming packet are combined into segments of a packet. The network interface 400 may provide the coalesced packet to the application.

[0077] The direct memory access (DMA) engine 452 can copy packet headers, packet payloads and / or descriptors directly from host memory to the network interface, and vice versa, rather than copying the packets to an intermediate buffer at the host and then using another copy operation from the intermediate buffer to the destination buffer.

[0078] Memory 410 can be any type of volatile or non-volatile memory device and can store any queue or any instructions for programming network interface 400. Transmit queue 406 can include data for the network interface to send or a reference to the data for the network interface to send. Receive queue 408 can include data received from the network by the network interface or a reference to the data received from the network by the network interface. Descriptor queue 420 can include descriptors that reference the data or packets in transmit queue 406 or receive queue 408. Host interface 412 can provide an interface with a host device (not depicted). For example, host interface 412 can be compatible with PCI, PCI Express, PCI-x, Serial ATA and / or USB compatible interfaces (however, other interconnect standards can also be used).

[0079] Figure 5A An example switch is depicted. As described herein, various examples can be used in or with the switch to generate congestion telemetry data. Switch 504 can route packets or frames of any format or according to any specification from any port 502-0 through 502-X to any port 506-0 through 506-Y (and vice versa). Any of ports 502-0 through 502-X can be connected to a network of one or more interconnected devices. Similarly, any of ports 506-0 through 506-Y can be connected to a network of one or more interconnected devices.

[0080] In some examples, the switch fabric 510 can provide routing for packets from one or more ingress ports for processing before being sent out of the switch 504. The switch fabric 510 can be implemented as one or more multi-hop topologies, where example topologies include a torus, a butterfly, a buffered multi-stage, or a shared memory switch fabric (SMSF), among other implementations. An SMSF can be any switch fabric connected to ingress and egress ports in a switch, where an ingress subsystem writes (stores) packet segments into the fabric's memory, and an egress subsystem reads (retrieves) packet segments from the fabric's memory.

[0081] The memory 508 can be configured to store packets received at a port before being sent out from one or more ports. The packet processing pipeline 512 can include ingress and egress packet processing circuitry to process incoming packets and outgoing packets, respectively. The packet processing pipeline 512 can use a table that maps packet characteristics to associated output ports to determine which port to use to transmit a packet or frame. The packet processing pipeline 512 can be configured to use information stored in a ternary content addressable memory (TCAM) table or, in some examples, an exact match table to perform match-actions on received packets to identify packet processing rules and next hops. For example, a match-action table or circuitry can be used whereby a hash of a portion of a packet is used as an index to find an entry (e.g., a forwarding decision based on packet header content). The packet processing pipeline 512 can implement access control lists (ACLs) or packet discards due to queue overflows. The packet processing pipeline 512 can be configured to provide transport layer and security protocol endpoints and perform computations on received data, as described herein. Configuration of the operation of the packet processing pipeline 512 (including its data plane) can be programmed using P4, C, Python, Broadcom Network Programming Language (NPL), or x86 compatible executable binary files or other executable binary files. Processor 516 and FPGA 518 can be used for packet processing or modification.

[0082] Traffic manager 513 may perform hierarchical scheduling and send rate shaping and metering for packet transmissions from one or more packet queues. Traffic manager 513 may perform congestion management such as flow control, Congestion Notification Message (CNM) generation and reception, Priority Flow Control (PFC), and the like.

[0083] Figure 5B An example network forwarding system that can be used as a network interface device or router is depicted. As described herein, the forwarding system can generate congestion telemetry data. For example, Figure 5B Several ingress pipelines 520, a traffic management unit (referred to as a traffic manager) 550, and several egress pipelines 530 are shown. Although shown as separate structures, in some examples, the ingress pipelines 520 and the egress pipelines 530 can use the same circuitry resources. In some examples, the egress pipelines 530 can perform operations of the collective unit circuitry, as described herein.

[0084] The operation of the pipeline can be programmed using a programming protocol-independent packet processor (P4), C, Python, Broadcom NPL, or an x86-compatible executable binary file or other executable binary file. In some examples, the pipeline circuit device is configured to synchronously process ingress and / or egress pipeline packets and non-packet data. That is, a particular stage of the pipeline can process any combination of ingress packets, egress packets, and non-packet data in the same clock cycle. However, in other examples, the ingress pipeline and the egress pipeline are separate circuit devices. In some of these other examples, the ingress pipeline also processes non-packet data.

[0085] In some examples, in response to receiving a packet, the packet is directed to one of the ingress pipelines 520, where the ingress pipelines can correspond to one or more ports of a hardware forwarding unit. After passing through the selected ingress pipeline 520, the packet is sent to the service manager 550, where the packet is queued and placed in the output buffer 554. In some examples, the ingress pipeline 520 that processes the packet specifies which queue the service manager 550 places the packet into (e.g., based on the packet's destination or the packet's flow identifier). The service manager 550 then dispatches the packet to the appropriate egress pipeline 530, where the egress pipeline can correspond to one or more ports of a forwarding unit. In some examples, there is no necessary correlation between which ingress pipeline 520 processes the packet and which egress pipeline 530 the service manager 550 dispatches the packet to. That is, a packet can initially be processed by the ingress pipeline 520b after being received through a first port, and then processed by the egress pipeline 530a to be emitted by a second port, and so on.

[0086] At least one ingress pipeline 520 includes a parser 522, a series of multiple match-action units or circuit devices (MAUs) 524, and an inverse parser 526. Similarly, the egress pipeline 530 may include a parser 532, a series of MAUs 534, and an inverse parser 536. In some examples, the parser 522 or 532 receives a packet (a formatted set of bits in a specific order) and parses the packet into its component header fields. In some examples, the parser starts at the beginning of the packet and assigns the header fields to fields (e.g., data containers) for processing. In some examples, the parser 522 or 532 separates the packet header from the payload of the packet (up to a specified point) and sends the payload (or the entire packet, including the header and payload) directly to the inverse parser without being processed by the MAUs. The egress parser 532 may use additional metadata provided by the ingress pipeline to simplify its processing.

[0087] The MAU 524 or 534 can perform processing on the packet data. In some examples, the MAU includes a series of stages, where each stage includes one or more matching tables and an action engine. The matching table can include a set of matching entries against which packet header fields are matched (e.g., using a hash table), where the matching entries reference action entries. When a packet matches a particular matching entry, the particular matching entry references a particular action entry that specifies a set of actions to be performed on the packet (e.g., sending the packet to a particular port, modifying one or more packet header field values, dropping the packet, mirroring the packet to a mirror buffer, etc.). The action engine of the stage can perform the action on the packet, which is then sent to the next stage of the MAU. For example, the MAU can provide transport layer and security protocol endpoints and perform calculations on the received data, as described herein.

[0088] The inverse parser 526 or 536 can reconstruct the packet using the PHV modified by the MAU 524 or 534 and the payload received directly from the parser 522 or 532. The inverse parser can construct a packet that can be sent out over the physical network or sent to the traffic manager 550. In some examples, the inverse parser can construct the packet based on data received with the PHV that specifies the protocol to be included in the packet header and its own stored list of data container locations for the header fields for each possible protocol.

[0089] Traffic Manager (TM) 550 may include a packet duplicator 552 and an output buffer 554. In some examples, the traffic manager 550 may include other components such as a feedback generator for signaling output port failures, a series of queues and schedulers for these queues, a queue status analysis component, and additional components. The packet duplicator 552 of some examples performs duplication on broadcast / multicast packets, generating multiple packets to be added to the output buffer (e.g., to be distributed to different egress pipelines).

[0090] In some examples, the output buffer 554 can be part of the queuing and buffering system of the service manager. The service manager 550 can provide a shared buffer that adapts to any queuing delays in the egress pipeline. In some examples, this shared output buffer 554 can store packet data, while references (e.g., pointers) to the packet data are maintained in different queues for each egress pipeline 530. The egress pipelines can request their respective data from the common data buffer using a queuing strategy configurable by the control plane. When a packet data reference reaches the head of its queue and is scheduled for dequeueing, the corresponding packet data can be read from the output buffer 554 and read into the corresponding egress pipeline 530.

[0091] Figure 5CAn example switch is depicted. As described herein, various examples can be used in or with the switch to generate congestion telemetry data. Switch 580 may include a network interface 582 that can provide an Ethernet-compliant interface. Network interface 582 can support 25GbE, 50GbE, 100GbE, 200GbE, and 400GbE Ethernet port interfaces. Cryptographic circuitry 584 can perform at least Media Access Control Security (MACsec) or Internet Protocol Security (IPSec) decryption on received packets or MACsec or IPSec encryption on transmitted packets.

[0092] The various circuit devices may perform one or more of: service metering, packet counting, operations, administration, and maintenance (OAM), protection engine, instrumentation and telemetry, and clock synchronization (eg, based on IEEE 1588).

[0093] The database 586 can store device profiles to configure the operation of the switch 580. The memory 588 may include high bandwidth memory (HBM) for packet buffering. The packet processor 590 may perform one or more of the following: decisions about the next hop for packet forwarding, packet counting, access list operations, bridging, routing, multi-protocol label switching (MPLS), virtual private LAN service (VPLS), L2VPN, L3VPN, OAM, data center tunnel encapsulation (e.g., VXLAN and NV-GRE), etc. The packet processor 590 may include one or more FPGAs. The buffer 594 may store one or more packets. The traffic manager (TM) 592 may provide per-subscriber bandwidth guarantees and perform layered quality of service (QoS) according to a service level agreement (SLA). The fabric interface 596 may include a serializer / deserializer (SerDes) and provide an interface to the switch fabric.

[0094] Figure 4 、 5A , 5B and / or 5C of the example switch components of the apparatus may be combined, and Figure 4 、 5A , 5B and / or 5C example switch components may be included in Figure 4 、 5A , 5B and / or 5C. For example, Figure 4 、 5A Components of the examples of switches 5B and / or 5C may be implemented in a switch system-on-chip (SoC) that includes at least one interface to other circuitry in the switch system. The switch SoC may be coupled to other devices in the switch system, such as ingress or egress ports, memory devices, or host interface circuitry.

[0095] Figure 6 The system is depicted. In some examples, the circuitry of system 600 can configure network interface device 650 to generate and transmit congestion telemetry data, as described herein. In some examples, the circuitry of network interface device 650 can be used to adjust the packet transmission rate or congestion window size and / or generate congestion telemetry data, as described herein. System 600 includes a processor 610 that provides processing, operation management, and instruction execution for system 600. Processor 610 can include any type of microprocessor, central processing unit (CPU), graphics processing unit (GPU), XPU, processing core, or other processing hardware that provides processing for system 600, or a combination of processors. The XPU can include one or more of the following: a CPU, a graphics processing unit (GPU), a general-purpose GPU (GPGPU), and / or other processing units (e.g., an accelerator or a programmable or fixed-function FPGA). Processor 610 controls the overall operation of system 600 and can be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of such devices.

[0096] In one example, system 600 includes an interface 612 coupled to processor 610, which may represent a higher speed interface or a high throughput interface for system components that require a higher bandwidth connection, such as memory subsystem 620 or graphics interface component 640 or accelerator 642. Interface 612 represents interface circuitry that may be a separate component or integrated onto the processor die. Where present, graphics interface 640 interfaces with the graphics component to provide a visual display to a user of system 600. In one example, graphics interface 640 generates a display based on data stored in memory 630 or based on operations performed by processor 610, or both. In one example, graphics interface 640 generates a display based on data stored in memory 630 or based on operations performed by processor 610, or both.

[0097] The accelerator 642 may be a programmable or fixed-function offload engine that can be accessed or used by the processor 610. For example, the accelerator in the accelerator 642 may provide data compression (DC) capabilities, cryptographic services (such as public key encryption (PKE)), cryptography, hashing / authentication capabilities, decryption, or other capabilities or services. In some cases, the accelerator 642 may be integrated into a CPU socket (e.g., a connector to a motherboard or circuit board that includes a CPU and provides an electrical interface with the CPU). For example, the accelerator 642 may include a single-core or multi-core processor, a graphics processing unit, a logic execution unit single-level or multi-level cache, a functional unit that can be used to independently execute programs or threads, an application-specific integrated circuit (ASIC), a neural network processor (NNP), programmable control logic, and a programmable processing unit (such as a field programmable gate array (FPGA)). The accelerator 642 may provide multiple neural networks, CPUs, processor cores, general-purpose graphics processing units, or graphics processing units that can be used for artificial intelligence (AI) or machine learning (ML) models. For example, the AI ​​model may use or include any one or a combination of the following: a reinforcement learning scheme, a Q-learning scheme, deep Q-learning, or an asynchronous advantage actor-critic (A3C), a combination neural network, a recurrent combination neural network, or other AI or ML models. Multiple neural networks, processor cores, or graphics processing units may be used by the AI ​​or ML model to perform learning and / or reasoning operations.

[0098] Memory subsystem 620 represents the main memory of system 600 and provides storage for code to be executed by processor 610 or data values ​​to be used when executing routines. Memory subsystem 620 may include one or more memory devices 630, such as read-only memory (ROM), flash memory, one or more random access memories (RAM) (such as DRAM), or other memory devices, or a combination of such devices. Memory 630 stores and hosts an operating system (OS) 632, among other things, which provides a software platform for executing instructions in system 600. Additionally, applications 634 may execute on the software platform of OS 632 from memory 630. Applications 634 represent running programs with their own operating logic for performing one or more functions. Processes 636 represent agents or routines that provide auxiliary functionality to OS 632, one or more applications 634, or a combination. OS 632, applications 634, and processes 636 provide software logic to provide functionality for system 600. In one example, the memory subsystem 620 includes a memory controller 622, which is a memory controller for generating and issuing commands to the memory 630. It should be understood that the memory controller 622 can be a physical part of the processor 610 or a physical part of the interface 612. For example, the memory controller 622 can be an integrated memory controller integrated into a circuit with the processor 610.

[0099] Application 634 and / or process 636 may alternatively or additionally refer to a virtual machine (VM), a container, a microservice, a processor, or other software. The various examples described herein can execute an application consisting of microservices, where the microservices run in their own processes and communicate using protocols (e.g., application programming interfaces (APIs), hypertext transfer protocol (HTTP) resource APIs, messaging services, remote procedure calls (RPCs), or Google RPC (gRPC)). Microservices can communicate with each other using a service grid and can be executed in one or more data centers or edge networks. Microservices can be deployed independently using centralized management of these microservices. The management system can be written in different programming languages ​​and use different data storage technologies. Microservices can be characterized by one or more of the following: polyglot programming (e.g., code written in multiple languages ​​to capture additional functionality and efficiency that cannot be provided in a single language), or lightweight container or virtual machine deployment, and decentralized continuous microservice delivery.

[0100] In some examples, OS 632 may be Server or personal computer, VMware vSphere, openSUSE, RHEL, CentOS, Debian, Ubuntu or any other operating system. The OS and drivers can be found in the Texas etc., sold or designed by the Company.

[0101] In some examples, OS 632, a system administrator, and / or an orchestrator may configure network interface 650 to adjust the packet sending rate or congestion window size and / or generate and send congestion telemetry data, as described herein.

[0102] Although not specifically shown, it should be understood that system 600 may include one or more buses or bus systems between devices, such as a memory bus, a graphics bus, an interface bus, etc. A bus or other signal line may communicate or electrically couple components together, or communicate and electrically couple these components. A bus may include a physical communication line, a point-to-point connection, a bridge, an adapter, a controller or other circuit device or a combination. A bus may include, for example, one or more of a system bus, a peripheral component interconnect (PCI) bus, a hypertransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (FireWire).

[0103] In one example, the system 600 includes an interface 614, which can be coupled to the interface 612. In one example, the interface 614 represents an interface circuit, which can include independent components and integrated circuits. In one example, multiple user interface components or peripheral components or both are coupled to the interface 614. The network interface 650 provides the system 600 with the ability to communicate with remote devices (e.g., servers or other computing devices) over one or more networks. The network interface 650 can include an Ethernet adapter, a wireless interconnect component, a cellular network interconnect component, a USB (Universal Serial Bus), or other wired or wireless standard-based or proprietary interface. The network interface 650 can send data to a device in the same data center or rack or to a remote device, which can include sending data stored in a memory. The network interface 650 can receive data from a remote device, which can include storing the received data in a memory. In some examples, the packet processing device or network interface device 650 can refer to one or more of the following: a network interface controller (NIC), a NIC that supports remote direct memory access (RDMA), a SmartNIC, a router, a switch, a forwarding unit, an infrastructure processing unit (IPU), or a data processing unit (DPU). Reference Figure 4 、 5A , 5B and / or Figure 5C An example IPU or DPU is described.

[0104] In one example, system 600 includes one or more input / output (I / O) interfaces 660. I / O interfaces 660 may include one or more interface components through which a user interacts with system 600. Peripheral interfaces 670 may include any hardware interfaces not specifically mentioned above. Peripheral devices generally refer to devices that are dependently connected to system 600.

[0105] In one example, system 600 includes a storage subsystem 680 for storing data in a non-volatile manner. In one example, in certain system implementations, at least some components of storage 680 may overlap with components of memory subsystem 620. Storage subsystem 680 includes storage devices 684, which may be or include any conventional medium for storing large amounts of data in a non-volatile manner, such as one or more magnetic, solid-state, or optical disks, or a combination thereof. Storage devices 684 maintain code or instructions and data 686 in a persistent state (e.g., values ​​are retained despite power interruptions to system 600). Storage devices 684 may generally be considered "memory," although memory 630 is generally execution or operational memory for providing instructions to processor 610. While storage devices 684 are non-volatile, memory 630 may include volatile memory (e.g., the value or state of data is undefined if power to system 600 is interrupted). In one example, storage subsystem 680 includes a controller 682 that interfaces with storage devices 684. In one example, controller 682 is a physical part of interface 614 or processor 610 , or may include circuitry or logic in both processor 610 and interface 614 .

[0106] Volatile memory may include memory whose state (and therefore data stored therein) is indeterminate if power to the device is interrupted. Non-volatile memory (NVM) devices may include memory whose state is deterministic even if power to the device is interrupted.

[0107] In some examples, the system 600 can be implemented using an interconnected computing platform of processors, memories, storage devices, network interfaces, and other components. High-speed interconnects such as Ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Quick UDP Internet Connection (QUIC), RDMA over Converged Ethernet (RoCE), Peripheral Component Interconnect Express (PCIe), Intel Quick Path Interconnect (QPI), Intel Ultra Path Interconnect (UPI), Intel System-on-Chip Fabric (IOSF), Omni-Path, Compute Express Link (CXL), HyperTransport, High-Speed ​​Fabric, NVLink, Advanced Microcontroller Bus Architecture (AMBA) Interconnect, OpenCAPI, Gen-Z, Infinity Fabric (IF), Cache Coherent Interconnect for Accelerators (CCIX), 3GPP Long Term Evolution (LTE), (4G), 3GPP 5G, and variants thereof can be used. Data may be copied or stored to a virtualized storage node or accessed using a protocol such as NVMe over Fabrics (NVMe-oF) or NVMe (e.g., a Non-Volatile Memory Express (NVMe) device capable of operating in a manner consistent with the Non-Volatile Memory Express (NVMe) Specification, Revision 1.3c, dated May 24, 2018 (the "NVMe Specification"), or a derivative or variant thereof).

[0108] Communications between devices may occur using a network that provides die-to-die communication, chip-to-chip communication, board-to-board communication, and / or package-to-package communication.

[0109] In an example, system 600 can be implemented using an interconnected computing platform of processors, memory, storage, network interfaces, and other components. A high-speed interconnect such as PCIe, Ethernet, or optical interconnect (or a combination thereof) can be used.

[0110] The examples herein can be implemented in various types of computing and networking equipment, such as switches, routers, racks, and blade servers, such as those employed in data center and / or server farm environments. The servers used in data centers and server farms include array server configurations, such as rack-based servers or blade servers. These servers are interconnected for communication via various network provisions, such as partitioning a collection of servers into local area networks (LANs) with appropriate switching and routing facilities between the LANs to form a private intranet. For example, a cloud hosting facility can typically employ a large data center with multiple servers. A blade comprises a separate computing platform configured to perform server-type functions, i.e., a "server on a card." Thus, a blade comprises components common to conventional servers, including a main printed circuit board (motherboard) that provides internal wiring (e.g., a bus) for coupling appropriate integrated circuits (ICs) and other components mounted to the board.

[0111] Various examples can be implemented using hardware units, software units, or a combination of the two. In some examples, a hardware unit can include a device, component, processor, microprocessor, circuit, circuit element (e.g., transistor, resistor, capacitor, inductor, etc.), integrated circuit, ASIC, PLD, DSP, FPGA, storage unit, logic gate, register, semiconductor device, chip, microchip, chipset, etc. In some examples, a software unit can include a software component, program, application, computer program, application program, system program, machine program, operating system software, middleware, firmware, software module, routine, subroutine, function, method, process, software interface, API, instruction set, computing code, computer code, code segment, computer code segment, word, value, symbol, or any combination thereof. Determining whether to implement an example using a hardware unit and / or software unit can vary according to any number of factors, such as desired computing rate, power level, heat resistance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed, and other design or performance constraints, as desired for a given implementation. A processor may be a hardware state machine, digital control logic, a central processing unit, or one or more combinations of any hardware, firmware, and / or software elements.

[0112] Some examples may be implemented using or as an article of manufacture or at least one computer-readable medium. The computer-readable medium may include a non-transitory storage medium for storing logic. In some examples, the non-transitory storage medium may include one or more types of computer-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. In some examples, the logic may include various software units, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, APIs, instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof.

[0113] According to some examples, computer-readable media may include non-transitory storage media for storing or holding instructions that, when executed by a machine, computing device, or system, cause the machine, computing device, or system to perform methods and / or operations according to the described examples. Instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Instructions may be implemented according to a predefined computer language, manner, or syntax to instruct a machine, computing device, or system to perform a specific function. Instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0114] One or more aspects of at least one example may be implemented as representative instructions stored on at least one machine-readable medium representing various logic within the processor, which, when read by a machine, computing device, or system, causes the machine, computing device, or system to fabricate the logic to perform the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing facilities to load into fabrication machines that actually produce the logic or processor.

[0115] The appearance of the phrase "one example" or "example" does not necessarily refer to the same example or embodiment. Any aspect described herein can be combined with any other aspect or similar aspect described herein, regardless of whether these aspects are described with respect to the same figure or unit. The division, omission, or inclusion of block functions depicted in the figures does not necessarily infer that the hardware components, circuits, software, and / or units used to implement these functions will necessarily be divided, omitted, or included in the embodiments.

[0116] Some examples may be described using the expressions "coupled" and "connected" and their derivatives. For example, descriptions using the terms "connected" and / or "coupled" may indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" may also mean that two or more elements are not in direct contact, but still cooperate or interact.

[0117] The terms "first", "second", etc., do not indicate any order, quantity or importance in this document, but are used to distinguish one unit from another. The terms "one" and "an" in this document do not indicate a limit on quantity, but rather indicate that there is at least one of the items mentioned. The term "assertion" used herein with respect to a signal refers to a state of a signal, wherein the signal is active and can be achieved by applying any logic level (logic 0 or logic 1) to the signal. The term "following" or "after" can refer to immediately following or following some other event or events. According to alternative embodiments, other sequences of operations may also be performed. In addition, additional operations may be added or removed depending on the specific application. Any combination of changes may be used, and those of ordinary skill in the art who benefit from this disclosure will understand its many variations, modifications, and alternative embodiments.

[0118] Unless expressly stated otherwise, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in the context as generally used to indicate that an item, term, or the like can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z. Additionally, unless expressly stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" should also be understood to mean X, Y, Z, or any combination thereof, including "X, Y, and / or Z."

[0119] Illustrative examples of the devices, systems, and methods disclosed herein are provided below. Embodiments of the devices, systems, and methods may include any one or more of the examples described below, and any combination.

[0120] Example 1 includes one or more examples and includes an apparatus comprising a network interface device, the network interface device comprising: a host interface; direct memory access (DMA) circuitry; a network interface; and circuitry for: based on telemetry data received from at least one switch: selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data, wherein: the telemetry data is based on congestion information of a first queue associated with a first traffic class, the telemetry data is based on per-network interface device hop-level congestion status from at least one network interface device, the first queue shares bandwidth of an egress port with a second queue, the first traffic class is associated with packet traffic subject to congestion control based on utilization of the first queue, and the utilization of the first queue is based on an egress rate of the first queue and a send rate from the egress port; and causing transmission of packets to the selected next-hop network interface device.

[0121] Example 2 includes one or more examples wherein the congestion control based on utilization of the first queue is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

[0122] Example 3 includes one or more examples wherein selecting the next-hop network interface device from the plurality of network interface devices based on the received telemetry data includes selecting the next-hop network interface device from the plurality of network interface devices based on a utilization level or a packet drop level.

[0123] Example 4 includes one or more examples wherein selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data comprises adjusting a weight assigned to a queue or an egress port for providing packets to the selected next-hop network interface device.

[0124] Example 5 includes one or more examples and includes an apparatus comprising a switch circuit device comprising: at least one network interface and a circuit device for: based on sharing bandwidth of an egress port between a first business class and a second business class that is subject to congestion control based on utilization of a first queue associated with the first business class: reducing a packet sending rate of the first business class based on a first level of a first queue associated with the first business class, wherein the utilization of the first queue is based on an outflow rate of the first queue and a sending rate from the egress port, and increasing a packet sending rate of packets of the first business class based on a second level of the first queue and the second business class having available shared bandwidth, wherein the first level of the first queue is higher than the second level of the first queue.

[0125] Example 6 includes one or more examples wherein the congestion control is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

[0126] Example 7 includes one or more examples wherein the first level comprises a congestion state of the first queue.

[0127] Example 8 includes one or more examples wherein the second level comprises a non-congested state of the first queue.

[0128] Example 9 includes one or more examples wherein the circuit apparatus is configured to determine a utilization of the first queue based on a depth of the first queue.

[0129] Example 10 includes one or more examples wherein the circuit apparatus is configured to determine a utilization of the first queue based on a percentage of a line rate utilized by packets sent from the first queue.

[0130] Example 11 includes one or more examples and includes a method comprising: at a switch: based on sharing bandwidth of an egress port between a first business class and a second business class that are subject to congestion control based on utilization of a first queue: based on a first level of a first queue associated with the first business class, reducing a packet sending rate of the first business class, wherein the utilization of the first queue is based on an outflow rate of the first queue and a sending rate from the egress port, and based on a second level of the first queue and the second business class having available shared bandwidth, increasing a packet sending rate of packets of the first business class, wherein the first level of the first queue is higher than the second level of the first queue.

[0131] Example 12 includes one or more examples wherein the congestion control is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

[0132] Example 13 includes one or more examples wherein the first level comprises a congestion state of the first queue.

[0133] Example 14 includes one or more examples wherein the second level comprises a non-congested state of the first queue.

[0134] Example 15 includes one or more examples and includes determining a utilization of the first queue based on a depth of the first queue.

[0135] Example 16 includes one or more examples and includes determining utilization of the first queue based on a percentage of the line rate utilized by packets sent from the first queue.

[0136] Example 17 includes one or more examples and includes a method comprising: at a switch: based on telemetry data received from at least one switch: selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data, wherein: the telemetry data is based on congestion information of a first queue associated with a first service class, the telemetry data is based on a per-network interface device hop congestion status from at least one network interface device, the first queue shares bandwidth of an egress port with a second queue, the first service class is associated with packet services that are subject to congestion control based on utilization of the first queue, and the utilization of the first queue is based on an egress rate of the first queue and a send rate from the egress port; and causing packets to be transmitted to the selected next-hop network interface device.

[0137] Example 18 includes one or more examples wherein the congestion control based on utilization of the first queue is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

[0138] Example 19 includes one or more examples wherein selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data comprises adjusting a weight assigned to a path of a plurality of paths to a destination receiver.

[0139] Example 20 includes one or more examples and includes adjusting a sending rate of packets of the first traffic class based on telemetry data.

Claims

1. A device comprising: Network interface devices, including: Host interface; Direct memory access (DMA) circuitry; Network interface; and Circuit arrangement for: Based on telemetry data received from at least one switch: selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data, wherein: The telemetry data is based on congestion information of a first queue associated with a first traffic class, The telemetry data is based on a per-network interface device hop congestion status from at least one network interface device, The first queue and the second queue share the bandwidth of the egress port, The first traffic class is associated with packet traffic that is subject to congestion control based on utilization of the first queue, and The utilization of the first queue is based on an outflow rate of the first queue and a sending rate from the egress port; and The packet is caused to be transmitted to the selected next-hop network interface device.

2. The device according to claim 1, wherein The congestion control based on utilization of the first queue is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

3. The device according to claim 1, wherein Selecting a next-hop network interface device from the plurality of network interface devices based on the received telemetry data includes selecting the next-hop network interface device from the plurality of network interface devices based on a utilization level or a packet drop level.

4. The device according to any one of claims 1 to 3, wherein: Selecting a next-hop network interface device from the plurality of network interface devices based on the received telemetry data includes adjusting a weight assigned to a queue or egress port for providing packets to the selected next-hop network interface device.

5. A device comprising: A switch circuit device, comprising: At least one network interface, and Circuit arrangement for: Based on sharing the bandwidth of the egress port between the first traffic class and a second traffic class subject to congestion control based on utilization of a first queue associated with the first traffic class: Based on a first level of the first queue associated with the first service class, a packet sending rate of the first service class is reduced, wherein: The utilization of the first queue is based on the outflow rate of the first queue and the sending rate from the egress port, and Based on the second level of the first queue and the available shared bandwidth of the second traffic class, the packet sending rate of packets of the first traffic class is increased, wherein the first level of the first queue is higher than the second level of the first queue.

6. The device according to claim 5, wherein The congestion control is based on one or more of the following: High Precision Congestion Control (HPCC) or Poseidon.

7. The device according to claim 5, wherein The first level comprises a congestion status of the first queue.

8. The device according to claim 5, wherein The second level comprises a non-congested state of the first queue.

9. The device according to any one of claims 5 to 8, wherein: The circuitry is configured to determine a utilization of the first queue based on a depth of the first queue.

10. The device according to any one of claims 5 to 8, wherein The circuitry is configured to determine a utilization of the first queue based on a percentage of a line rate utilized by packets sent from the first queue.

11. A method comprising: At the switch: Based on sharing the bandwidth of the egress port between a first traffic class and a second traffic class subject to congestion control based on utilization of the first queue: causing a reduction in a packet sending rate for the first traffic class based on a first level of the first queue associated with the first traffic class, wherein utilization of the first queue is based on an egress rate of the first queue and a sending rate from the egress port, and Based on the second level of the first queue and the available shared bandwidth of the second traffic class, the packet sending rate of packets of the first traffic class is increased, wherein the first level of the first queue is higher than the second level of the first queue.

12. The method according to claim 11, wherein The first level comprises a congestion status of the first queue.

13. The method according to claim 11, wherein The second level comprises a non-congested state of the first queue.

14. The method according to claim 11, comprising: A utilization of the first queue is determined based on the depth of the first queue.

15. The method according to claim 11, comprising: A utilization of the first queue is determined based on a percentage of the line rate utilized by packets sent from the first queue.

16. The method according to any one of claims 11 to 15, wherein: The congestion control is based on one or more of the following: High Precision Congestion Control (HPCC) or Poseidon.

17. A method comprising: At the switch: Based on telemetry data received from at least one switch: selecting a next-hop network interface device from a plurality of network interface devices based on the received telemetry data, wherein: The telemetry data is based on congestion information of a first queue associated with a first traffic class, The telemetry data is based on a per-network interface device hop congestion status from at least one network interface device, The first queue and the second queue share the bandwidth of the egress port, The first traffic class is associated with packet traffic that is subject to congestion control based on utilization of the first queue, and The utilization of the first queue is based on an outflow rate of the first queue and a sending rate from the egress port; and The packet is transmitted to the selected next-hop network interface device.

18. The method according to claim 17, wherein The congestion control based on utilization of the first queue is based on one or more of: High Precision Congestion Control (HPCC) or Poseidon.

19. The method according to claim 17, wherein Selecting a next-hop network interface device from the plurality of network interface devices based on the received telemetry data includes adjusting weights assigned to paths of the plurality of paths to the destination receiver.

20. The method according to any one of claims 17 to 19, comprising: A sending rate of packets of the first traffic class is adjusted based on the telemetry data.

Images