High performance connection scheduler

Abstract

Aspects of the disclosure relate to high performance connection schedulers, and in particular, to a high performance connection scheduler for reliable transport protocols in data center networking. The connection scheduler can handle enqueue events, dequeue events, and update events. The connection scheduler can include a connection queue, a dispatch queue, and a quality of service arbiter to support dispatching a large number of connections at a high rate.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a high performance connection scheduler. BACKGROUND

[0002] Reliable transport protocols can support a large number of connections, which can include both initiated requests and received requests. The large number of connections can be subject to ordering requirements and gating requirements. Determining how to queue the large number of connections while considering the ordering requirements and gating requirements can reduce performance and increase latency for receiving data packets, transmitting data packets, and / or updating data packets. SUMMARY

[0003] Aspects of the present disclosure relate to a high performance connection scheduler for reliable transport protocols in data center networking. The connection scheduler can include connection queues, scheduling queues, and quality of service arbiters to support scheduling a large number of connections at a high rate. The connection scheduler can also include eligibility checks and solicitation checks when scheduling connections. The connection scheduler can handle enqueue events for receiving data packets, dequeue events for transmitting data packets, and update events for modifying data packets.

[0004] Aspects of the present disclosure provide a method for scheduling connections related to data packets. The method includes performing, with one or more processors, an eligibility check for a connection, the eligibility check including determining, based on a transaction type of the connection, that the connection is within a congestion window; in response to the connection passing the eligibility check, queuing, with the one or more processors, the connection in a scheduling queue of a plurality of scheduling queues based on the transaction type; selecting, with the one or more processors, the connection from the scheduling queue based on a quality of service (QoS) arbiter decision; and queuing, with the one or more processors, the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

[0005] In an example, the scheduling queues and the connection queues each include a plurality of linked lists. In another example, the plurality of linked lists of the scheduling queues and the plurality of linked lists of the connection queues each include linked lists for at least one of: push requests, un-ordered push requests, push authorizations, push data, or pull data transaction types.

[0006] In yet another example, the congestion window includes a network congestion window and an end node congestion window. In yet another example, passing the eligibility check includes determining that a number of outstanding requests is less than the end node congestion window. In yet another example, passing the eligibility check includes determining that a packet sequence number is less than a base sequence number plus the network congestion window.

[0007] In yet another example, the method further includes performing, with the one or more processors, a solicitation check on the connection, the solicitation check including determining an in-flight count based on the connection being within a solicitation window. In yet another example, the solicitation check further includes determining that a rate counter is greater than 0.

[0008] In yet another example, selecting the connection from the scheduling queue further includes selecting the scheduling queue based on at least one of a priority or a weight of the scheduling queue. In yet another example, selecting the connection from the scheduling queue is based on a weight of the connection.

[0009] Another aspect of the disclosure provides a system comprising: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for scheduling a connection related to a data packet. The operations include: performing an eligibility check on the connection, the eligibility check including determining that the connection is within a congestion window based on a transaction type of the connection; in response to the connection passing the eligibility check, queuing the connection in a scheduling queue of a plurality of scheduling queues based on the transaction type; selecting the connection from the scheduling queue based on a quality of service (QoS) arbiter decision; and queuing the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

[0010] In an example, the scheduling queue and the connection queue each include a plurality of linked lists. In another example, the congestion window includes a network congestion window and an end node congestion window; passing the eligibility check includes determining that a number of outstanding requests is less than the end node congestion window; and passing the eligibility check includes determining that a packet sequence number is less than a base sequence number plus the network congestion window. In yet another example, the operations further include performing a solicitation check on the connection, the solicitation check including determining an in-flight count based on the connection being within a solicitation window. In yet another example, selecting the connection from the scheduling queue is based on a weight of the connection and further includes selecting the scheduling queue based on at least one of a priority or a weight of the scheduling queue.

[0011] Yet another aspect of the disclosure provides a non-transitory computer- readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for scheduling a connection related to a data packet. The operations include: performing an eligibility check on the connection, the eligibility check including determining that the connection is within a congestion window based on a transaction type of the connection; in response to the connection passing the eligibility check, queuing the connection in a scheduling queue of a plurality of scheduling queues based on the transaction type; selecting the connection from the scheduling queue based on a quality of service (QoS) arbiter decision; and queuing the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

[0012] In an example, the scheduling queues and the connection queues each include a plurality of linked lists. In another example, the congestion window includes a network congestion window and an end node congestion window; the passing the eligibility check includes determining that the number of outstanding requests is less than the end node congestion window; and the passing the eligibility check includes determining that the packet sequence number is less than the base sequence number plus the network congestion window. In yet another example, the operations further include performing a solicitation check for the connection, the solicitation check including determining an in-flight count based on the connection being within a solicitation window. In still another example, selecting the connection from the scheduling queue is based on a weight of the connection, and further includes selecting the scheduling queue based on at least one of a priority or a weight of the scheduling queue. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A block diagram of an example network environment with network devices is depicted in accordance with aspects of the present disclosure.

[0014] Figure 2 A block diagram of an example connection scheduler for scheduling connections is depicted in accordance with aspects of the present disclosure.

[0015] Figure 3 A block diagram of an example scheduling queue is depicted in accordance with aspects of the present disclosure.

[0016] Figure 4 A block diagram of an example quality of service arbiter for multiple scheduling queues is depicted in accordance with aspects of the present disclosure.

[0017] Figure 5 A flow diagram of an example process for scheduling connections is depicted in accordance with aspects of the present disclosure.

[0018] Figure 6 An example computing system is depicted in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of a high-performance connection scheduler for reliable transport protocols in data center networking are generally disclosed herein. The connection scheduler can support scheduling a large number of connections at a high rate.

[0020] The connection scheduler can handle three types of events: enqueue events, dequeue events, and update events. Enqueue events can correspond to the connection scheduler receiving a new data packet to be queued for transmission. Dequeue events can correspond to a queued data packet being removed from a queue to be scheduled for transmission. Update events can correspond to the connection scheduler receiving an update for a queued data packet.

[0021] For an enqueue event, the connection scheduler can receive a data packet. The scheduler can add a pointer to the data packet to a corresponding packet type queue. The head and tail pointers of the queue can be updated. As an example, the data packet can correspond to an upper layer protocol (ULP) request to send a data packet on the network or a received packet such as a push grant. If the data packet passes the scheduling eligibility check, the connection scheduler can push the connection ID of the data packet to the corresponding scheduling queue or can set the corresponding bit in the scheduling bitmap.

[0022] For a dequeue event, the quality of service (QoS) arbiter of the connection scheduler can select a connection queue to be scheduled. The connection scheduler can pop a data packet off the head of the selected connection queue and can read the metadata of the popped data packet from the state of the popped data packet. The connection scheduler can send the popped data packet for transmission such as through a packet transmission interface. If the next data packet in the selected connection queue passes the scheduling eligibility check, the connection scheduler can push the connection ID of the data packet to the corresponding scheduling queue or can set the corresponding bit in the scheduling bitmap.

[0023] For an update event, the connection scheduler can receive an update event. The connection scheduler can update the connection state of the corresponding connection based on the update event. As an example, these updates can include an acknowledgment packet of the connection, a congestion window (Cwnd) update, or a basic sequence number (BSN) update. If a new data packet passes the scheduling eligibility check after the connection state update, the connection scheduler can push the new data packet to the corresponding scheduling queue or can set the corresponding bit in the scheduling bitmap. The scheduling queue / bitmap can support multiple pushes / bits from a single event as the update event can result in multiple data packets eligible for scheduling.

[0024] The connection scheduler includes connection queues that can use linked list data structures. The connection queues can be structured as multiple linked lists per connection to meet ordering requirements while preventing line head blocking due to data packet gating requirements. The total number of connections in the connection queues can correspond to the number of supported active connections multiplied by the number of linked lists. Maintaining separate lists for different transaction types can allow for improved performance.

[0025] The connection scheduler also includes eligibility checks. In response to network fabric congestion and target resources being full, network congestion control can use separate definitions of network congestion window (fCwnd) and end node congestion window (nCwnd). To satisfy ordering and packet gating requirements, the connection scheduler can include multiple eligibility checks for each data packet type before a data packet is pushed into the corresponding arbitration snapshot. For push requests and pull requests, the eligibility check can include checking fCwnd and nCwnd. For push grants, the eligibility check can include checking fCwnd. For push unsolicited data, the eligibility check can include checking fCwnd and nCwnd and checking RSN for pull requests. For push solicited data, the eligibility check can include checking fCwnd and checking if a push grant was received. For pull responses, the eligibility check can include checking fCwnd.

[0026] The connection scheduler can also include solicitation checks for push request, pull request, and push grant packets to be transmitted. The request check can include a solicitation window check to limit unsolicited solicitation buffer space.

[0027] The solicitation check can also include a solicitation rate limiter to control receiver bandwidth for incoming solicited pull data packets and / or push data packets and to control transmission bandwidth for pushed solicited data.

[0028] The solicitation rate limiter can include a rate counter that increments at a configurable rate and also a configurable maximum value. The maximum value can limit the number of serviced packets in a burst. Data packets can only be scheduled when the rate counter is greater than zero.

[0029] The connection scheduler can also include a scheduling queue that can use a linked list data structure. The number of scheduling queues can be determined by the host device and the mapping of traffic classes to queues. The connection scheduler can push an entry into a scheduling queue when a new packet is queued into the scheduler, the packet passes eligibility checks, and the connection ID has not appeared in the corresponding scheduling queue. The connection scheduler can also push an entry into a scheduling queue when a packet is dequeued and the packet passes eligibility checks. The connection scheduler can also push an entry into a scheduling queue when an update event is processed and one or more packets are eligible for scheduling.

[0030] The solicitation window check can be performed after the scheduling queue such that only eligible packet type queues can participate in the arbitration process performed by the QoS arbiter. The QoS arbiter can pop an entry from a queue that wins arbitration. The pop can occur once per cycle.

[0031] The QoS arbiter can include a dispatch queue (SQ) selector configured to select the next dispatch queue to be serviced. The SQ selector can be configured to select the next SQ based on priority or weight of the SQs, and each SQ can be rate limited. SQs assigned to higher priority can be serviced before SQs assigned to lower priority. SQs not assigned a priority can participate in a weighted round robin (WRR) scheduler.

[0032] The QoS arbiter can include multiple packet selectors for each SQ that use a similar weighted round robin scheme as the SQ selector for selecting packets from the SQ for transmission.

[0033] The QoS arbiter can include a rate limiter per SQ. The rate limiter can include a counter that increments at a configurable rate. When the counter is greater than or equal to the size of a data packet, the data packet is allowed to be selected (via the priority selector or weight round robin selector).

[0034] Figure 1 A block diagram of an example network environment 100 having a network device 110 is depicted. The network environment 100 includes a network 120 of interconnected network nodes 125. The network nodes 125 participate in the network 120 as data sources, data destinations or data sinks, and / or intermediate nodes such as switches, routers, and / or gateways, that propagate data from sources to destinations over the network 120. The network 120 includes the network device 110 having links 127 to various other participating network nodes 125.

[0035] The network 120 can facilitate interaction between participating devices. Example networks include the Internet, intranet, extranet, or any other local- or wide-area network. The network 120 can be constructed of multiple connected sub-networks or autonomous networks. The network 120 can be a local area network (LAN), such as a company Intranet, a metropolitan area network (MAN), a wide area network (WAN), an inter-network of networks (such as the Internet), or a peer-to-peer network. Any type and / or form of data network and / or communication network can be used for the network 120. The network can be public, private, or a combination of public and private. Generally, the network 120 can be used to communicate information among computing devices; for example, the network nodes 125 and the network device 110.

[0036] The network device 110 can be a host device or a server executing one or more virtual machines 130 on one or more processors of the network device 110. The network device 110 can include a network interface driver 140, a memory 150, a network interface card 160, a real operating system (OS) 170, a hypervisor 180, and virtual machines 130. The network device 110 can communicate with other network devices over a network 120. The network device 110 and other network devices can create or participate in one or more virtual networks to facilitate communication between virtual machines 130. The one or more virtual networks can overlay the physical network that makes up the network 120. Data packets can be routed within one of the virtual networks that overlays the physical network using a data packet encapsulation system. The network device 110 and network nodes 125 can route data packets according to virtual internet protocol addresses (VIPs) and MAC addresses. In some examples, the network device 110 can host one or more virtual machines 130 that transmit data packets or receive data packets. In other examples, the network device 110 can be an intermediate node of a path between two virtual machines executing on two separate network devices. Thus, the network device 110 can act as a source, a destination, or an intermediate switch in one or more virtual networks.

[0037] The hypervisor 180 can manage the operation of the virtual machines 130, including starting and terminating the virtual machines 130, managing the allocation of memory 150 to the virtual machines 130, and live migrating the virtual machines 130 to other network devices.

[0038] The memory 150 can store data and / or computer-executable instructions related to the operation and use of the network interface driver 140. As an example, the memory 150 can include high bandwidth memory (HBM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), static random access memory (SRAM), and / or flash memory. The memory 150 can store computer-executable instructions of a transport protocol module 175 (such as a transmission control protocol (TCP) module or a TCP layer of a network stack) to be executed on a processor of the network device 110. While the transport protocol module 175 is shown in the network interface card 160, it can also be included in the real operating system. The functionality that occurs within the TCP layer of the network stack can additionally or alternatively be performed in another network protocol module within the transport layer, the network layer, or a combination transport layer / network layer of the network stack. For example, the functionality can be implemented in a user datagram protocol (UDP) module, a reliable datagram protocol (RDP) module, a reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module. The network layer, the transport layer, or the combination transport layer / network layer can generally be referred to as a packet layer of the network stack.

[0039] The network interface driver 140 can include a network interface driver software module running on the real OS 170. As an example, the network interface driver 140 can be a set of computer executable instructions stored in the memory 150 that, when executed by the processor, help facilitate network communications. As another example, the network interface driver 140 can be implemented as logic in a hardware processor or other integrated circuit, or as a combination of hardware and software logic. The network interface driver 140 can communicate with one of the software virtual machines 130 directly or via the guest OS and / or hypervisor 180 of the virtual machine 130.

[0040] The network interface driver 140 can be included within a layer of a network stack of the real OS 170 of the network device 110. In an example, the network interface driver 140 can be included within a data link layer of the network stack and can communicate with software modules or applications included in an application layer of the network stack. In another example, the network interface driver 140 can be included within a link layer of the network stack and can communicate with a TCP / IP module included in an internet / transport layer of the network stack. In other examples, functionality appearing within a network stack layer can additionally or alternatively be configured to receive packets from another network or transport layer protocol module, such as a UDP module, RDP module, RUDP module, or DCCP module. Although not shown, the network interface driver 140 can be included as part of the network interface card 160.

[0041] The network interface card 160 can be configured to transmit data packets over a network protocol, such as a layer 2 protocol. The network interface card 160 can include a packet forwarder 162, a connection scheduler 164, and a memory 166.

[0042] The memory 166 can store data and / or computer executable instructions related to the operation and use of the network interface card 160. For example, the memory 166 can store connection queues and scheduling queues as well as instructions related to eligibility checks and solicitation checks. As an example, the memory 166 can include SRAM, HBM, DRAM, SDRAM, and / or flash memory.

[0043] The packet forwarder 162 can be implemented in an application-specific integrated circuit (ASIC) or as software executing on a microprocessor. The packet forwarder 162 can receive data packets from the virtual machines 130 for transmission and route them to one or more destination virtual machines executing on a remote network device. The packet forwarder 162 can also perform functions that allow the network interface card 160 to act as a destination for data packets received over the links 127. For example, the packet forwarder 162 can receive a data packet over one of the links 127, decapsulate the data packet, and forward the data packet to one or more destination virtual machines 130 executing on the network device 110. The packet forwarder 162 can transmit and receive data packets based on connection entries stored in the cache 166. Each transmission or reception of a data packet can correspond to a deallocation or allocation, respectively, of memory 150.

[0044] The connection scheduler 164 can be implemented in an application-specific integrated circuit (ASIC) or as software executing on a microprocessor. The connection scheduler 164 can support scheduling a large number of connections at a high rate through connection queues, a scheduling queue, and a quality of service (QoS) arbiter. The connection scheduler 164 can also include eligibility checks and solicitation checks when scheduling connections. The connection scheduler 164 can handle enqueue events for receiving data packets, dequeue events for transmitting data packets, and update events for modifying data packets.

[0045] Figure 2 A block diagram of an example connection scheduler 200 for scheduling connections is depicted. The connection scheduler 200 can correspond to the connection scheduler 164 as shown in Figure 1

[0046] The connection scheduler 200 can include a least recently used arbiter 202 for receiving enqueue events, update events, and dequeue events. An enqueue event can correspond to the connection scheduler receiving a new data packet to be queued. A dequeue event can correspond to a queued data packet being removed from a queue to be scheduled for transmission. An update event can correspond to the connection scheduler receiving an update for a queued data packet.

[0047] The least recently used arbiter 202 can determine which received event the connection scheduler 200 should next process. For example, the least recently used arbiter 202 can determine which received event was received first or has been waiting the longest to be processed based on a timestamp in the metadata of the event. Based on the decision of the least recently used arbiter 202, the event is sent to one or more connection queues 204 or eligibility check modules 206.

[0048] ​The connection scheduler 200 can include a connection queue 204 structured as a plurality of linked lists per connection. Structuring the connection queue 204 as a linked list per connection can satisfy ordering requirements while preventing line head blocking due to data packet gating requirements. For ordered connections, the ordering requirements can include ensuring relative ordering of request sequence numbers (RSNs) and packet sequence numbers (PSNs) within each sliding window. For unordered connections, the ordering requirements can include ensuring relative ordering of PSNs within each sliding window. The gating requirements can include congestion control and solicitation windows depending on the data packet type.

[0049] The total number of connection queues 204 can correspond to the number of active connections supported multiplied by the number of linked lists. For example, if 5 linked lists support 64k active connections, the total number of connection queues can correspond to 320k connections.

[0050] Example linked lists per connection based on transaction type can include an ordered push request list, an unordered push request list, a push grant list, a push data list, and a pull data list. The ordered push request list can include solicited push requests, dummy unsolicited requests, pull requests for ordered connections, and pull requests for unordered connections. The unordered push request list can include solicited push requests for unordered connections. The push grant list can include push grants for ordered connections and unordered connections. The push data list can include solicited and unsolicited push data for ordered connections and unordered connections. The pull data list can include pull data for ordered connections and unordered connections.

[0051] Maintaining linked lists for different transaction types can improve performance. The connection scheduler 200 can maintain ordering independently within initiator transactions or target transactions, without maintaining combined ordering for all initiator transactions and target transactions. Initiator transactions can correspond to transactions related to sending requests, such as write requests or read requests. Target transactions can correspond to transactions related to receiving requests, such as receiving write or read requests and committing write or read requests. Initiator pull requests and target push grants can be blocked independently of initiator pushes. Further, maintaining linked lists allows for separation between initiator and target traffic, regardless of the number of active connections. Arbitrating first between transaction types allows the connection scheduler 200 to respond to remote requesters with improved performance, whose resources can be reserved due to pending transactions.

[0052] Based on the connection state and packet state stored in the per-connection state memory 208 and the per-packet state memory 210, respectively, the connection scheduler 200 is able to determine which packet in the connection queue 204 to transmit. For example, the connection state stored in the per-connection state memory 208 can include a network congestion window and an end node congestion window, and the packet state stored in the per-packet state memory 210 can include a request sequence number.

[0053] The connection scheduler 200 can also include eligibility checks performed via the eligibility check module 206. The eligibility check module 206 can perform eligibility checks related to network congestion control, such as ordering and packet gating requirements.

[0054] Network congestion control can use separate definitions of a network congestion window (fCwnd) and an end node congestion window (nCwnd). The network congestion window can be defined as the number of outstanding transmission packets that the connection scheduler can have in each sliding window. The end node congestion window can be defined as the number of outstanding ULP requests that the connection scheduler 200 as an initiator can have in each sliding window.

[0055] To satisfy the ordering and packet gating requirements, the eligibility check module 206 can include multiple eligibility checks for each packet type before a data packet is pushed to the corresponding scheduling queue 212. For push requests and pull requests, the eligibility check can include checking fCwnd and nCwnd. For push grants, the eligibility check can include checking fCwnd. For push unsolicited data, the eligibility check can include checking fCwnd and nCwnd and checking RSN for pull requests. For push solicited data, the eligibility check can include checking fCwnd and checking whether a push grant is received. For pull responses, the eligibility check can include checking fCwnd. Checking fCwnd and nCwnd can include comparing the number of outstanding data packets in the corresponding request or data window to the fCwnd / nCwnd value, where the data packet can be eligible for scheduling when the number of outstanding data packets is less than the fCwnd / nCwnd value. Checking RSN for pull requests can include comparing the RSN of the push unsolicited data packet to the RSN of the line head pull request packet, where the push unsolicited data can be eligible for scheduling when the RSN of the push unsolicited data packet is less than the RSN of the line head pull request packet. The RSN of both push unsolicited data and pull requests can be stored in the scheduler state. If a connection does not pass the eligibility check, the data packet for that connection will not be selected for scheduling and waits to become eligible.

[0056] Data packet transmission and retransmission can be based on the following eligibility checks. For pull and push requests in a request sliding window, the PSN should be less than the BSN in the request sliding window plus the network congestion window, and the number of incomplete requests should be less than the end node congestion window. For push requests for unsolicited data in a data sliding window, the PSN should be less than the BSN in the data sliding window plus the network congestion window, and the number of incomplete requests should be less than the end node congestion window. For pull requests, push authorizations, and push requests for data in a data sliding window, the PSN should be less than the BSN in the data sliding window plus the network congestion window.

[0057] For initial transmissions, the eligibility check module 206 can maintain multiple incomplete request status variables in each sliding window. The count of incomplete requests can be incremented when the connection scheduler 200 initially schedules a pull request, push request, or push of unsolicited data, and can be decremented when the corresponding pull request is acknowledged or the corresponding push data is acknowledged. For retransmissions, the eligibility check module 206 can dynamically determine the number of incomplete requests based on the retransmission packet type. For example, the count of incomplete requests can only be incremented when the connection scheduler 200 schedules a push of unsolicited data.

[0058] Based on the successful qualification check, the connection can be placed in the scheduling queue 212 of the connection scheduler 200. The scheduling queue 212 can be constructed as multiple linked lists. Figure 3 A block diagram of an example scheduling queue 300 is depicted. Scheduling queue 300 can correspond to, for example... Figure 2 The depicted scheduling queue 212 is capable of locating multiple connections 302 within the scheduling queue 300 based on a linked list data structure. The Quality of Service (QoS) arbitrator 304 is capable of determining which scheduling queue 300 to select and which connection within the selected scheduling queue 300 to dequeue the data packet, as will be described further below.

[0059] The number of scheduling queues 300 can be determined through a mapping of host devices and service categories to queues. This mapping can be a Compressed Sparse Line (CSR) mapping. The host devices can correspond to servers connected to the network interface cards used for scheduling connection 302, such as... Figure 1 The network interface card 160 is depicted. Service categories can be classified according to connection 302, for example, based on port number or protocol. As an example, the scheduling queue 300 can support 16 mappings and 5 packet queues for various packet types in each queue (such as pull requests, solicited push requests, push authorizations, solicited and unsolicited push data, and pull data). Push and pull requests can be separated into different scheduling queues 300 to allow independent solicitation rate limiters to push or pull requests and avoid line head congestion issues.

[0060] Referring back to Figure 2 , the connection scheduler 200 is able to push entries into the scheduling queues 212 for enqueue events, dequeue events, and update events. For enqueue events, the connection scheduler 200 is able to push entries into the scheduling queues 212 when a new packet passes the eligibility check and the connection ID of the new data packet does not already exist in the corresponding scheduling queue 212. For dequeue events, the connection scheduler 200 is able to push entries into the scheduling queues 212 when a packet to be transmitted passes the eligibility check. For update events, the connection scheduler 200 is able to push entries into the scheduling queues 212 when one or more packets become eligible for scheduling after processing the update event.

[0061] The connection scheduler 200 is also able to include solicitation checks performed via the solicitation check module 214. The solicitation check module 214 is able to perform solicitation checks on push requests, pull requests, and push grant packets after the scheduling queues 212 such that only eligible packet type queues are able to participate in the arbitration process performed by the QoS arbiter 216.

[0062] The solicitation check module 214 can perform a solicitation window check to limit outstanding solicitation buffer space. The solicitation check module 214 is able to track the total bytes solicited (referred to as in-flight count). For each new pull request and push grant, the solicitation check module 214 is able to add the bytes associated with the new request to the current total in-flight count to calculate an increased in-flight count. The solicitation check module 214 is able to compare the increased in-flight count to a solicitation window multiplied by a scaling factor a. The scaling factor a is able to expand or shrink the window. If the increased in-flight count is less than the scaled solicitation window, the new request is able to be executed and the solicitation check module 214 is able to set the in-flight count to the increased value. If the packet type does not pass the solicitation check, the packet type is not selected for scheduling and must wait until it is eligible.

[0063] The solicitation check module 214 is also able to include a solicitation rate limiter to control receiver bandwidth and / or transmission bandwidth. The solicitation rate limiter is able to control receiver bandwidth for incoming solicited pull data packets and / or push data packets for pull requests and push grants, respectively. The solicitation rate limiter is able to control transmission bandwidth for push solicited data by adjusting push requests based on the size of the push solicited data packets for the push requests.

[0064] The solicitation rate limiter can include a rate counter that increments at a configurable rate and also a configurable maximum value. The rate counter can start at 0 and increment at the configured rate. Once the rate counter reaches the configured maximum value, the rate counter can stop incrementing. The maximum value can limit the number of packets served in a burst. Data packets can only be scheduled when the rate counter is greater than zero. When a pull request or push grant is scheduled by the connection scheduler 200, the number of solicited bytes is subtracted from the rate counter. The counter can be negative to track extra bytes that have been solicited.

[0065] After a check by solicitation, the connection scheduler 200 can also include a QoS arbiter 216 to determine which scheduling queue to select and which connection within the selected scheduling queue to select for an enqueue event. The QoS arbiter 216 can pop the entry that wins arbitration for use in enqueuing from a scheduling queue. The popping can occur once per cycle. Figure 4 A block diagram of an example QoS arbiter 400 for multiple scheduling queues 402 is depicted. The QoS arbiter 400 can correspond to the QoS arbiter 216 as Figure 2 depicted.

[0066] The QoS arbiter 400 can include multiple packet selectors 404 for each scheduling queue 402 configured to select packets from each scheduling queue 402 for transmission using a weighted round robin scheme.

[0067] For the weighted round robin scheme, each packet selector 404 can have a credit counter that is initialized to a configurable weight. Each time a packet is selected, the packet size is subtracted from the credit counter. If the credit counter is greater than or equal to 0, the packet can be eligible for selection. Selection can be made among all packets with sufficient credits on a round robin basis.

[0068] If all packets have negative credit counters, one packet is selected on a round robin basis. When a packet is selected and its credit counter is negative, the credit counters of all packet selectors 404 are reinitialized. If the credit counter is negative at reinitialization, the configurable weight corresponding to that credit counter is added to its current value. If the credit counter is positive at reinitialization, the programmed weight is added to the counter again. A maximum credit value can be provided to limit the amount of credit that a packet selector can accumulate.

[0069] Once a packet is selected from each of the scheduling queues 402, the QoS arbiter 400 can also include a plurality of scheduling queue selectors 406 configured to select the next scheduling queue 402 to be serviced. The scheduling queue selectors 406 can be configured to select the next scheduling queue 402 based on the priority and / or weight of the scheduling queue 402. Each scheduling queue 402 can also be rated as limited.

[0070] The scheduling queue selectors 406 can be configured to select a scheduling queue 402 assigned to a higher priority before a scheduling queue 402 assigned to a lower priority. For example, 8 priorities can be available, with one or more scheduling queues assignable to each priority. The highest priority can be 0 and the lowest priority can be 7. Thus, a scheduling queue assigned to priority 3 can be serviced before a scheduling queue assigned to priority 5.

[0071] The scheduling queue selectors 406 can use a weighted round robin scheme to schedule queues not assigned a priority or queues assigned the same priority. Each scheduling queue 402 can have a credit counter initialized to a configurable weight. Each time a scheduling queue 402 is selected, the packet size is subtracted from the credit counter. If the credit counter is greater than or equal to 0, the scheduling queue 402 can be eligible to be selected. Selection can be made among all scheduling queues 402 with sufficient credits on a round robin basis.

[0072] If all scheduling queues with pending packets have negative credit counters, a packet is selected on a round robin basis. When a scheduling queue 402 is selected and its credit counter is negative, the credit counters of all scheduling queues 402 are reinitialized. If the credit counter is negative at reinitialization, the configurable weight corresponding to the credit counter is added to its current value. For example, if the credit counter is -5 at reinitialization, the credit counter is set to its configurable weight of -5. If the credit counter is positive at reinitialization, the programmed weight is added to the counter again. A maximum credit value can be provided to limit the amount of credit a scheduling queue 402 can accumulate.

[0073] Once a scheduling queue 402 is selected from each of the scheduling queue selectors 406, the QoS arbiter 400 can also include a host selector 408 configured to select the next host device to be serviced using a weighted round robin scheme.

[0074] For a weighted round robin scheme, each host selector 408 can have a credit counter that is initialized to a configurable weight. Each time a host device is selected, the credit counter is decremented by the size of the packet. If the credit counter is greater than or equal to 0, the host device is eligible to be selected. Selection can be made among all host devices with sufficient credits on a round robin basis.

[0075] If all host devices have negative credit counters, one host device is selected on a round robin basis. When a packet is selected and its credit counter is negative, the credit counters of all host selectors 408 are reinitialized. If the credit counter is negative at reinitialization, the configurable weight corresponding to that credit counter is added to its current value. If the credit counter is positive at reinitialization, the programmed weight is added to the counter again. A maximum credit value can be provided to limit the amount of credit that a host selector 408 can accumulate.

[0076] Once a host device is selected, the selected packet corresponding to the selected dispatch queue and the selected host device can be dequeued for transmission.

[0077] The QoS arbiter 40 can include a rate limiter 410 for each host device. Each rate limiter 410 can include a counter that is incremented at a configurable rate. The increment period should be the same for all dispatch queues 402, but the increment amount can be configurable for each dispatch queue 402. Each dispatch queue 402 can also include a configurable maximum value.

[0078] The counter can start at 0 and can be incremented at a configurable rate. When the counter is greater than or equal to the size of a packet from the dispatch queue 402, the dispatch queue 402 is allowed to be selected. When a dispatch queue 402 is selected, the credit is subtracted from the counter. Once the counter reaches the configurable maximum value, the counter can stop incrementing. The maximum value can limit the number of packets that can be serviced in a burst. The rate limiter 410 can be enabled or disabled for each dispatch queue 402. The rate limit can be a hard limit such that once a dispatch queue 402 exceeds the rate credit, no packet will be allowed to be transmitted even if there are no other dispatch queues 402 available to be serviced.

[0079] Figure 5 A flow diagram depicting an example process 500 for scheduling connections involving transmission, reception, and / or modification of data packets is depicted. The example process 500 can be executed on a system of one or more processors in one or more locations, such as Figure 2 The depicted connection scheduler 200.

[0080] As represented by block 510, the connection scheduler 200 can perform eligibility checks on the connection. The eligibility checks can include determining that the connection is within a congestion window based on the transaction type of the connection to satisfy ordering and packet gating requirements.

[0081] For example, when the transaction type is a push request or a pull request, the eligibility checks can include checking the network congestion window and the end node congestion window. When the transaction type is a push grant, the eligibility checks can include checking the network congestion window. When the transaction type is push unsolicited data, the eligibility checks can include checking the network congestion window and the end node congestion window, and checking the request sequence number for a pull request. When the transaction type is push solicited data, the eligibility checks can include checking the network congestion window and checking if a push grant is received. When the transaction type is a pull response, the eligibility checks can include checking the network congestion window.

[0082] As represented by block 520, when the connection passes the eligibility checks, the connection scheduler 200 can enqueue the connection in a dispatch queue of a plurality of dispatch queues based on the transaction type of the connection. The dispatch queues can use a linked list data structure. The number of dispatch queues can be determined by a mapping of host devices and traffic classes to dispatch queues.

[0083] As represented by block 530, the connection scheduler 200 can perform solicitation checks on the connection. The solicitation checks can include determining that an in-flight count based on the connection is within a solicitation window to limit unsolicited solicitation buffer space. The solicitation checks can compare the incremented in-flight count to the solicitation window multiplied by a scaling factor each time the in-flight count is incremented based on a new request.

[0084] As represented by block 540, the connection scheduler 200 can select the connection from the dispatch queues based on decisions from a QoS arbiter. The QoS arbiter can include a plurality of packet selectors to select the connection from each dispatch queue using a weighted round robin scheme. The QoS arbiter can also include a plurality of dispatch queue selectors to select a dispatch queue including the selected connection using a priority and / or weighted round robin scheme. The QoS arbiter can also include a host selector to select a host device including the selected dispatch queue using a weighted round robin scheme.

[0085] As represented by block 550, the connection scheduler 200 can enqueue the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection. The connection queues can use a linked list data structure. The total number of connection queues can correspond to the number of active connections supported multiplied by the number of linked lists.

[0086] Figure 6 An example computing system 600 is depicted that can be configured as Figure 1The network device 110. The computing system 600 may include one or more processors 610 communicating with memory 620, at least one network interface controller 650 having a network interface port 660 for connecting to a network, input / output (I / O) components 670, and other components 680. As an example, the processor 610 may communicate with memory 620, network interface controller 650, I / O components 670, and other components 680 via bus 640. The processor 610 may be incorporated into or connected to cache memory 630. In some cases, instructions are read from memory 620 into cache memory 630 and executed by processor 610 from cache memory 630.

[0087] Processor 610 can be any logic circuit system that executes instructions fetched from memory 620 or cache 630. Processor 610 can be a microprocessor unit or a dedicated processor. Computing device 600 can be based on any processor or set of processors capable of operating as described herein. Processor 610 can be a single-core or multi-core processor. Processor 610 can be multiple different processors. Processor 610 can be implemented as one or more on-chip circuit systems.

[0088] Memory 620 can be any device suitable for storing computer-readable data. Memory 620 can be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and memory devices, semiconductor memory devices such as EPROM, EEPROM, SDRAM and flash memory devices, magnetic disks, magneto-optical disks and optical disks such as CD ROM, DVD-ROM, or... Optical disc. The computing system 600 can have any number of memory devices 620.

[0089] The cache memory 630 can be in the form of computer memory placed near the processor 610 for fast access. The cache memory 630 can be part of the processor 610 or on the same chip as the processor 610. Multiple levels of cache 630 can exist, for example, two-level cache layers and three-level cache layers.

[0090] Network interface controller 650 can manage data exchange via network interface 660. Network interface controller 650 can handle the physical and data link layers of the Open Systems Interconnection (OSI) model for network communication. Some tasks of network interface controller 650 can be processed by one or more processors 610. Network interface controller 650 can be integrated into processor 610, such as through on-chip circuitry.

[0091] The computing system 600 can have multiple network interfaces 660 controlled by a single controller 650 or multiple network interface controllers 650. Each network interface 660 can be a connection point for a physical network link, such as a Category 5 Ethernet link. The network interface controller 650 can support wireless network connections, and the interface port 660 can be wireless, such as a radio, receiver / transmitter. The network interface controller 650 can implement one or more network protocols, such as Ethernet. The computing system 600 can exchange data with other computing systems via physical or wireless links over the network interfaces 660. The network interfaces 660 can link directly to another device or link to another device via an intermediate device, such as a network device, e.g., a hub, bridge, switch, or router, that connects the computing device 600 to a data network, such as the Internet.

[0092] The computing system 600 can include, or be provided with, one or more input or output (I / O) components 670. As examples, input devices can include a keyboard, microphone, touchscreen, sensor, and pointing device, such as a mouse or trackball. As examples, output devices can include a video display, speaker, and printer.

[0093] Other components 680 can include I / O interfaces, external serial device ports, and any additional coprocessors. For example, the computing system 600 can include an interface, such as a Universal Serial Bus (USB) interface, for connecting input devices, output devices, or additional memory devices, such as a portable flash drive or external media drive. The computing device 600 can include additional coprocessors, such as a mathematical coprocessor for assisting the processor 610 with high precision or complex calculations.

[0094] The foregoing alternative examples described above are not mutually exclusive unless otherwise specified, but can be implemented in various combinations to obtain unique advantages. The foregoing description of the embodiments shall be taken as illustrative and not restrictive, in view of the fact that these and other variations and combinations of the features described above can be utilized without departing from the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as "such as," "including" and the like, should not be interpreted as limiting the subject matter defined by the claims to the specific examples presented; rather, these examples are intended to illustrate only one of many possible embodiments. In addition, identical reference numerals have been used, where possible, to designate identical elements that are found in similar or analogous embodiments.

Claims

1. A method for scheduling a connection related to a data packet, the method comprising: performing, with one or more processors, an eligibility check for a connection, the eligibility check comprising determining, based on a transaction type of the connection, whether the connection is within a congestion window, wherein the congestion window comprises a network congestion window and an end node congestion window; in response to the connection passing the eligibility check, queuing, with the one or more processors, the connection in a scheduling queue of a plurality of scheduling queues based on the transaction type; selecting, with the one or more processors, the connection from the scheduling queue based on a quality of service (QoS) arbiter decision; and queuing, with one or more processors, the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

2. The method of claim 1, wherein the plurality of scheduling queues comprises a plurality of linked lists and the plurality of connection queues also comprises a plurality of linked lists.

3. The method of claim 2, wherein the plurality of linked lists of the plurality of scheduling queues and the plurality of linked lists of the plurality of connection queues each comprise a linked list for at least one of: a push request, an unsorted push request, a push grant, a push data, and a pull data transaction type.

4. The method of claim 1, wherein passing the eligibility check comprises determining that a number of outstanding requests is less than the end node congestion window.

5. The method of claim 1, wherein passing the eligibility check comprises determining that a packet sequence number is less than a base sequence number plus the network congestion window.

6. The method of claim 1, further comprising performing, with the one or more processors, a solicitation check for the connection, the solicitation check comprising determining, based on the connection being within a solicitation window, an in-flight count.

7. The method of claim 6, wherein passing the solicitation check further comprises determining that a rate counter is greater than 0.

8. The method of claim 1, wherein selecting the connection from the scheduling queue further comprises selecting the scheduling queue based on at least one of a priority and a weight of the scheduling queue.

9. The method of claim 1, wherein selecting the connection from the scheduling queue is based on a weight of the connection.

10. A system comprising: one or more processors; and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for scheduling a connection related to a data packet, the operations comprising: performing an eligibility check for a connection, the eligibility check comprising determining, based on a transaction type of the connection, whether the connection is within a congestion window, wherein the congestion window comprises a network congestion window and an end node congestion window; in response to the connection passing the eligibility check, queuing the connection in a scheduling queue of a plurality of scheduling queues based on the transaction type; selecting the connection from the scheduling queue based on a quality of service (QoS) arbiter decision; and queuing the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection. ​ queuing the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

11. The system of claim 10, wherein the plurality of dispatch queues comprise a plurality of linked lists, and the plurality of connection queues also comprise a plurality of linked lists.

12. The system of claim 10, wherein: the eligibility check comprises determining that a number of outstanding requests is less than the end node congestion window; and the eligibility check comprises determining that a packet sequence number is less than a base sequence number plus the network congestion window.

13. The system of claim 10, wherein the operations further comprise performing a solicitation check on the connection, the solicitation check comprising determining an in-flight count based on the connection being within a solicitation window.

14. The system of claim 10, wherein selecting the connection from the dispatch queue is based on a weight of the connection, and the operations further comprise selecting the dispatch queue based on at least one of a priority and a weight of the dispatch queue.

15. A non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for dispatching connections related to data packets, the operations comprising: performing an eligibility check on a connection, the eligibility check comprising determining whether the connection is within a congestion window based on a transaction type of the connection, wherein the congestion window comprises a network congestion window and an end node congestion window; in response to the connection passing the eligibility check, queuing the connection in a dispatch queue of a plurality of dispatch queues based on the transaction type of the connection; selecting the connection from the dispatch queue based on a quality of service (QoS) arbiter decision; and queuing the connection in a connection queue of a plurality of connection queues based on the transaction type of the connection.

16. The non-transitory computer-readable medium of claim 15, wherein the plurality of dispatch queues comprise a plurality of linked lists, and the plurality of connection queues also comprise a plurality of linked lists.

17. The non-transitory computer-readable medium of claim 15, wherein: the eligibility check comprises determining that a number of outstanding requests is less than the end node congestion window; and the eligibility check comprises determining that a packet sequence number is less than a base sequence number plus the network congestion window.

18. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise performing a solicitation check on the connection, the solicitation check comprising determining an in-flight count based on the connection being within a solicitation window.

19. The non-transitory computer-readable medium of claim 15, wherein selecting the connection from the dispatch queue is based on a weight of the connection, and the operations further comprise selecting the dispatch queue based on at least one of a priority and a weight of the dispatch queue.

Images