Load balancing system, device and method

Abstract

The present invention relates to load balancing systems, devices and methods, and embodiments relate to load balancing data on one or more network ports. A device may include a processing circuit, the processing circuit being configured to: transmit a first data packet in a first series of data packets to a destination device via a first port, determine a time interval between the first data packet and a second data packet in the first series of data packets, and in response to determining that the time interval is greater than a time threshold, transmit a second data packet to the destination device via a second port.

Description

Technical Field

[0001]

[0013] Embodiments discussed herein relate to systems, devices, and methods for determining a path through which data is provided from a source device to a destination device. Background Art

[0002] A network may include multiple paths through which data from a source device may be provided to a destination device. The paths may differ in one or more attributes (delay, bandwidth, load, etc.). Such a change in attribute may be evidence that the load on a first path is different than the load on a second path. A path with a greater load may include a path traversal time that is greater than a path traversal time for a path with a lesser load. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A block diagram of an embodiment of a network is illustrated as an example;

[0004] Figure 2 As an example, Figure 1 An exploded view of an embodiment of a network interface controller (NIC);

[0005] Figure 3 As an example, a logical diagram of a series of data packets is illustrated;

[0006] Figure 4 A block diagram illustrating an embodiment of a method for load balancing is shown as an example;

[0007] Figure 5 A logical block diagram of an embodiment of the system is illustrated as an example. DETAILED DESCRIPTION

[0008] The following description and accompanying drawings sufficiently illustrate specific embodiments to enable those skilled in the art to practice them. Other embodiments may incorporate structural changes, logical changes, electrical changes, process changes, and other changes. Portions and features of some embodiments may be included in other embodiments or replace portions and features of other embodiments. The embodiments set forth in the claims encompass all available equivalents of these claims.

[0009] Network traffic can be bursty. There have been several studies examining the causes of this phenomenon. Transmission Control Protocol (TCP) window exhaustion is a simple example of what can cause traffic bursts, but it is not the only source of this behavior. Bursts of packets are sometimes referred to as "packet trains." A packet train is a group of packets in the same flow (going to the same destination), and they can have a specified maximum time gap. A packet train consists of a series of packets coming from the same source and going to the same destination. When the time between two packets exceeds a certain time gap, the packets are considered to belong to different trains.

[0010] A string of packets is sometimes referred to herein as a microflow. In other instances, a microflow may refer to a very short-lived connection. In the context of this application, a microflow is a series of packets within an established connection, such as a string of packets. The time parameter between microflows may be user-configurable, with a value that depends on the frequency with which the application uses the network. The inter-arrival time of packets (packet interval) of a microflow may be a system parameter, such as may depend on network hardware and / or software.

[0011] Some data centers may be connected to minimize latency and allow high-bandwidth communications. One of the aspects of this data center architecture is that all sleds will have a specified number of corresponding connections to different switches. A sled is a device that may be located on a sled rack. Each sled may provide computing power and / or communication capabilities. The connection may be provided by a NIC that includes a number of ports equal to the number of connections. Any given packet may be sent via any of the ports. However, it may be important that packets in the same stream arrive in the same order. If packets in the same stream are sent via different ports, this may cause the packets to arrive at their destination out of order. This may be due to different delays or different congestion levels encountered by packets on different paths. Out-of-order packet arrival may result in unnecessary retransmissions. Retransmissions increase latency and consume bandwidth unnecessarily. To avoid this, packets in the same stream (connection) are currently sent via the same port.

[0012] Traffic may be divided by flows or connections between ports of a NIC in one of the following general approaches: 1. Deterministic, such as may be based on a defined forwarding table; 2. Random flow assignment, such as may be based on a hash value; and 3. Dynamic load balancing, such as may be based on congestion. Embodiments discussed herein relate to dynamic load balancing.

[0013] Deterministic flows follow predefined paths defined in memory, such as may include a routing table, etc. Random flow assignment may determine a hash value for a data packet, compare the hash value to a predetermined hash table, and forward the data packet to a port based on an entry in the hash table. Dynamic load balancing techniques may include determining properties such as the amount of traffic on a path, and changing the next data packet path based on the property.

[0014] Deterministic and random allocation methods are static and may not allow rebalancing even when bandwidth can be improved via rebalancing. Dynamic load balancing (DLB) is considered herein. However, out-of-order packets may occur, and it may be advantageous to avoid such out-of-order packets in order to reduce processing time and / or errors after receiving the packets.

[0015] The path between the source device and the destination device can be permanent or transient. When a given shipper has transient connections, it may be easy for the shipper to use DLB to balance traffic on its ports. As new connections are established, they can be assigned to the ports that are currently least loaded. This simple approach may not be sufficient when some or all of the connections are permanent.

[0016] For some traffic patterns, "elephant flows" (e.g., large, continuous flows) can greatly impact bandwidth management. Elephant flows can consume significant bandwidth and can be long-lived. It is not obvious when a connection is established how long the connection will last, or how much throughput the connection will require. Given this, it is possible that any of the above methods will randomly place an excessive number of elephant flows on the same port, saturating the bandwidth on that port while bandwidth on other ports is still available. This worst-case scenario is unlikely, but it is also unlikely that all connections will end up being allocated in a way that maximizes the utilization of the NIC's ports.

[0017] If, for example, each port on a NIC has 50 Mbps of bandwidth, then the ports of the NIC have an aggregate bandwidth of 200 Mbps (assuming the NIC has four ports). Deterministic or random flow allocation can place two demand flows on the same port. These flows will compete with each other and be limited to only 50 Mbps total even when the other three ports of the NIC are lightly loaded.

[0018] Similarly, DLB technology can make a bandwidth management decision when placing these two flows on the same port would have been the correct decision. Then the short-lived flows on the other ports finish their transmission and are torn down. However, two large amounts of data are now bound to the same port and may not be moved even when bandwidth becomes available on other ports.

[0019] What is needed is a DLB method to reallocate ports being used by flows without causing out-of-order packets. The systems, devices, and methods discussed herein can monitor flows and look for gaps between packets being sent. If the gaps are large enough, this allows subsequent packets to be transmitted on any port on the system (e.g., NIC, multi-system on chip (SOC) type system, etc.). This allows opportunistic rebalancing while avoiding packet reordering.

[0020] Figure 1 A block diagram of an embodiment of a network 100 is illustrated as an example. The illustrated network 100 includes a source device 110 and a destination device 120. The source device 110 is a device from which a data packet originates (a data packet source). The destination device 120 is a device to which a data packet is to be transmitted. Between the source device 110 and the destination device 120 is a switching network 130. A first data packet microflow 144 ("STRM A MF 1") and a second data packet microflow 146 ("STRM A MF 2") flow from the source device 110 to the destination device 120.

[0021] As shown, source device 110 includes a network interface controller (NIC) 112. NIC 112 (e.g., together with switch network 130) connects source device 110 to one or more other devices of a network. NIC 112 is a hardware component. Some other common terms for NICs may include: host fabric interface (HFI), network interface card, network adapter, local area network (LAN) adapter, and physical network interface. NIC 112 includes electrical and electronic components, such as processing circuits (see Figure 2 ). The NIC 112 includes a plurality of ports 114 and 116. The ports 114 and 116 are configured to transmit or receive data packets to or from the switching network 130. The NIC 112 may be replaced by a multi-SOC system or other interface device or system.

[0022] As shown, the destination device 120 includes a network interface controller (NIC) 122. The NIC 122 (e.g., along with the switched network 130) connects the destination device 120 to one or more other devices of the network 100. The NIC 122 can be similar to the NIC 112. The NIC 112 includes a plurality of ports 124 and 126. The ports 124 and 126 are configured to transmit packets to or receive packets from the switched network 130.

[0023] As shown, the switching network 130 includes a plurality of switches 132(1), 132(2), 132(3), 132(4), 132(5), 132(6), 132(7), 132(8), 132(9), and 132(10). As shown, the switching network 130 includes ten switches 132(1)-132(10), however, the switching network 130 may include more or fewer switches. Each of the switches 132(1)-132(10) may include one or more queues 136(1), 136(2), 136(3), 136(4), 136(5), 136(6), 136(7), 136(8), 136(9), and 136(10), respectively. While queues 136(1)-136(10) are illustrated as being part of switches 132(1)-132(10), respectively, queues 136(1)-136(10) may be separate from and coupled to switches 132(1)-132(10). For example, NICs 112 and / or 122 may include one or more queues.

[0024] A queue provides an asynchronous communication protocol so that the sender and receiver of a message do not need to interact with the queue at the same time. A queue typically receives data, temporarily stores the data, and provides the data for transmission in a predetermined order. Many queues provide data in a first-in, first-out (FIFO) manner, while other queues may prioritize data based on attributes of the data (e.g., metadata or the data itself), or have some other technique for ordering the transmission of received data.

[0025] In one or more embodiments, the switching network 130 may include a Clos network. A Clos network may be a multi-stage circuit switching network including an ingress stage, an intermediate stage, and an egress stage. Each stage includes a plurality of switches. Each switch at the ingress stage may be coupled to each switch at the intermediate stage, and each switch at the egress stage may be coupled to each switch at the intermediate stage. The egress stage includes r Each switch at the entry level includes n Each switch in the entry-level switch includes m There are output terminals in the intermediate stage.m Each switch in the intermediate stage consists of r Input terminals and r There are output terminals in the output stage. r switches. Each switch in the egress stage has m Input terminals and n output terminals. Figure 1 In the example, n Equal to two, m Equal to two and r Equals four.

[0026] Switches 132(1)-132(10) in switching network 130 may include multiple ports (e.g., input ports, output ports, and / or bidirectional ports) such that any input on any port can be switched to any port. Switches 132(1)-132(10) may be network switches that forward data to a single device that is to receive the data. Some switches include routing functionality for packet forwarding.

[0027] exist Figure 1 In the network 100, source device 110 can transmit data flow "A" (eg, a data flow conforming to TCP) to destination device 120. Flow A has two microflows 144 and 146 with a time interval between them (see Figure 3 ). Each of microflows 144 and 146 may include a portion of a data stream from source device 110. Each of microflows 144 and 146 is composed of ordered packets. The time interval between microflows 144 and 146 is a specific example of an interval between packets, specifically, an interval between the last packet of microflow 144 and the first packet of microflow 146.

[0028] Figure 2 An exploded view of an embodiment of NIC 112 is illustrated as an example. NIC 122 may include similar components as NIC 112. NIC 112 as illustrated includes processing circuitry 202, inputs 204(1), 204(2), 204(3), 204(4), and 204(5), ports 114 and 116, and output links 206(1) and 206(2).

[0029] Processing circuit 202 may include electrical or electronic components, such as may include one or more transistors, resistors, capacitors, inductors, diodes, regulators (e.g., current, voltage, and / or power regulators), multiplexers, logic gates, switches, buffers, amplifiers, oscillators, modulators, demodulators, interconnects (e.g., wired or wireless signal transmission mechanisms), antennas, radios (receiving radios and / or transmitting radios), etc. Processing circuit 202 may include an application specific integrated circuit (ASIC), a programmable gate array (e.g., a programmable chip, such as may include a field programmable gate array (FPGA)), a central processing unit (CPU), a graphics processing unit (GPU), etc. Processing circuit 202 may be configured as a state machine that is configured to receive data on inputs 204(1)-204(5) and generate flows (e.g., microflows) of data packets to ports 114 and 115 and to outputs 206(1) and 206(2).

[0030] Processing circuit 202 may determine a time interval (eg, a packet interval) between packets of microflows 144 and 146 to determine when the time between packets may be greater than (or equal to) a time threshold. Figure 3 As an example, a logical diagram of a series of data packets 300 is illustrated. Each data packet 302(1), 302(2), 302(3), 302(4), 302(5), and 302(6) includes a start time and a stop time. The stop time of the first data packet in the series of data packets 300 (e.g., Figure 3 t1, t3, t5, t7, t9 and t11) and the start time of the next data packet (consecutive data packets) (e.g., Figure 3 The differences between the times t0, t2, t4, t6, t8, and t10 of the input terminals 204(1), 204(2), 204(3), 204(4), and 204(5), respectively, define time intervals 304(1), 304(2), 304(3), 304(4), and 304(5). Processing circuitry 202 may determine times t0-t11 and / or time intervals 304(1)-304(5), and determine to which port 114 or 116 data from input(s) 204(1)-204(5) is to be forwarded. Such routing decisions may be made by comparing time intervals 304(1)-304(5) to a time threshold. In response to determining that time intervals 304(1)-304(5) are greater than (or equal to) the time threshold, processing circuitry 202 may switch the port 116 or 114 to which the data may be forwarded.

[0031] For example, assume that processing circuit 202 has provided data packet 302(1) to port 114. Processing circuit 202 may determine from transmitting or receiving data packet 302(1) ( Figure 3t1) is completed to transmit or receive data packet 302(2) ( Figure 3 This may be accomplished in a variety of ways, such as by counting a number of clock cycles (such as clock cycles of an oscillator of processing circuit 202), or recording time t1 and subtracting it from recorded time t2. Processing circuit 202 may then compare determined time interval 304(1) to a time threshold. The time threshold may be expressed in terms of seconds, sub-seconds, oscillator cycles, etc. In response to determining that time interval 304(1) is less than (or equal to) the time threshold, processing circuit 202 may provide data packet 302(2) to the same port 114 that provided data packet 302(1). In response to determining that time interval 304(1) is greater than (or equal to) the time threshold, processing circuit 202 may provide data packet 302(2) to another port ( Figure 1 and 2 In the embodiment, it is port 116).

[0032] The time threshold may be set based on hardware properties, such as may be determined empirically, such as by experimentation and / or by reviewing data sheets indicating delays of a device or link. The entity may try a first time threshold, allow the system to operate, and recognize that a second data packet stream may arrive at a destination before the first data packet stream is completely received. In such an instance, to help mitigate conflicts, the time threshold may be increased to help increase the chance that the second data packet stream is received only after the entirety of the first data packet stream has been received. In response to receiving the data packets out of order, the destination device 120 may issue a communication to the source device 110 indicating that the data packets were received out of order. In response to receiving the communication out of order, the processing circuit 202 may increase the time threshold.

[0033] In one or more embodiments, the time threshold may be set to the delta between the time it takes (1) data to travel from port 114 to destination device 120 and (2) data to travel from port 116 to destination device 120. If the delta is negative or less than the minimum delay, the time threshold may be set to the minimum delay.

[0034] In one or more embodiments, the time threshold may be determined based on the time it takes for data to travel from port 114 or 116 to destination device 120. The time it takes for data to travel from port 114 or 116 to destination device 120 may be estimated based on the time between the transmission of the data packet and the time an acknowledgement (ACK) packet or a negative ACK (NACK) packet is received from destination device 120. The time threshold may be set to half the time between the transmission of the data packet and the time the ACK / NACK is received.

[0035] Attributes of ports 114 and 116 to which the packet flow is destined (sometimes referred to as telemetry or local parameter data) may include bandwidth, latency, packet loss rate, bit error rate, etc. Attributes of the traversal path from source device 110 to destination device 120 may include bandwidth, traversal path transit time, packet loss rate, bit error rate, congestion, queue depth (a number of items stored in a queue of switches 132(1)-132(10) that are to be transmitted before the next received data, etc.). The traversal path transit time may include: the time from when a packet is received and / or detected at NIC 112 of source device 110 to the time when the same packet is received and / or detected at NIC 122. In one or more embodiments, the traversal path transit time includes: the time it takes processing circuitry 202 to provide a packet to one of ports 114 and 116; the time it takes a packet to traverse switching network 130; and / or the time it takes a packet to be provided from one of ports 124 and 126 to processing circuitry of destination device 120.

[0036] The packet loss rate may be a value indicating the number of packets lost per a set number of packets transmitted. For example, a packet loss rate of 0.01 indicates that one of a thousand packets from a port did not make it to the destination. The bit error rate indicates how many packets in a set number of packets include errors. For example, a bit error rate of 0.01 indicates that the data in one packet out of every hundred packets from a port includes errors in the data. Congestion on a path indicates how much traffic is currently on a port. For example, congestion may be a number between zero and two hundred and fifty-five (or other maximum numbers), or a value in a hierarchy of values ​​indicating to what extent the path to the destination device 120 is supported. The queue depth indicates the degree to which a queue is full or empty, so as to indicate how much data is stored on the queue, or how much storage is available on the queue. The queue depth may indicate the amount of packets to be transmitted before the next packet.

[0037] In one or more embodiments, NICs 112 and 122 include more than two ports. Each port in NIC 112 can be associated with a cost. The cost of a port can be a sum, such as can include a weighted sum, or other combination of one or more of the attributes associated with the port or the traversal path including the port. The cost of the port can be used when determining which port to switch a packet flow to. For example, processing circuit 202 can switch a packet flow to a port that includes a cost that is less than, greater than, or equal to the cost of the port to which the packet is being provided. It can be switched to a port that includes a cost that is greater than the cost in the case where packets are arriving out of order and the transmission of packets to destination device 120 is to be slowed down.

[0038] Figure 4 A block diagram of an embodiment of a method 400 for load balancing is illustrated as an example. The method 400 may be implemented by the processing circuit 202 or firmware, a device driver, or other machine-readable medium. The method 400 as shown includes: at operation 410, monitoring one or more attributes of port 1 and port 2 for a device to estimate corresponding path traversal times; at operation 420, transmitting data packet 1 of a first microflow via port 1; at operation 430, determining whether the time interval between data packet 1 and data packet 2 is greater than a time threshold; at operation 440, in response to determining that the time interval between data packet 1 and data packet 2 is not greater than the time threshold, transmitting data packet 2 of the first microflow via port 1; at operation 450, in response to determining that the time interval between data packet 1 and data packet 2 is greater than the time threshold, transmitting data packet 2 in a new microflow via port 2. In one or more embodiments, operation 450 may be implemented after receiving a NACK / ACK for data packet 1.

[0039] The method 400 may further include estimating a first transmission time associated with the first port of the source device to the destination device. Operation 430 may be performed for each port of the NIC. The method 400 may further include, in response to a transmission interval on a second port of the source device after the transmission of the first data packet being greater than the estimated first transmission time, transmitting a second data packet of the second microflow to the destination device via the second port.

[0040] Operation 410 may include estimating a first transmission time based on first attribute data. The attribute data may include one or more attributes, such as congestion information about traffic on a path between a port of a source device and a destination device. The attribute data may include queue (e.g., cache) information of a memory along the path between the port of the source device and the destination device. The queue information may include the depth of data in the queue (e.g., how much data is currently in the queue). The processing circuit 202 may forward the data packet along a path that includes less data in the queue, such as in response to determining that the time interval is greater than or equal to a time threshold. Operation 410 may include measuring the time between the transmission of a data packet to the destination device and the receipt of an associated confirmation message.

[0041] The method 400 may further include: increasing the estimated first transmission time in response to out-of-order information from the destination device. The method 400 may further include: estimating a second transmission time associated with a second port of the source device to the destination device. The method 400 may further include: transmitting a third data packet of the second microflow to the destination device via the second port in response to a transmission interval on the first port that is greater than the estimated second transmission time after the transmission of the second data packet.

[0042] Operation 420 may include transmitting the first data packet via a network interface controller of the source device. Operation 420 may include transmitting the first data packet over a Clos network. Method 400 may further include transmitting the second data packet via the first port in response to receiving an acknowledgement associated with the first data packet and before the second data packet has been transmitted via the second port.

[0043] Embodiments may be implemented in NIC hardware, firmware, and / or device drivers. Embodiments may not require changes to the host operating system or protocol stack. Traffic (eg, TCP traffic) provided from a source device to a destination device may remain unchanged.

[0044] One aspect of one or more embodiments is to determine sufficient interval time to allow microflow port switching. There may be multiple ways to achieve this. The simplest way would be to start with a small amount (such as 1500 byte times). If out-of-order packets occur, the required interval time is increased.

[0045] Another method may include determining the required interval length, such as by collecting attribute data from ports and / or various path switches and queues. This may be used to determine queue depth and congestion levels on the path between the source device and the destination device. The attribute data may be used to estimate various path traversal times, and the required interval time may then be set as the delta from the current path to the new path traversal time.

[0046] Figure 5 As an example, a logical block diagram of an embodiment of the system is illustrated. For example, Figure 5 An example of an electronic device (e.g., a system) including a device or component is depicted, which may be included in source device 110, destination device 120, NIC 112 and / or 122, switch 132(1)-132(10), queue 136(1)-136(10), processing circuit 202, or other devices discussed herein. In one embodiment, system 500, such as an example that may include source and / or destination devices, includes, but is not limited to: a desktop computer, a laptop computer, a netbook computer, a tablet device, a notebook computer, a personal digital assistant (PDA), a server, a workstation, a cellular phone, a mobile computing device, a smart phone, an Internet appliance, or any other type of computing device. In some embodiments, system 500 is a system on a chip (SOC) type system.

[0047] In one embodiment, the processor 510 has one or more processing cores 512 and 512N, where 512N represents the Nth processor core inside the processor 510, where N is a positive integer. In one embodiment, the system 500 includes multiple processors (including 510 and 505), where the processor 505 has logic similar to or the same as the logic of the processor 510. In some embodiments, the processing core 512 includes, but is not limited to, pre-fetch logic for fetching instructions, decoding logic for decoding instructions, execution logic for executing instructions, and the like. In some embodiments, the processor 510 has a cache memory 516 for caching instructions and / or data for the system 500. The cache memory 516 can be organized into a hierarchical structure including one or more levels of cache memory.

[0048] In some embodiments, the processor 510 includes a memory controller 514 that is operable to implement functions that enable the processor 510 to access and communicate with a memory 530, which includes a volatile memory 532 and / or a non-volatile memory 534. In some embodiments, the processor 510 is coupled to the memory 530 and the chipset 520. The processor 510 may also be coupled to a wireless antenna 578 to communicate with any device configured to transmit and / or receive wireless signals. In one embodiment, the wireless antenna interface 578 operates in accordance with, but is not limited to, the IEEE 802.11 standard and its related families, Home Plug AV (HPAV), Ultra Wideband (UWB), Bluetooth, WiMax, or any form of wireless communication protocol.

[0049] In some embodiments, the volatile memory 532 includes, but is not limited to, synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 534 includes, but is not limited to, flash memory, phase change memory (PCM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or any other type of non-volatile memory device.

[0050] Memory 530 stores information and instructions to be executed by processor 510. In one embodiment, memory 530 may also store temporary variables or other intermediate information while processor 510 is executing instructions. In the illustrated embodiment, chipset 520 is connected to processor 510 via point-to-point (PtP or PP) interfaces 517 and 522. Chipset 520 enables processor 510 to connect to other elements in system 500. In some embodiments of the present invention, interfaces 517 and 522 operate according to PtP communication protocols such as Intel® QuickPath Interconnect (QPI) and the like. In other embodiments, different interconnects may be used.

[0051] In some embodiments, the chipset 520 is operable to communicate with the processors 510, 510N, the display device 540, and other devices 572, 576, 574, 560, 562, 564, 566, 577, etc. The chipset 520 may also be coupled to a wireless antenna 578 to communicate with any device configured to transmit and / or receive wireless signals.

[0052] The chipset 520 is connected to a display device 540 via an interface 526. The display 540 may be, for example, a liquid crystal display (LCD), a plasma display, a cathode ray tube (CRT) display, or any other form of visual display device. In some embodiments of the present invention, the processor 510 and the chipset 520 are merged into a single SOC. In addition, the chipset 520 is connected to one or more buses 550 and 555 that interconnect various elements 574, 560, 562, 564, and 566. The buses 550 and 555 may be interconnected together via a bus bridge 572. In one embodiment, the chipset 520 is coupled to a non-volatile memory 560, (one or more) mass storage devices 562, a keyboard / mouse 564, and a network interface 566 via interfaces 524 and / or 504, a smart TV 576, a consumer electronic device 577, and the like.

[0053] In one embodiment, the mass storage device 562 includes, but is not limited to, a solid state drive, a hard disk drive, a universal serial bus flash memory drive, or any other form of computer data storage medium. In one embodiment, the network interface 566 is implemented by any type of known network interface standard, including but not limited to: an Ethernet interface, a universal serial bus (USB) interface, a peripheral component interconnect (PCI) express interface, a wireless interface, and / or any other suitable type of interface. In one embodiment, the wireless interface operates according to, but is not limited to, the IEEE 802.11 standard and its related families, Home Plug AV (HPAV), Ultra Wideband (UWB), Bluetooth, WiMax, or any form of wireless communication protocol.

[0054] Although in Figure 5 The modules shown in are depicted as separate blocks within system 500, but the functions performed by some of these blocks may be combined within a single semiconductor circuit, or may be implemented using two or more separate integrated circuits. For example, although cache memory 516 is depicted as a separate block within processor 510, cache memory 516 (or selected aspects of 516) may be incorporated into processor core 512.

[0055] Figure 5 The device 500 of illustrative embodiments illustrates one configuration. For example, each processing core 512 may include its own dedicated memory 516. In another example, each processing core 512 may include its own dedicated peripheral component interconnect (PCI) slot. In another example, each processing core 512 may communicate with other processing cores via an interconnect such as a Quick Path Interconnect (QPI) or an Ultra Path Interconnect (UPI).

[0056] To better illustrate the methods and apparatus disclosed herein, a non-limiting list of examples is provided here:

[0057] Example 1 may include: a device including a processing circuit, the processing circuit being used to: transmit a first data packet in a first series of data packets to a destination device via a first port, determine a time interval between the first data packet and a second data packet in the first series of data packets, and in response to determining that the time interval is greater than a time threshold, transmit a second data packet to the destination device via a second port.

[0058] In Example 2, Example 1 may further include, wherein the processing circuit is further configured to estimate a first transmission time associated with the first port to a destination device.

[0059] In Example 3, Example 2 may further include, wherein the processing circuit is further configured to estimate a second transmission time associated with the second port to the destination device.

[0060] In Example 4, Example 3 may further include, wherein the first transmission time and the second transmission time are estimated based on the first attribute data and the second attribute data, respectively.

[0061] In Example 5, Example 4 may further include, wherein if the second transmission time is less than the first transmission time, the first time threshold is greater than or equal to a difference between the estimated first transmission time and the estimated second transmission time.

[0062] In Example 6, at least one of Examples 2-5 may further include, wherein estimating a first transmission time associated with the first port to the destination device based on the first attribute data includes: the processing circuit is used to measure the time between transmission of a data packet to the destination device and receipt of an associated confirmation message.

[0063] In Example 7, at least one of Examples 2-6 may further include, wherein the processing circuit is further configured to determine a time interval between the second data packet and the third data packet, and in response to determining that the time interval is greater than a second time threshold, transmit the third data packet to the destination device via the first port.

[0064] In Example 8, Example 7 may further include, wherein if the first transmission time is less than the second transmission time, the second time threshold is a difference between the second transmission time and the first transmission time, and if the second transmission time is less than the first transmission time, the second time threshold is a minimum time threshold.

[0065] In Example 9, at least one of Examples 2-8 may further include, wherein the first attribute data includes at least one of congestion information and a queue depth along a transmission path between the computer device and the destination device through the first port.

[0066] In Example 10, at least one of Examples 1-9 may further include, wherein the processing circuit is further configured to increase the first time threshold in response to an out-of-sequence message from the destination device.

[0067] In Example 11, at least one of Examples 1-10 may further include a network interface controller having a first port and a second port.

[0068] In Example 12, at least one of Examples 1-11 may further include, wherein the data packet is transmitted via a Clos network coupled between the computer device and the destination device.

[0069] In Example 13, at least one of Examples 1-12 may further include, wherein the processing circuit is further to transmit the second data packet via the first port in response to receiving an acknowledgement associated with the first data packet on the first port and before the second data packet has been transmitted via the second port.

[0070] Example 14 may include a method for balancing a load on a network, the method comprising: estimating a first transmission time associated with a first port of a source device to a destination device, transmitting a first data packet in a first series of data packets to the destination device via the first port, and in response to a transmission interval on a second port of the source device after the transmission of the first data packet being greater than the estimated first transmission time, transmitting a second data packet in a second series of data packets to the destination device via the second port.

[0071] In Example 15, Example 14 may further include, wherein estimating the first transmission includes estimating the first transmission time based on the first attribute data.

[0072] In Example 16, Example 15 may further include, wherein the first attribute data includes: at least one of congestion information and queue depth along the transmission path.

[0073] In Example 17, at least one of Examples 14-16 may further include, in response to an out-of-sequence message from the destination device, increasing the estimated first transmission interval.

[0074] In Example 18, at least one of Examples 14-17 may further include, wherein estimating a first transmission time associated with the first port to the destination device based on the first attribute data includes measuring a time between transmission of a data packet to the destination device and receipt of an associated confirmation message.

[0075] In Example 19, at least one of Examples 14-18 may further include estimating a second transmission time associated with a second port of the source device to the destination device, and in response to a transmission interval on the first port being greater than the estimated second transmission time after transmission of the second data packet, transmitting a third data packet to the destination device via the second port.

[0076] In Example 20, at least one of Examples 14-19 may further include transmitting the first data packet via a network interface controller of the source device.

[0077] In Example 21, at least one of Examples 14-20 may further include, wherein the first data packet is transmitted via a Clos network.

[0078] In Example 22, at least one of Examples 14-21 may further include transmitting the second data packet via the first port in response to receiving an acknowledgement associated with the first data packet and before transmission of the second data packet via the second port has completed.

[0079] Example 23 may include at least one non-transitory machine-readable medium comprising instructions that, when executed by a processing circuit of a source device, cause the processing circuit to perform operations including: transmitting a first data packet in a first series of data packets to a destination device via a first port, determining a time interval between the first data packet and a second data packet in the first series of data packets, and in response to determining that the time interval is greater than a time threshold, transmitting a second data packet to the destination device via a second port.

[0080] In Example 24, Example 23 may further include, wherein the operations further include estimating a first transmission time associated with the first port to the destination device.

[0081] In Example 25, Example 24 may further include, wherein the operations further include estimating a second transmission time associated with the second port to the destination device.

[0082] In Example 26, Example 25 may further include, wherein the first transmission time and the second transmission time are estimated based on the first attribute data and the second attribute data, respectively.

[0083] In Example 27, Example 26 may further include, wherein (1) if the second transmission time is less than the first transmission time, the first time threshold is greater than or equal to a difference between the estimated first transmission time and the estimated second transmission time, or (2) if the estimated second transmission time is less than the estimated first transmission time, the first time threshold is a minimum time threshold.

[0084] In Example 28, at least one of Examples 24-27 may further include, wherein estimating a first transmission time associated with the first port to the destination device based on the first attribute data includes measuring a time between transmission of a data packet to the destination device and receipt of an associated confirmation message.

[0085] In Example 29, at least one of Examples 24-28 may further include, wherein the operation further includes: determining a time interval between the second data packet and the third data packet, and in response to determining that the time interval is greater than a second time threshold, transmitting the third data packet to the destination device via the first port.

[0086] In Example 30, Example 29 may further include, wherein (1) if the estimated first transmission time is less than the estimated second transmission time, the second time threshold is a difference between the second transmission time and the first transmission time, and if the estimated second transmission time is less than the estimated first transmission time, the second time threshold is a minimum time threshold, or (2) if the estimated second transmission time is less than the estimated first transmission time, the second time threshold is a minimum time threshold.

[0087] In Example 31, at least one of Examples 24-30 may further include, wherein the first attribute data includes at least one of congestion information and a queue depth along a transmission path between the computer device and the destination device through the first port.

[0088] In Example 32, at least one of Examples 23-31 may further include, wherein the operations further include: increasing the first time threshold in response to an out-of-sequence message from the destination device.

[0089] In Example 33, at least one of Examples 23-32 may further include, wherein the operations further include: transmitting the second data packet via the first port in response to receiving an acknowledgement associated with the first data packet on the first port and before the second data packet has been transmitted via the second port.

[0090] In Example 34, at least one of Examples 1-33 may further include, wherein the first data packet and the second data packet are consecutive data packets.

[0091] These examples are intended to provide non-limiting examples of the present subject matter - they are not intended to provide an exclusive or exhaustive explanation.The above detailed description is included to provide further information about the present apparatus and methods.

[0092] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The accompanying drawings illustrate as diagrams specific embodiments in which the present disclosure can be practiced. These embodiments are also referred to as "examples" in this article. Such examples may include elements other than those shown or described. However, the inventors also anticipate examples in which only those elements shown or described are provided. In addition, the inventors also anticipate the following examples, which use any combination or arrangement of those elements shown or described (or one or more aspects thereof) with respect to a specific example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0093] In this document, the terms "one" or "an", as is common in patent documents, are used to include one or more than one, independent of any other instance or use of "at least one" or "one or more". In this document, the term "or" is used to refer to a non-exclusive or, so that "A or B" includes "A but not B", "B but not A", and "A and B", unless otherwise indicated. In this document, the terms "including" and "wherein" are used as the plain language equivalents of the corresponding terms "comprising" and "wherein". In addition, in the appended claims, the terms "including" and "comprising" are open, that is, systems, devices, articles, compositions, formulas or processes that include elements other than those listed after such terms in the claims are still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first", "second", and "third", etc. are used only as labels and are not intended to impose numerical requirements on their objects.

[0094] The above description is intended to be illustrative rather than restrictive. For example, the examples described above (or one or more aspects thereof) may be used in combination with one another. Other embodiments may be used, such as by a person of ordinary skill in the art, when reviewing the above description. The abstract is provided to comply with 37 CFR §1.72 (b) to allow the reader to quickly clarify the nature of the technical disclosure. The understanding advocated is that the abstract will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above-mentioned specific embodiments, various features may be combined to simplify the present disclosure. This should not be understood as an intention to make the unclaimed public features necessary for any claim. Instead, the inventive subject matter may be less than all the features of a particular disclosed embodiment. Therefore, the attached claims are incorporated into the specific embodiments in this manner, wherein each claim is a separate embodiment by itself, and it is contemplated that such embodiments may be combined with one another in various combinations or arrangements. The scope of the present disclosure should be determined with reference to the attached claims and the full range of equivalents given to such claims.

Claims

1. A device comprising a processing circuit, the processing circuit being configured to: transmitting a first packet of a first series of packets to a destination device via a first port; determining a time interval between a first data packet and a second data packet in the first series of data packets; In response to determining that the time interval is greater than a first time threshold, transmitting the second data packet to the destination device via a second port; estimating a first transmission time associated with the first port to a destination device; estimating a second transmission time associated with the second port to the destination device; determining a time interval between the second data packet and a third data packet; as well as In response to determining that the time interval is greater than a second time threshold, the third data packet is transmitted to the destination device via the first port, wherein if the first transmission time is less than the second transmission time, the second time threshold is a difference between the second transmission time and the first transmission time, and if the second transmission time is less than the first transmission time, the second time threshold is a minimum time threshold. 2 . The apparatus of claim 1 , wherein the first transmission time and the second transmission time are estimated based on first attribute data and second attribute data, respectively.

3. The apparatus of claim 2, wherein if the second transmission time is less than the first transmission time, the first time threshold is greater than or equal to a difference between the estimated first transmission time and the estimated second transmission time.

4. The device of claim 1 , wherein estimating a first transmission time associated with the first port to the destination device based on first attribute data comprises: The processing circuit is to measure a time between transmission of a data packet to the destination device and receipt of an associated acknowledgement message.

5. The apparatus according to claim 1, wherein the first attribute data comprises: At least one of congestion information and a queue depth along a transmission path between the device and the destination device through the first port.

6. The device of claim 1, wherein the processing circuit is further configured to increase the first time threshold in response to an out-of-sequence message from the destination device.

7. The apparatus according to claim 1, further comprising: A network interface controller having the first port and the second port.

8. The device of claim 1, wherein the data packet is transmitted via a Clos network coupled between the device and the destination device.

9. The apparatus of claim 1, wherein the processing circuit is further to transmit the second data packet via the first port in response to receiving an acknowledgement associated with the first data packet on the first port and before the second data packet has been transmitted via the second port.

10. A method for balancing a load on a network, the method comprising: estimating a first transmission time associated with a first port of the source device to a destination device; transmitting a first packet of a first series of packets to the destination device via the first port; in response to a transmission interval on a second port of the source device after transmission of the first data packet being greater than the estimated first transmission time, transmitting a second data packet in a second series of data packets to the destination device via the second port; estimating a second transmission time associated with a second port of the source device to the destination device; determining a time interval between the second data packet and a third data packet; as well as In response to determining that the time interval is greater than a time threshold, the third data packet is transmitted to the destination device via the first port, wherein if the first transmission time is less than the second transmission time, the time threshold is a difference between the second transmission time and the first transmission time, and if the second transmission time is less than the first transmission time, the time threshold is a minimum time threshold.

11. The method of claim 10, wherein estimating the first transmission time comprises: The first transmission time is estimated based on first attribute data.

12. The method according to claim 11, wherein the first attribute data comprises: At least one of congestion information and queue depth along the transmission path.

13. The method according to claim 10, further comprising: In response to an out-of-sequence message from the destination device, the estimated first transmission time is increased.

14. The method of claim 10, wherein estimating a first transmission time associated with the first port to a destination device based on first attribute data comprises: The time between transmission of a data packet to the destination device and receipt of an associated acknowledgement message is measured.

15. The method according to claim 10, further comprising: In response to a transmission interval on the first port being greater than the estimated second transmission time after transmission of the second data packet, a third data packet is transmitted to the destination device via the second port.

16. The method according to claim 10, further comprising: The first data packet is transmitted via a network interface controller of the source device.

17. The method of claim 10, wherein the first data packet is transmitted via a Clos network.

18. The method according to claim 10, further comprising: In response to receiving an acknowledgment associated with the first data packet and before transmission of the second data packet via the second port has concluded, the second data packet is transmitted via the first port.

19. At least one non-transitory machine-readable medium comprising instructions that, when executed by processing circuitry of a source device, cause the processing circuitry to perform operations comprising: transmitting a first packet of a first series of packets to a destination device via a first port; determining a time interval between a first data packet and a second data packet in the first series of data packets; In response to determining that the time interval is greater than a first time threshold, transmitting the second data packet to the destination device via a second port; estimating a first transmission time associated with the first port to a destination device; estimating a second transmission time associated with the second port to the destination device determining a time interval between the second data packet and a third data packet; as well as In response to determining that the time interval is greater than a second time threshold, the third data packet is transmitted to the destination device via the first port, wherein if the first transmission time is less than the second transmission time, the second time threshold is a difference between the second transmission time and the first transmission time, and if the second transmission time is less than the first transmission time, the second time threshold is a minimum time threshold.

20. A computer readable medium having instructions stored thereon, which when executed by a computing device, cause the computing device to perform the method according to any one of claims 10 to 18.

21. An apparatus comprising means for performing the steps of the method according to any one of claims 10 to 18.

Images