Scheduling mechanism for approximate fine-grained, per-flow rate adjustment and periodic-grained inter-packet spacing in network applications

Abstract

Innovations in packet scheduling are described herein that allow scheduling mechanisms to approximate fine-grained rate adjustments and periodic-grained inter-packet spacing for packets of a flow. For example, in an iteration of a scheduler loop, the transmitter determines whether a proximity condition is satisfied for a next packet of the flow. The proximity condition depends, at least in part, on how long the target next transmission time follows the current time. If the next packet expires or if a proximity condition is satisfied for the next packet, the next packet is scheduled for transmission. When a next packet is scheduled for transmission, the transmitter transmits the next packet and updates the target next transmission time based at least in part on the target transmission rate.

Description

Copyright Notice

[0001] The disclosed portion of this patent document contains copyrighted material. The copyright owner does not object to anyone copying the patent document or patent disclosure appearing in the Patent and Trademark Office's patent files or records, but otherwise reserves all copyright rights. Background Technology

[0002] In computer networking, a transport protocol is a set of rules and procedures that govern the exchange of data between computing devices on a computer network. Typically, one computing device, acting as a sender, breaks down data (such as messages or files) into smaller units called packets. The sender transmits these packets across the computer network to another computing device, which acts as a receiver and is able to reconstruct the data from the information in the packets.

[0003] Transport protocols interoperate with network protocols at other layers. For example, a transport layer implementation receives data (such as messages or files) from implementations of the application, presentation, or session layers. The transport layer implementation provides transport layer packets for data to network layer implementations that can implement a version of the Internet Protocol (“IP”). Depending on the transport protocol, transport layer processing can provide features such as error detection, retransmission of dropped transport layer packets, control over the rate at which transport layer packets are transmitted (sometimes called flow control), and sorting of transport layer packets. Transmission Control Protocol (“TCP”) and User Datagram Protocol (“UDP”) are two examples of transport protocols.

[0004] Reliable transport protocols use mechanisms to guarantee, or at least take steps to ensure, the delivery of transport layer packets from sender to receiver. These mechanisms can include error detection, retransmission of dropped packets, and flow control. TCP is an example of a reliable transport protocol. Mechanisms providing reliable delivery of transport layer packets can increase the latency of detecting and retransmitting dropped packets; if many transport layer packets are dropped, the latency can be significant. UDP is an example of an unreliable transport protocol. UDP can provide more timely delivery of transport layer packets without the overhead of reliability mechanisms or attempts to guarantee delivery.

[0005] Transport layer packets can be delayed for various reasons. For example, a switch may be temporarily busy, causing delays in delivering packets through that switch. In extreme cases, packets may be dropped due to congestion at a switch. Rarely, packet bits may be flipped due to interference, unreliable links, or other reasons, causing the packet to be dropped when information loss is detected. The transmitter is capable of retransmitting dropped packets.

[0006] In transport protocols, flow control helps avoid congestion and packet loss. Generally, window-based flow control is one method, while rate-based flow control is another.

[0007] In window-based flow control, the sender adjusts the size of the window used to transmit packets. For example, a sliding window mechanism can be used to implement window-based flow control. The sender maintains a sliding window that defines the range of packets the sender can transmit. The sender begins transmitting packets for the stream within its sliding window and waits for acknowledgment (“ACK”) packets from the receiver. The receiver also maintains a sliding window that defines the range of packets the receiver can receive. As the receiver receives packets for the stream, it sends an ACK packet back to the sender to acknowledge receipt of the corresponding packet. As the sender receives an ACK packet from the receiver, it can move its sliding window. If the sender receives an ACK packet for a packet in the stream, it can move its window forward, indicating that a previous packet in the stream has been successfully received. On the other hand, if the sender does not receive an ACK packet for a packet in a timely manner, it eventually assumes the packet was lost and retransmits the stream packet. The sender and receiver can negotiate the initial size of the sliding window, and the size of the sliding window can be dynamically adjusted (e.g., using a smaller window size after a retransmission or for slow start, then increasing the window size). In window-based flow control, the transmission rate of packets transmitted by the sender is indirectly affected by adjusting the window size.

[0008] In contrast, rate-based flow control directly adjusts the transmission rate of packets transmitted by the transmitter. The transmission rate can be specified as an inter-packet gap value (sometimes called the inter-packet interval value), which indicates the number of intervals between consecutive packets, for example, as the difference between the start times for transmitting consecutive packets. The transmitter can adapt the transmission rate to network conditions. The transmitter can respond to congestion or packet loss by reducing the transmission rate. Alternatively, the transmitter can increase the transmission rate in response to positive feedback about network conditions. When using rate-based flow control, the transmitter and receiver can also maintain a sliding window to track which packets have been delivered and identify packets for retransmission.

[0009] In some implementations of rate-based flow control, the transmission rate cannot be effectively regulated in a fine-grained manner (e.g., with inter-packet intervals at the level of a single cycle in the transport layer scheduling mechanism). Specifically, the transport layer scheduling mechanism may include operations in a scheduler loop whose duration is variable and coarse-grained (e.g., on the scale of tens or hundreds of cycles). Furthermore, the scheduler loop may iterate over multiple active packet streams, potentially sending packets from any of the multiple streams. These factors generally prevent the scheduling mechanism from sending the next packet of a particular stream in a specific cycle that precisely matches the inter-packet interval for an arbitrary transmission rate. Thus, in many scenarios, the sender cannot effectively regulate packet streams using fine-grained adjustments to the transmission rate. Summary of the Invention

[0010] In summary, the detailed description presents an innovation in packet scheduling that allows the scheduling mechanism to adjust the packet rate for a flow at approximately fine-grained levels and to use periodic-granular inter-packet intervals. In some example implementations, the sender can specify periodic-granular inter-packet intervals. The scheduling mechanism includes scheduling packet transmission operations at approximately the specified inter-packet intervals. This allows the sender to effectively adjust the transmission rate in a fine-grained manner. Furthermore, in some example implementations, the inter-packet interval can be specified on a per-flow basis, allowing the sender to set different transmission rates for different packet flows.

[0011] Based on some of the techniques and tools described herein, packets of a given packet stream are scheduled for transmission. In a scheduler loop used for packet scheduling (e.g., as part of a transport layer scheduling mechanism), the sender processes the given packet stream. The sender is capable of processing multiple packet streams comprising a given packet stream within the scheduler loop, in which case the sender performs operations on multiple packet streams capable of having different transmission rates within the scheduler loop.

[0012] Specifically, during the scheduler loop iterations, the sender can determine whether the next packet for a given packet stream has expired. For example, if the target's next transmission time is earlier than the current time, the next packet has expired. Or, as another example, if a new packet stream is added to the given packet stream, the next packet has expired.

[0013] In some cases, the next packet for a given packet stream is not due. During the scheduler loop iterations, the sender determines whether a proximity condition is met for the next packet in a given packet stream. The proximity condition depends at least in part on how long after the current time the target next transmission time is. Generally, if the target next transmission time (even if later than the current time) is close enough to the current time, the proximity condition is met, and the next packet can be scheduled for transmission. For example, the sender determines the difference between the target next transmission time and the current time. The sender also determines a threshold, which depends on an estimate of the duration of the iterations for the scheduler loop. The threshold can be calculated in different ways, depending on the implementation. The sender compares the difference to the threshold. The satisfaction of the proximity condition depends on the result of the comparison between the difference and the threshold.

[0014] The transmitter determines whether to schedule the next packet of a given packet stream for transmission. The next packet is scheduled for transmission if it expires or if the proximity condition for the next packet is met. When the next packet of a given packet stream is scheduled for transmission, the transmitter sends the next packet and updates the target next transmission time, at least in part, based on the target transmission rate. The target transmission rate can be a target inter-packet gap (“IPG”) value associated with the given packet stream. If so, different active packet streams can have different target IPG values.

[0015] The transmitter can dynamically adjust the target transmission rate. For example, when the transmitter receives network feedback, it can set the target transmission rate based at least in part on the network feedback.

[0016] In a first method for approximate fine-grained rate adjustment, the transmitter sends some packets of a given packet stream ahead of schedule, while delaying the transmission of other packets, but transmitting packets of the given packet stream at intervals averaging the target IPG value. According to the first method, when the next packet of a given packet stream has not yet expired, the transmitter probabilistically determines in iteration whether to send the next packet of the given packet stream. Specifically, to determine a threshold for the proximity condition, the transmitter determines a random number between 0 and an estimate of the iteration duration for the scheduler loop, and then sets the threshold to the random number. When updating the target next transmission time, the transmitter determines the transmission time and adds the target IPG value to the transmission time.

[0017] For the second method of approximate fine-grained rate adjustment, the transmitter may send packets of a given packet stream ahead of schedule, but if so, adjusts the target next transmission time for subsequent packets to compensate for the early transmission. Packets of the given packet stream are transmitted at intervals averaging the target IPG value. According to the second method, to determine a threshold for the proximity condition, the transmitter sets the threshold as an estimate of the iteration duration for the scheduler loop. When updating the target next transmission time, the transmitter determines the transmission time and also determines whether the difference between the target next transmission time and the current time is greater than zero. If so, the transmitter combines the transmission time, the target IPG value, and the difference. Otherwise, the transmitter adds the target IPG value to the transmission time. Therefore, if the difference is greater than zero, the updated target next transmission time is equal to the transmission time plus the target IPG value plus the difference.

[0018] In some example implementations, using either the first or second method, the sender can perform fine-grained rate adjustment on packets in the packet stream and set a periodic granular target IPG value. The scheduling mechanism then schedules packets for transmission in a manner that approximates the fine-grained rate adjustment and the periodic granular target IPG value.

[0019] The innovations described herein can be implemented as part of a method, as part of a computer system (physical or virtual, as described below) or network interface device configured to perform the method, or as part of a tangible computer-readable medium storing computer-executable instructions for causing one or more processors to execute the method when programmed. Various innovations can be used in combination or individually. The innovations described herein include those covered by the claims. This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the invention will become more apparent from the following detailed description, which continues with reference to the accompanying drawings and illustrates several examples. The examples may also have other and different applications, and some details may be modified in various aspects without departing from the spirit and scope of the disclosed innovations. Attached Figure Description

[0020] The following figures illustrate some features of the disclosed innovation.

[0021] Figure 1 This is a diagram illustrating an example computer system that can implement some of the described embodiments.

[0022] Figure 2This is a diagram illustrating example transport layer processing that can be incorporated to implement some of the described embodiments.

[0023] Figure 3 This is a diagram of an example network that can implement some of the described embodiments.

[0024] Figure 4 This is a diagram illustrating the packet interval based on a transport layer scheduler cycle with variable and coarse-grained iteration duration.

[0025] Figure 5 This is a diagram illustrating the scheduling of grouping intervals based on approximately fine-grained rate adjustments and periodic-grained grouping intervals.

[0026] Figure 6 and Figure 7 These are code segments for the first and second methods, respectively, used for approximate fine-grained rate adjustment and periodic granularity grouping intervals.

[0027] Figure 8 This is a flowchart illustrating a general method for group scheduling with approximate fine-grained rate adjustment and periodic-grained inter-group intervals.

[0028] Figure 9 This is a graph showing the relative frequency of different iteration durations of the scheduler loop for the transport layer scheduling mechanism.

[0029] Figure 10 This is a graph showing the effective throughput for different IPG values ​​based on two different transport layer scheduling mechanisms. Detailed Implementation

[0030] The specific implementation presents innovations in packet scheduling, allowing the scheduling mechanism to adjust packets for a flow at approximately fine-grained rates and periodically granular inter-packet intervals. In some example implementations, the scheduler loop iterations of the transport layer scheduling mechanism have variable and coarse-grained durations. Even so, the sender can specify periodically granular inter-packet intervals. The scheduling mechanism includes operations that schedule packets for transmission at approximately the specified inter-packet intervals. This allows the sender to effectively adjust the transmission rate in a fine-grained manner. In some example implementations, the inter-packet interval can be specified on a per-flow basis, allowing the sender to set different transmission rates for different packet flows.

[0031] For example, in a scheduler loop, the sender processes a given packet stream. Specifically, during iterations of the scheduler loop, the sender can determine whether the next packet of a given packet stream is due (e.g., because the given packet stream is newly added, or because the target next transmission time is earlier than the current time). In some cases, the next packet of a given packet stream is not due, and the sender determines whether a proximity condition is met for the next packet of the given packet stream. The proximity condition depends at least in part on how long after the current time the target next transmission time is. If the next packet is due or if the proximity condition is met for the next packet, the next packet is scheduled for transmission. When the next packet of a given packet stream is scheduled for transmission, the sender transmits the next packet and updates the target next transmission time at least in part based on the target transmission rate (e.g., the target inter-packet gap value). In this way, the scheduling mechanism can schedule packets for transmission in a manner that adjusts the rate at approximately fine granularity and the inter-packet gap at a periodic granularity.

[0032] In the examples described herein, the same reference numerals in different figures indicate the same parts, modules, or operations. More generally, various alternatives to the examples described herein are possible. For example, some methods in the methods described herein can be modified by changing the order of the described method actions, by splitting, repeating, or omitting certain method actions, etc. Various aspects of the disclosed techniques can be used in combination or individually. Some innovations in the innovations described herein solve one or more of the problems mentioned in the background art. Generally, a given technique or tool cannot solve all such problems. It should be understood that other examples can be utilized, and structural, logical, software, hardware, and electrical changes can be made without departing from the scope of this disclosure. Therefore, the following description is not intended to be limiting. I. Example computer system.

[0033] Figure 1 A generalized example of a suitable computer system (100) in which several of the described innovations can be implemented is illustrated. The innovations described herein relate to packet scheduling with approximately fine-grained rate adjustment and periodic-grained inter-packet intervals. The computer system (100) is not intended to impose any limitations on the scope of its use or functionality, as the innovations can be implemented in various computer systems, including dedicated computer systems.

[0034] refer to Figure 1The computer system (100) includes a central processing unit (“CPU”) or one or more processing cores (110…11x) of multiple CPUs and local memory (118). The processing cores (110…11x) are, for example, processing cores on a single chip and execute computer-executable instructions. The number of processing cores (110…11x) depends on the implementation and can be, for example, 4 or 8. The local memory (118) can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, accessible by the respective processing cores (110…11x). Alternatively, the processing cores (110…11x) can be a system-on-a-chip (“SoC”), an application-specific integrated circuit (“ASIC”), or part of other integrated circuits. Figure 1 In this context, the local memory (118) is an on-chip memory (such as one or more caches), for which access operations, transfer operations, etc., using (multiple) processing cores (110 … 11x) are fast.

[0035] The computer system (100) also includes a graphics processing unit (“GPU”) or processing cores (130 … 13x) of multiple GPUs and local memory (138). The number of processing cores (130 … 13x) of the GPU depends on the implementation. The processing cores (130 … 13x) are, for example, portions of a GPU’s Single Instruction Multiple Data (“SIMD”) unit. The SIMD width depends on the implementation. n Indicates the number of elements (sometimes called channels) of a SIMD unit. For example, for an ultrawide SIMD architecture, the number of elements (channels) of a SIMD unit can be 16, 32, 64, or 128. GPU memory (138) can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both, accessible by the corresponding processing core (130 … 13x).

[0036] The computer system (100) includes a main memory (120), which may be volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible to the processing cores (110…11x, 130…13x). Figure 1 In this context, the main memory (120) is an off-chip memory. Access operations and transfer operations using the processing cores (110 … 11x, 130 … 13x) are relatively slow for this off-chip memory.

[0037] More generally, the term "processor" can refer to any device capable of processing computer-executable instructions and can include microprocessors, microcontrollers, programmable logic devices, digital signal processors, and / or other computing devices. A processor can be a CPU, other general-purpose units, or the processing core of a GPU. A processor can also be a special-purpose processor implemented using, for example, an ASIC or a field-programmable gate array ("FPGA").

[0038] The term "control logic" can refer to a controller, or more generally, to one or more processors operable to process computer-executable instructions, determine results, and generate output. Depending on the implementation, control logic can be implemented by software executable on the CPU, by software controlling dedicated hardware (e.g., a GPU or other graphics hardware), or by dedicated hardware (e.g., in an ASIC).

[0039] The computer system (100) includes one or more network interface devices (140), such as network interface cards (“NICs”). The network interface devices (140) enable communication with another computing entity (e.g., a server, other computer system) over a network. In some example implementations, the network interface devices (140) support wired connections for high-performance computer networks. In practice, a network can include thousands, tens of thousands, or even more network interface devices. See below for reference. Figure 3 Example of a network description. Alternatively, the network interface devices (140) may support wired and / or wireless connections for wide area networks, local area networks, personal area networks, or other networks. For example, the network interface devices may include one or more Wi-Fi transceivers, Ethernet ports, cellular transceivers, and / or other types of network interface devices, along with associated drivers, software, etc.

[0040] Multiple network interface devices (140) implement logic or software (141) for packet scheduling with approximate fine-grained rate adjustment and periodic granular inter-packet intervals. For example, one of the network interface devices (140) is implemented using an FPGA that provides logic for packet scheduling with approximate fine-grained rate adjustment and periodic granular inter-packet intervals. Alternatively, one of the network interface devices (140) includes a memory that stores software implementing aspects of packet scheduling with approximate fine-grained rate adjustment and periodic granular inter-packet intervals in the form of firmware or other computer-executable instructions for the FPGA, ASIC, or other processor of the network interface device.

[0041] Multiple network interface devices (140) transmit information (such as computer-executable instructions, arbitrary data from an application, or other data in modulated data signals) through multiple network connections. A modulated data signal is a signal having one or more characteristics set or altered in such a way as to encode information in the signal. By way of example, network connections can use electrical, optical, RF, or other carriers.

[0042] The computer system (100) optionally includes motion sensor / tracker inputs (142) for the motion sensor / tracker, which are capable of tracking the movement of the user and objects around the user. For example, the motion sensor / tracker allows a user (e.g., a player in a game) to interact with the computer system (100) through a natural user interface using gestures and verbal commands. The motion sensor / tracker may combine gesture recognition, facial recognition, and / or voice recognition.

[0043] The computer system (100) optionally includes a game controller input (144) that receives control signals from one or more game controllers via a wired or wireless connection. The control signals are capable of indicating user input from one or more steering wheels, buttons, triggers, and / or one or more joysticks of the game controller. The control signals are also capable of indicating user input from a touchpad or touchscreen, gyroscope, accelerometer, angular rate sensor, magnetometer, and / or other controls or meters of the game controller.

[0044] The computer system (100) optionally includes a media player (146) and a video source (148). The media player (146) can play DVDs, Blu-ray discs, other disc media, and / or other formats of media. The video source (148) can be a camera input that accepts video input in analog or digital form from a video camera that captures natural video. Alternatively, the video source (148) can be a screen capture module (e.g., a driver for an operating system or software that interfaces with an operating system) that provides screen capture content as input. Alternatively, the video source (148) can be a graphics engine that provides texture data to graphics in a computer-represented environment. Alternatively, the video source (148) can be an input video card, a TV tuner card, or other video input (e.g., from a cable input, HDMI input, or other input) that accepts input video in analog or digital form.

[0045] An optional audio source (150) accepts audio input in analog or digital form from a microphone that captures audio or other audio input.

[0046] The computer system (100) optionally includes a video output (160) that provides video output to a display device. The video output (160) may be an HDMI output or other type of output. An optional audio output (160) provides audio output to one or more speakers.

[0047] The storage device (170) may be removable or non-removable and includes magnetic media (such as disks, magnetic tapes or tape cartridges), optical disc media and / or any other media that can be used to store information and can be accessed within a computer system (100).

[0048] The computer system (100) may have additional features. For example, the computer system (100) includes one or more other input devices and / or one or more other output devices. The (multiple) other input devices may be touch input devices (such as a keyboard, mouse, pen, or trackball), scanning devices, or other devices that provide input to the computer system (100). The (multiple) other output devices may be printers, CD burners, or other devices that provide output from the computer system (100).

[0049] Interconnection mechanisms (not shown) (such as buses, controllers, or networks) interconnect the components of the computer system (100). Typically, operating system software (not shown) provides an operating environment for other software running in the computer system (100) and coordinates the activities of the components of the computer system (100).

[0050] Figure 1 The computer system (100) is a physical computer system. Virtual machines can include, for example, Figure 1 The components of the organization shown.

[0051] The terms "application" or "program" can refer to software (such as any user-mode instructions that provide functionality). The software of an application (or program) can also include instructions for an operating system and / or device driver. The software can be stored in associated memory. The software can be, for example, firmware. While it is envisioned that such software can be executed using a suitably programmed general-purpose computer or computing device, it is also envisioned that hardwired circuit systems or custom hardware (e.g., ASICs) can be used in place of or in combination with software instructions. Therefore, the examples described herein are not limited to any particular combination of hardware and software.

[0052] The term "computer-readable medium" refers to any medium that provides data (e.g., instructions) that can be read by a processor and accessed within a computing environment. Computer-readable media can take many forms, including both non-volatile and volatile media. Non-volatile media include, for example, optical discs or magnetic disks and other persistent storage. Volatile media include dynamic random access memory ("DRAM"). Common forms of computer-readable media include, for example, solid-state drives, flash drives, hard disks, any other magnetic media, CD-ROMs, DVDs, any other optical media, RAM, programmable read-only memory ("PROM"), erasable programmable read-only memory ("EPROM"), USB flash drives, any other memory chips or memory cartridges, or any other medium from which a computer can read. The term "non-transitory computer-readable medium" specifically excludes transient propagating signals, carrier waves and waveforms, or other intangible or transient media that can still be read by a computer. The term "carrier wave" can refer to an electromagnetic wave that transmits a signal by amplitude or frequency modulation.

[0053] The innovation is described in the general context of computer-executable instructions that can be executed on a computer system targeting a real or virtual processor. Computer-executable instructions can include instructions executable on the processing core of a general-purpose processor to provide the functionality described herein, instructions executable to control a GPU or dedicated hardware to provide the functionality described herein, instructions executable on the processing core of a GPU to provide the functionality described herein, and / or instructions executable on the processing core of a dedicated processor to provide the functionality described herein. In some implementations, computer-executable instructions can be organized into program modules. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of program modules can be combined or divided among program modules as desired in various embodiments. Computer-executable instructions for program modules can be executed within a local or distributed computer system.

[0054] The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, no term implies any limitation on the type of computer system or device. In general, a computer system or device can be local or distributed and can include any combination of dedicated hardware and / or hardware with software that implements the functions described herein.

[0055] Numerous examples are described in this disclosure and are presented for illustrative purposes only. The described examples are not and are not intended to be limiting in any sense. The innovations disclosed herein are broadly applicable to many contexts, as are apparent from this disclosure. Those skilled in the art will recognize that the disclosed innovations can be practiced with various modifications and alterations, such as structural, logical, software, and electrical modifications. While specific features of the disclosed innovations may be described with reference to one or more particular examples, it should be understood that, unless expressly specified otherwise, such features are not limited to their use in the one or more particular examples described with reference to them. This disclosure is neither a verbatim description of all examples nor a list of inventive features that must be presented in all examples.

[0056] When ordinal numbers (such as "first," "second," "third," etc.) are used as adjectives preceding terms, the ordinal number is used only (unless explicitly specified otherwise) to indicate a particular characteristic (such as distinguishing that particular characteristic from another characteristic described by the same or similar terms). The use of ordinal numbers "first," "second," "third," etc. alone does not indicate any physical order or location, any temporal sequence, or any ranking in terms of importance, quality, or other aspects. Furthermore, the use of ordinal numbers alone does not impose any limitations on the numerical definition of characteristics identified by ordinal numbers.

[0057] When introducing elements, the articles “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more elements among the elements. The terms “comprise” and “have” are intended to be inclusive and mean that additional elements may exist in addition to the listed elements.

[0058] When describing a single device, component, module, or structure, multiple devices, components, modules, or structures (whether or not they cooperate) may be used instead of a single device, component, module, or structure. A function described as having a single device may be possessed by multiple devices, whether or not they cooperate. Similarly, where multiple devices, components, modules, or structures are described herein, whether or not they cooperate, a single device, component, module, or structure may be used instead of multiple devices, components, modules, or structures. A function described as having multiple devices may be possessed by a single device. Generally, a computer system or device may be local or distributed and may include any combination of dedicated hardware and / or hardware with software that implements the functions described herein.

[0059] Furthermore, the techniques and tools described herein are not limited to the specific examples described herein. Rather, the corresponding techniques and tools can be used independently and separately from the other techniques and tools described herein.

[0060] Unless otherwise explicitly specified, devices, components, modules, or structures that communicate with each other do not need to communicate continuously. Instead, such devices, components, modules, or structures only need to transmit data to each other as needed or desired, and in practice, data exchange can be avoided most of the time. For example, a device communicating with another device via the Internet may not transmit data to the other device for several weeks at a time. Furthermore, devices, components, modules, or structures that communicate with each other can communicate directly or indirectly through one or more middleware.

[0061] As used herein, the term "transmit" refers to any manner in which information is conveyed from one device, component, module, or structure to another. The term "receive" refers to any manner in which information is obtained from one device, component, module, or structure at another. A device, component, module, or structure can be part of the same computer system or different computer systems. Information can be passed by value (e.g., as a parameter of a message or function call) or by reference (e.g., in a buffer). Information can be transmitted directly or conveyed through one or more intermediate devices, components, modules, or structures, depending on the context. As used herein, the term "connection" refers to an operational communication link between devices, components, modules, or structures that can be part of the same computer system or different computer systems. An operational communication link can be a wired or wireless network connection, and can be direct or through (e.g., a network) one or more middleware.

[0062] The description of an example with several features does not imply a requirement for all or even any of these features. Rather, a variety of optional features are described to illustrate a wide range of possible examples of the innovation described herein. Unless otherwise explicitly specified, no feature is required or necessary.

[0063] Furthermore, while processing steps and stages can be described in a sequential order, such processing can be configured to operate in different orders. A description of a particular order or sequence does not necessarily indicate a requirement to perform the steps / stages in that order. Steps or stages can be performed in any actual order. Moreover, some steps or stages can be performed simultaneously, although they may be described or implied as not occurring simultaneously. Describing a process as including multiple steps or stages does not imply that all or even any of the steps or stages are necessary or required. Various other examples may omit some or all of the steps or stages described. Unless otherwise explicitly specified, no step or stage is necessary or required. Similarly, while a product may be described as including multiple aspects, qualities, or characteristics, this does not mean that all of them are necessary or required. Various other examples may omit some or all of the aspects, qualities, or characteristics.

[0064] Unless otherwise expressly specified, the list of enumerated items does not imply that any or all of the items are mutually exclusive. Similarly, unless otherwise expressly specified, the list of enumerated items does not imply that any or all of the items are a combination of any category.

[0065] For presentation purposes, the specific implementation uses terms (such as "determine" and "select") to describe computer operations in a computer system. These terms indicate operations performed by one or more processors or other components in the computer system and should not be confused with human actions. The actual computer operations corresponding to these terms vary depending on the implementation. II. Example Transport Protocol.

[0066] The innovations described in this paper relate to transport protocols. Generally, a transport protocol is a set of rules and procedures that govern the exchange of data between computing devices on a computer network. Typically, a sender breaks down data (such as messages or files) into smaller units called packets. The sender then transmits these packets over the computer network to a receiver, which is able to reconstruct the data from the information within the packets. As used herein, the terms "sender" and "receiver" respectively indicate the roles of sending and receiving packets for a stream. Depending on the transport protocol, the receiver may send feedback to the sender (e.g., as an acknowledgment packet). Furthermore, in many cases, a computing device acts as a sender for one stream of packets and as a receiver for different streams of packets.

[0067] Transport protocols interoperate with network protocols at other layers. For example, a transport layer implementation (or transport layer processing) receives data (such as messages or files) from implementations at the application, presentation, or session layers. Transport layer processing provides transport layer packets for the data to network layer implementations capable of implementing Internet Protocol (“IP”) versions. Depending on the transport protocol, transport layer processing can provide features such as error detection, retransmission of dropped transport layer packets, control over the rate at which transport layer packets are transmitted (sometimes called flow control), and sorting of transport layer packets. Transmission Control Protocol (“TCP”) and User Datagram Protocol (“UDP”) are two examples of transport protocols.

[0068] Reliable transport protocols use mechanisms to guarantee, or at least take steps to ensure, the delivery of transport layer packets from sender to receiver. These mechanisms can include error detection, retransmission of dropped packets, and flow control. TCP is an example of a reliable transport protocol. Mechanisms providing reliable delivery of transport layer packets can increase the latency of detecting and retransmitting dropped packets; if many transport layer packets are dropped, the latency can be significant. UDP is an example of an unreliable transport protocol. UDP provides more timely delivery of transport layer packets without the overhead of reliability mechanisms or guaranteed delivery operations.

[0069] In some example implementations, transport layer processing implements a lightweight, reliable, message-based transport protocol. Transport layer processing adds flow control mechanisms, as described in this paper, on top of UDP and uses IP routing. Alternatively, the innovations described in this paper can be combined with another transport protocol.

[0070] Figure 2 An example transport layer process (200) is shown, which, in conjunction with it, enables some of the described embodiments. An application layer entity provides data (210) (such as a message or file) to a sender for transport layer processing. In some example implementations, the data (210) is a Remote Direct Memory Access (“RDMA”) message, for which the data is delivered directly to the receiver’s memory upon receipt. The address offset in memory can be included in the payload or header. Alternatively, the data (210) is another type of data.

[0071] At the transmitter, the transport layer processing divides the data (210) into transport layer stream packets. Specifically, the transport layer processing packetizes the data (210) into multiple payloads for each stream packet. The payloads can have, for example... Figure 2 The size can be uniform or varied. The transport layer processing generates a header for each packet, which is ordered in the packet sequence by the packet sequence number (“PSN”). A given transport layer packet includes a header followed by one of the payloads. The header can include various fields. Typically, the header includes fields indicating the source port of the sender, the destination port of the receiver, the PSN, and the length of the header or the entire transport layer packet, depending on the protocol. One or more flag bits (also called control bits) in the header can indicate the type or attributes of the packet. Typically, the header also includes a checksum for the transport layer packet. The checksum can be calculated by applying a checksum function (e.g., XOR, binary complement sum, or hash function) to data subject to the checksum, which may include the header, payload, and other header information (e.g., for IP routing). The receiver can use the checksum to detect errors introduced during transmission.

[0072] The transmitter transmits corresponding packets of a stream to the receiver over the network. In a typical configuration, traffic for a stream is bursty—a short period of dense traffic for the stream can be followed by a long period of no traffic for the stream. To transmit bursts of stream packets for a stream, the transmitter can transmit packets one after another, where the start of a given packet is separated from the start of the next packet by an inter-packet gap (“IPG”) value (also called the inter-packet interval value). The IPG value does not need to be exactly the same across all stream packets. An idle interval follows the burst. III. Example Network.

[0073] Figure 3 An example network (300) is shown in which some of the described embodiments can be implemented. The example network (300) includes multiple endpoints (301…308) and multiple network switches (311, 312, 321…32p). The network switches are hierarchically organized into level 0 switches (311, 312) and level 1 switches (321…32p). Each level 0 switch (311, 312) is connected to some endpoints of the corresponding endpoints (301…308) via a dedicated link. The dedicated link between the level 0 switches (311, 312) and the endpoints (301…308) is, for example, a high-speed, high-bandwidth wired connection. Each level 0 switch (311, 312) can be connected to 4, 8, 16, or some other number of endpoints. Each level 0 switch (311, 312) is also connected to each level 1 switch (321…32p) via a dedicated link. The network (300) can include 8, 16, 32, or some other number of Tier 1 switches (321…32p). For example, a Tier 0 switch (311) has a first dedicated link to a Tier 1 switch (321), a second dedicated link to a Tier 1 switch (322), a third dedicated link to a Tier 1 switch (323), and so on. The dedicated links between Tier 0 switches (311, 312) and Tier 1 switches (321…32p) are, for example, high-speed, high-bandwidth wired connections.

[0074] exist Figure 3 In this configuration, one endpoint (301) acts as a transmitter for the packet stream, while the other endpoint (306) acts as a receiver for the packet stream. In practice, each endpoint (301 … 308) can act as both a receiver and a transmitter through the network interface device at that endpoint.

[0075] Example network (300) is used for multipath delivery of packets. Transport layer packets (which can be encapsulated as IP packets) from a stream originating from a given sender traverse any and all paths through multiple paths via network (300) to reach a given receiver. For example, transport layer packets from a stream originating from a sender at one endpoint (301) traverse... p Different paths to the receiver at the other endpoint (306). Transport layer packets pass through the first level 0 switch (311) (which is designed for...). p Different paths will divide the transport layer packets across p (Distributed by different Level 1 switches (321 … 32p)) and via a Level 0 switch (312) to a receiver at another endpoint (306). Transport layer packets can be routed through the network (300) along different paths, for example, by adjusting the bits of the destination port field in the header of the corresponding packet. The header bits of the destination port field in the header of a given packet can control which path is used for the given packet. The header bits for the corresponding packet can be adjusted according to the result from the hash function, so that different packets “spray” across different paths. Alternatively, the header bits can be rotated according to a pattern, so that packets are routed on different paths according to that pattern. Traffic between any given combination of transmitter and receiver can be bursty—dense traffic can exist for short periods of time, followed by long periods of no traffic. By delivering transport layer packets along multiple paths of the network (300), the available bandwidth of the network (300) can be used more consistently and evenly.

[0076] In some example implementations, paths in the network (300) are symmetric. At least for paths connecting endpoints to different Level 0 switches, the paths have the same length in terms of hop count, latency, and throughput across the network (300). Alternatively, paths in the network (300) are asymmetric. That is, paths across the network (300) can have different lengths in terms of different hop counts, different latency, and / or different throughput across the network (300).

[0077] Even in networks with symmetrical paths for multipath delivery, the amount of delay along different paths can vary. A switch can be temporarily busy, causing delays in packet delivery through that switch. Due to the different delays along different paths, a receiver can receive packets in a different order than the order in which the sender transmits them. In extreme cases, packets can be dropped due to congestion at a switch. Rarely, packet bits can be affected by bit flipping due to interference, unreliable links, or other reasons, causing packets to be dropped when information loss is detected. The sender can retransmit dropped packets.

[0078] The innovations described herein can be used for flow control incorporating multipath delivery of packets over a network. Alternatively, the innovations described herein can be used for flow control incorporating single-path delivery of packets over a network, wherein all transport layer packets of a flow are delivered along a single path between the sender and receiver. IV. Example of a rate-based flow control mechanism.

[0079] In transport protocols, flow control helps avoid congestion and packet loss. In window-based flow control, the transmission rate of packets from the sender is indirectly affected by adjusting the window size. Conversely, in rate-based flow control, the transmission rate of packets from the sender is directly adjusted.

[0080] The transmission rate can be specified in various ways. In some example implementations, the transmission rate used for rate-based flow control is specified as an inter-packet gap (“IPG”) value, sometimes referred to as the inter-packet interval value. The IPG value indicates the amount of interval between consecutive packets, for example, as the time difference between the start times for transmitting consecutive packets. The transmitter can adapt to network conditions by adjusting the IPG value. For example, the transmitter can reduce the transmission rate by increasing the IPG value in response to network congestion or packet loss. Alternatively, in response to positive feedback or the absence of negative feedback regarding network conditions, the transmitter can decrease the IPG value, thereby increasing the transmission rate.

[0081] When using rate-based flow control, the transmitter and receiver can maintain a sliding window to track which packets have been delivered and identify packets for retransmission. Furthermore, the transmitter monitors the network for feedback on network conditions. The transmitter can infer the presence of network congestion from implicit feedback from the receiver or other entities regarding network conditions such as packet loss, packet delay, or increased round-trip time (“RTT”). The transmitter can also receive explicit feedback on network conditions from network routers or other entities.

[0082] Ideally, the transmitter can adjust the transmission rate in a fine-grained manner to respond to minor changes in network conditions. For example, the transmitter can adjust the per-stream IPG value at the level of a single cycle, schedule packets for transmission at precise cycles that correspond to the per-stream IPG value, and then transmit packets at those precise cycles according to the ideal scheduling.

[0083] In practice, several complexities make it difficult to adjust transmission rates in a fine-grained manner. The primary complexity involves the granularity of the operations. In some implementations, the operations of scheduling packets and adjusting transmission rates are performed as part of a scheduler loop that iterates on a scale of tens or hundreds of cycles. This is coarser-grained compared to specifying the target transmission rate in a fine-grained manner (e.g., with precision for a single cycle of the periodic IPG value). In some example implementations, the scheduler loop iterates over multiple phases of the transport layer scheduling mechanism (as long as the "while(true)" condition is satisfied). Each iteration includes a phase for accepting new messages, a phase for processing acknowledgment metadata (such as ACK packets and selective ACK ("SACK") information) and checking fast retransmission conditions, a phase for bookkeeping tasks, a phase for scheduling and sending packets, and a phase for updating the state. A typical iteration of the scheduler loop requires 50 cycles, but packets can only be transmitted during a portion of the scheduler loop's pipeline. Packets cannot be scheduled for transmission at any arbitrary cycle in the pipeline. Therefore, achieving arbitrarily fine-grained cycle-level IPG values ​​is challenging. It may be possible to precisely schedule packets based on IPG values ​​that are multiples of the typical iteration duration (e.g., sending packets every 150, 200, or 250 cycles for 50 iterations of the scheduler cycle), but it is not possible to precisely schedule packets based on other IPG values.

[0084] Another complexity is that the iteration duration of the scheduler loop can be variable. In some example implementations, certain phases of the scheduler loop (e.g., phases for bookkeeping tasks) have conditional paths that are rarely followed but significantly increase latency when they are. Even though a typical iteration of the scheduler loop takes 50 cycles, other iterations can take as many as 95 cycles. This variability in iteration duration makes precisely scheduling groups based on IPG values ​​at arbitrary cycle granularity more challenging.

[0085] Figure 4 An example (400) of packet intervals based on a transport layer scheduler cycle with variable and coarse-grained iteration durations is shown. A target IPG value (410) specifies the periodic granularity interval between packets. Most iterations of the scheduler cycle have the same iteration duration (e.g., 50 cycles). However, the target IPG value (410) is not an integer multiple of the typical iteration duration, and packets cannot be transmitted at any arbitrary cycle within an iteration. Therefore, in the first packet... p After the transmission begins, no packets are transmitted at the exact period following the target IPG value (410). p +1. Conversely, after the delay of the transmission (420) for the delayed transmission, the delayed transmission packet is transmitted. p+1. The degree of transmission delay may be worse for different target IPG values. Furthermore, the scheduler loop iterations have variable iteration durations—although the iterations... n , n +1、 n +2、 n +3、 n +5, and n +6 has the same duration, but the iterations n +4 has significantly longer iterations. Therefore, for grouping... p The delay of +2 for transmission (422) is even longer.

[0086] Furthermore, the scheduler loop can iterate over streams of multiple active packets, potentially sending packets from any of the multiple streams. This further complicates scheduling packets based on arbitrary periodicity IPG values.

[0087] In typical use cases, the coarse-grained duration of scheduler loop iterations, the variability in the duration of scheduler loop iterations, and the scheduler loop's handling of multiple active flows often prevent the transport layer scheduling mechanism from sending the next packet of a specific flow at a specific period that precisely matches the inter-packet interval for any transmission rate. Thus, in typical use cases, the sender cannot effectively regulate packet flow using fine-grained adjustments to the transmission rate and periodic inter-packet interval. V. Approximate fine-grained rate adjustment and grouping scheduling between periodic granular groups.

[0088] This section describes an innovation in packet scheduling that allows the scheduling mechanism to adjust packet rates at approximately fine-grained intervals and periodically granular inter-packet intervals for a flow. In some example implementations, the scheduler loop iterations of the transport layer scheduling mechanism have variable and coarse-grained durations. Even so, the sender can specify periodically granular inter-packet intervals, for example, using periodically granular IPG values. The scheduling mechanism includes operations that schedule packets for transmission at approximately the specified inter-packet intervals. This allows the sender to efficiently adjust the transmission rate in a fine-grained manner. Moreover, in some example implementations, the inter-packet interval can be specified on a per-flow basis, allowing the sender to set different transmission rates for different packet flows.

[0089] For example, in the scheduler loop, the sender processes a given packet stream. Specifically, in the iterations of the scheduler loop, the sender determines whether the next packet of the stream has expired. The next packet has a target next transmission time. For example, the next packet expires if the target next transmission time is before the current time or if a new packet stream has been added. In some cases, the next packet has not expired, and the sender determines whether the proximity condition for the next packet of the stream is satisfied. Generally, if the target next transmission time (although later than the current time) is close enough to the current time, the proximity condition is satisfied, and the next packet can be scheduled for transmission. If the next packet expires, it is also scheduled for transmission. When the next packet of the stream is scheduled for transmission, the sender transmits the next packet and updates the target next transmission time at least in part based on the target transmission rate. In this way, the scheduling mechanism can schedule packets for transmission in a manner that adjusts the average approximate fine-grained rate and the periodic granularity of the inter-packet interval.

[0090] Figure 5 An example (500) of scheduling based on approximate fine-grained rate adjustment and periodic granularity inter-group intervals is shown. (As in...) Figure 4 In example (400), the target IPG value (410) specifies the periodic granularity interval between packets. The target IPG value (410) is not an integer multiple of the typical iteration duration and cannot transmit packets at any arbitrary period within an iteration. Therefore, in the first packet p After the transmission begins, no packets are transmitted at the exact period following the target IPG value (410). p +1. Conversely, after the delay of the transmission (420) for the delayed transmission, the delayed transmission packet is transmitted. p +1. However, the next packet is transmitted ahead of schedule, before the transmission time specified by the target IPG value (410). p +2. The transmission time specified by the target IPG value (410) and the transmission time for packets. p The difference between the actual transmission time and +2 is the degree of advance transmission (510). Using this method, some packets are delayed (delayed transmission) while others are advanced (early transmission), but on average, packets are separated by the target IPG value (410).

[0091] This section then describes two alternative approaches with corresponding lightweight hardware implementations that allow for approximately fine-grained rate adjustment of the transport layer scheduling mechanism and periodic granular inter-packet intervals. A. The first method for approximate fine-grained adjustment.

[0092] According to the first method, the transport layer scheduling mechanism can approximate fine-grained rate adjustment and periodic inter-packet intervals by probabilistically selecting the iterations of a scheduling cycle in which packets are scheduled for transmission. In the first method, when packets of a packet stream are processed by the transport layer scheduling mechanism, the packets are timestamped with a target "ideal" transmission time according to the target transmission rate. For individual packets, the transmitter probabilistically selects the scheduler iterations in which the packets are transmitted. This results in some packets being sent early (before scheduling according to the ideal transmission time), while other packets are sent late (after scheduling according to the ideal transmission time). Logic controls the ratio of early to late packets for the stream, so that the average transmission rate closely approximates the target transmission rate.

[0093] Figure 6 Code (600) is shown for a first method for approximate fine-grained rate adjustment and periodic-granular inter-packet spacing. The code (600) in the hardware description language describes the behavior and structure of the digital circuitry for the scheduling mechanism. Specifically, code (600) shows the operation of the iterative portion of the scheduler loop, capable of performing these operations when deciding whether to schedule the next packet of a given proprietary flow for transmission. Data structures track characteristics of a given packet flow, including the target next send time (flow.next_send_time) for the next packet of the flow and an indicator of whether the given packet flow has been newly added (flow.newly_added). The variable throttle_gap indicates the target IPG value. Although... Figure 6 The target IPG value can be presented as a separate variable, but conversely, it can also be a characteristic of the data structure for a given packet stream.

[0094] The variable `iteration_window` indicates the time required for the scheduling mechanism to complete one iteration of the scheduler loop. The value of `iteration_window` depends on the implementation. This value can be fixed to a typical iteration duration (such as 50 cycles), even if the iteration duration varies. For example, the value of `iteration_window` can be the average iteration duration, the median iteration duration, or the minimum iteration duration.

[0095] During the scheduler loop iterations, the sender determines a random number within the range of 0 to the variable `iteration_window`. Figure 6 In code (600), the sender calls the function `urandom_range(0, iteration_window)`, which returns an unsigned integer between 0 and `iteration_window`, including both ends of the range. The sender sets the threshold `lfsr_seq` to the returned random number.

[0096] The sender also uses a call to the function `cycles()` to determine the current time (variable `current_time`), which returns a value in terms of cycles. The sender determines the difference (`throttle_compare`) between the current time (`current_time`) and the target next send time (`flow.next_send_time`) for the next packet of the stream. The variable `ready_to_send` indicates whether the next packet has expired. If the target next send time is earlier than the current time, the difference is negative (i.e., `msb(throttle_compare = 1)`), and the next packet has expired (`ready_to_send = 1`). Alternatively, if the stream is newly added (i.e., `flow.newly_added = 1`), the next packet has expired (`ready_to_send = 1`).

[0097] If the next packet expires or if the proximity condition is met, the transmitter schedules the next packet for transmission, setting the variable `transmit_now` to true. For the proximity condition, the transmitter compares (a) the difference between the target next transmission time and the current time (`throttle_compare`) with (b) a threshold (`lfsr_seq`). If the difference is less than the threshold, the proximity condition is met. Therefore, the proximity condition is met if the difference is within a range from zero to a randomly determined value in the variable `iteration_window`.

[0098] If the next packet is marked for transmission (transmit_now is true), the sender transmits the packet by calling the function send_packet(packet). The sender then updates the target next transmission time by adding the target IPG value (throttle_gap) to the packet's transmission time returned by another call to the function cycles(). Finally, the sender sets flow.newly_added to false because a packet has already been transmitted for this stream.

[0099] The following table shows the results of processing the sample packet stream according to the first method. In the table, the scheduler loop iteration duration is 50 cycles for all iterations (the value of `iteration_window` is 50 cycles). The target IPG value is 155 cycles. When a packet is transmitted, the transmission time is 3 cycles after the time checked for scheduling purposes (`current_time`).

[0100] In iteration nIn this context, the target's next transmission time is 6218 cycles, which is after the current time (6160 cycles), therefore the value of `ready_to_send` is false. Because the difference between the target's next transmission time and the current time (6218-6160=58 cycles) is greater than the duration of the iteration window (50 cycles), the proximity condition cannot be satisfied.

[0101] In the next iteration n In +1, the target next send time for the next packet is still after the current time (6210 cycles), so the value of ready_to_send is false. The proximity condition is satisfied if the difference between the target next send time and the current time (6218-6210=8 cycles) is less than a threshold randomly determined (by the RND() function) between 0 and the duration of the iteration window (50 cycles). In the table, the proximity condition is satisfied (shown as ? → Yes), and the packet is transmitted. The target next send time is updated by adding the target IPG value to the transmission time (6213 cycles).

[0102] As shown, additional groups are processed in continuous iterations. In some cases, the proximity condition is checked but not met (displayed as ? → No). In this case, although the difference between the target's next transmission time and the current time is less than the duration of the iteration window, this difference is not less than a threshold randomly determined between 0 and the duration of the iteration window (50 cycles).

[0103] For most groups, the interval between groups is 150 periods. For a pair of consecutive groups, the interval between groups is 200 periods. On average, according to Figure 6 The method shown has an inter-group interval that is very close to the target IPG value (155 cycles). B. A second method for approximate fine-grained rate adjustment.

[0104] According to the second method, by delaying the transmission of some packets to compensate for the early transmission of others, the transport layer scheduling mechanism can approximate fine-grained rate adjustments and periodic inter-packet intervals. As in the first method, when packets of a packet stream are processed by the transport layer scheduling mechanism, the packets are timestamped with the target "ideal" transmission time according to the target transmission rate. When processing a given packet in the current iteration of the scheduler loop, the sender determines the difference between the target next transmission time for the given packet and the current time. If the difference is less than the time required for the current iteration to complete, the sender schedules the given packet for transmission in the current iteration. The target next transmission time for subsequent packets is updated to delay them, thereby compensating for the early transmission of the given packet. In summary, some packets are sent early (before scheduling according to the ideal transmission time), while others are sent late (after scheduling according to the ideal transmission time), but the average transmission rate closely approximates the target transmission rate.

[0105] Figure 7 Code (700) is shown for a second method for approximate fine-grained rate adjustment and periodic-granular inter-packet spacing. The code (700) in the hardware description language describes the behavior and structure of the digital circuitry used for the scheduling mechanism. Specifically, code (700) shows the operation of the iterative portion of the scheduler loop, which is able to perform these operations when deciding whether to schedule the next packet of a given packet flow for transmission. Data structures track characteristics of a given packet flow, including the target next send time of the next packet in the flow (flow.next_send_time) and an indicator of whether the given packet flow has been newly added (flow.newly_added). The variable throttle_gap indicates the target IPG value. Although... Figure 7 The target IPG value can be presented as a separate variable, but conversely, it can also be a characteristic of the data structure for a given packet stream.

[0106] The variable `iteration_window` indicates the time required for the scheduling mechanism to complete one iteration of the scheduler loop. The value of `iteration_window` depends on the implementation. This value can be fixed to a typical iteration duration (such as 50 cycles), even if the iteration duration varies. For example, the value of `iteration_window` can be the average iteration duration, the median iteration duration, or the minimum iteration duration.

[0107] During the scheduler loop iterations, the sender uses a call to the function `cycles()` to determine the current time, which returns a value in terms of cycles. The sender determines the difference (throttle_compare) between the current time (`current_time`) and the target next send time (`flow.next_send_time`) for the next packet of the stream. The variable `ready_to_send` indicates whether the next packet has expired. If the target next send time is earlier than the current time, the difference is negative (i.e., `msb(throttle_compare)` is 1), and the next packet has expired (`ready_to_send` is 1). Alternatively, if the stream is newly added (i.e., `flow.newly_added` is 1), the next packet has expired (`ready_to_send` is 1).

[0108] The sender determines the value of the variable `ipg_comp`, which is used to update the target next send time after the next packet is transmitted. If the next packet expires (i.e., `ready_to_send` is 1), then the value of `ipg_comp` is 0. Otherwise, the value of `ipg_comp` is the difference between the current time and the target next send time of the next packet in the stream (throttle_compare).

[0109] If the next packet expires or if the proximity condition is met, the transmitter schedules the next packet for transmission, thus setting the variable `transmit_now` to true. For the proximity condition, the transmitter compares (a) the difference between the target next transmission time and the current time (`throttle_compare`) with (b) a threshold (`lfsr_seq`), which is the iteration duration (`iteration_window`). If the difference is less than the threshold, the proximity condition is met.

[0110] If the next packet is marked for transmission (transmit_now is true), the sender transmits the packet by calling the function send_packet(packet). The sender then updates the target next transmission time by adding the target IPG value (throttle_gap) to the packet's transmission time returned by another call to the function cycles(). Finally, the sender sets flow.newly_added to false because a packet has already been transmitted for this stream.

[0111] The following table shows the results of processing the sample packet stream according to the second method. In the table, the scheduler loop iteration duration is 50 cycles for all iterations (the value of `iteration_window` is 50 cycles). The target IPG value is 155 cycles. When a packet is transmitted, the transmission time is 3 cycles after the current time (`current_time`) checked for scheduling purposes.

[0112] In iteration n In this context, the next target transmission time is 6218 cycles, which is 6160 cycles after the current time. Therefore, the value of `ready_to_send` is false. Because the difference between the next target transmission time and the current time (6218-6160=58 cycles) is greater than the duration of the iteration window (50 cycles), the proximity condition is not satisfied.

[0113] In the next iteration n In +1, the target next transmission time for the next packet is still after the current time (6210 cycles), so the value of ready_to_send is false. However, because the difference between the target next transmission time and the current time (6218-6210=8 cycles) is less than the duration of the iteration window (50 cycles), the proximity condition is satisfied. The packet is transmitted, and the target next transmission time is updated by adding the target IPG value and the ipg_comp value (26 cycles) to the transmission time (6213 cycles).

[0114] As shown, additional groupings are processed in successive iterations.

[0115] For most grouping cycles, the interval between groups is 150. For a pair of consecutive groups, the interval between groups is 200 cycles. On average, according to Figure 7 The method shown has an inter-group interval that is very close to the target IPG value (155 cycles).

[0116] In the table, each iteration lasts for 50 cycles. Therefore, for most groups, the interval between groups is 150 cycles, but sometimes it is 200 cycles. In some example implementations, the iteration duration can vary, which tends to cause variations in the interval between groups. Even so, according to... Figure 7 The method shown still results in an inter-group interval that is very close to the target IPG value (155 cycles).

[0117] Compared to the first method, the second method is computationally simpler because it does not involve generating random numbers against a threshold to check for proximity conditions. Furthermore, the second method works well when the iteration duration varies, even though the value of the variable `iteration_window` is set to its minimum but not changed for simplicity. However, the second method assumes that adjusting the next transmission time for a single packet is possible, and while it works for some implementations of transport layer scheduling, it may not be possible in other implementations. C. Example techniques for group scheduling.

[0118] Figure 8 A general technique (800) for packet scheduling with approximate fine-grained rate adjustment and periodic-grained inter-packet intervals is illustrated. (See reference...) Figure 1 Or as otherwise described, the network interface device is capable of performing technology (800). Technology (800) operates from the perspective of a transmitter that manages packet scheduling according to a network protocol that implements flow control (e.g., a reliable transport protocol, other transport protocols, network layer protocols).

[0119] Figure 8 The operation of the scheduler loop related to flow control is shown. Other operations of the scheduler loop, such as data packetization, processing acknowledgment feedback from the receiver, and checking timeout conditions or other conditions for the retransmission mechanism, are not shown. In some example implementations, the scheduler loop has multiple phases for the transport layer scheduling mechanism, including: (a) a first phase for accepting new data to be packetized (if any); (b) a second phase for receiving acknowledgment feedback (if any) and checking timeout conditions and / or other conditions for the retransmission mechanism; (c) a third phase for bookkeeping tasks (if any); (d) a fourth phase for scheduling and sending packets (if any); and (e) a fifth phase for updating state information (if appropriate). Packets can only be sent during the fourth phase of the scheduler loop. However, the iterations of the scheduler loop have a significantly longer duration than a single cycle, preventing the sender from sending packets at any arbitrary cycle.

[0120] As another source of complexity, the iteration duration of the scheduler loop can be variable due to conditional paths in some stages. In particular, bookkeeping tasks are skipped in most iterations, but when executed, the duration of the iteration can be increased by a significant but variable amount.

[0121] Figure 8The operation of a scheduler loop for processing a given packet stream is illustrated. In some example implementations, the sender is able to process multiple packet streams within the scheduler loop. During iterations of the scheduler loop, the sender can perform operations on multiple active streams, further complicating the scheduling of packets for transmission.

[0122] refer to Figure 8 The sender checks (810) whether to start an iteration of the scheduler loop (the next iteration). If so, the sender performs various operations for processing a given packet stream during the iteration of the scheduler loop.

[0123] The transmitter can dynamically adjust the target transmission rate in response to network feedback. The transmitter checks (820) whether network feedback has been received. If so, the transmitter sets (822) the target transmission rate based at least in part on the network feedback. For example, in response to network congestion, the transmitter decreases the target transmission rate. Or, in response to a positive assessment of the network condition, the transmitter increases the target transmission rate. The target transmission rate can have an initial value (such as a default value or a value from previous activity) before any network feedback is received.

[0124] The transmission rate can be specified in various ways. In some example implementations, the target transmission rate is a target IPG value, which indicates the amount of interval between consecutive packets, such as the difference between the start times of consecutive packets. To increase the transmission rate, the transmitter can decrease the target IPG value. Alternatively, to decrease the transmission rate, the transmitter can increase the target IPG value. The target IPG value can also be a per-stream target IPG value associated with a given packet stream, in which case different active packet streams can have different per-stream target IPG values.

[0125] To obtain network feedback, the transmitter can monitor the network and receive feedback about network conditions. Network feedback can include explicit feedback from network routers, such as information provided according to the Explicit Congestion Notification (“ECN”) method or another method. Network feedback can also include implicit feedback from network switches or receivers regarding packet loss or packet delay, such as information in ACK packets or SACK messages. Alternatively, network feedback can be information about the round-trip time (“RTT”) for packets. Generally, network feedback can come from network routers, network switches, receivers, and / or another entity, and can provide information about network congestion, packet loss, packet delay, packet latency, and / or other indicators of network conditions. In some example implementations, the scheduler loop operates at a first time scale (e.g., at a nanosecond or tens of nanosecond scale), but adjustments to the target transmission rate occur at a longer time scale (e.g., at a tenth of a second or longer).

[0126] The target IPG value can be set at the granularity of a single cycle within the scheduler loop. In some example implementations, the target IPG value is set in the range of approximately 120 to approximately 300 cycles. For example, the target IPG value could be 128 cycles, 129 cycles, 130 cycles, etc. In contrast, the iteration duration for any given iteration of the scheduler loop is between 50 and 95 cycles, and the average iteration duration of the scheduler loop is slightly longer than 50 cycles. Therefore, the target IPG value is between two and six times longer than the average iteration duration of the scheduler loop.

[0127] refer to Figure 8 In the iteration of the scheduler loop, the transmitter determines (830) whether the next packet of a given packet stream is due. The next packet is due if the target next transmission time is earlier than the current time. To determine whether the next packet is due, the transmitter can simply compare the current time with the target next transmission time. Alternatively, the transmitter can determine the difference between the target next transmission time and the current time, and then consider that difference when determining whether the next packet is due. The next packet of a given packet stream can be considered due for other reasons. For example, if a given packet stream is newly added, the next packet of that given packet stream can be considered due. To check this, the transmitter can determine whether the given packet stream has been newly added. Alternatively, the next packet of a given packet stream can be considered due if no packets of that given packet stream have already been sent. To check this, the transmitter can determine whether any packets of the given packet stream have been sent.

[0128] As part of the scheduler loop iteration, the transmitter determines (840) whether the proximity condition is satisfied for the next packet in a given packet stream. The proximity condition depends at least in part on how long after the current time the target next transmission time is. Generally, if the target next transmission time (although later than the current time) is close enough to the current time, the proximity condition is satisfied, and the next packet can be scheduled for transmission.

[0129] For example, to determine whether a proximity condition is met for the next packet, the transmitter determines the difference between the target next transmission time and the current time. The transmitter also determines a threshold that depends on an estimate of the iteration duration for the scheduler loop (e.g., an estimate of the minimum, median, or average period per iteration of the scheduler loop; or an estimate of the actual period for a particular iteration). In some example implementations, the iteration duration can vary between at least some iterations of the scheduler loop.

[0130] The threshold can be determined in different ways, depending on the implementation. For example, for the first approximation method, the sender determines a random number between 0 and an estimate of the iteration duration for the scheduler loop, and then sets the threshold to the random number. As another example, for the second approximation method, the sender simply sets the threshold to an estimate of the iteration duration for the scheduler loop. Alternatively, the threshold can be determined in another way.

[0131] The transmitter compares the difference to a threshold—the satisfaction of the proximity condition depends on the result of this comparison. For example, if the difference is less than the threshold, the proximity condition is satisfied.

[0132] The transmitter determines (850) whether to schedule the next packet of a given packet stream for transmission. If the next packet expires or if the proximity condition for the next packet is met, the next packet is scheduled for transmission. When the next packet of a given packet stream is scheduled for transmission, the transmitter sends (852) the next packet and updates (854) the target next transmission time at least in part based on the target transmission rate. After updating (880) the target next transmission time, or if the next packet is not scheduled for transmission, the transmitter checks (830) whether to begin the next iteration of the scheduler loop.

[0133] The target next transmission time can be updated in different ways, depending on the implementation. For example, for the first approximation method, the transmitter determines the transmission time and adds the target IPG value to the transmission time. The updated target next transmission time equals the transmission time plus the target IPG value. As another example, for the second approximation method, the transmitter determines the transmission time and also determines whether the difference between the target next transmission time and the current time is greater than zero. If so, the transmitter combines the transmission time, the target IPG value, and the difference. Otherwise, the transmitter adds the target IPG value to the transmission time. The updated target next transmission time then equals the transmission time plus the target IPG value plus (if the difference is greater than zero) the difference.

[0134] Figure 8 This illustrates an operation that dynamically adjusts the target transmission rate as part of the same scheduler loop used for transmitting the next packet of a given packet stream. Alternatively, the operation of dynamically adjusting the target transmission rate can be part of a different processing loop.

[0135] exist Figure 8In this context, the scheduler loop includes determining whether the next packet in a given packet stream has expired, and also determining whether the proximity condition for the next packet has been met. In practice, the operation to determine whether the proximity condition for the next packet has been met can be performed conditionally—the sender performs this operation only after determining that the next packet has not expired. Alternatively, as another alternative, the operation to determine whether the next packet has expired can be performed conditionally—the sender performs this operation only after determining that the proximity condition for the next packet has not been met. D. Results from some example implementations.

[0136] Figure 9 The relative frequency (900) of different iteration durations of the scheduler loop for the transport layer scheduling mechanism is shown in some example implementations. Specifically, Figure 9 This is a histogram showing the relative frequency of different numbers of cycles for iterations of the scheduler loop. The horizontal axis indicates the number of cycles. The vertical axis indicates the frequency of different cycle counts on a logarithmic scale. Most iterations of the scheduler loop have an iteration duration of 50 cycles. A very small number of iterations (less than 2% overall) have durations between 50 and 95 cycles. The average iteration duration is slightly longer than 50 cycles.

[0137] Figure 10 The effective throughput (1000) for different IPG values ​​is shown based on two different transport layer scheduling mechanisms. The first transport layer scheduling mechanism uses the reference above. Figure 7 The second approximation method is described. Effective throughput is shown as "actual throughput" (i.e., useful throughput excluding retransmission overhead). For IPG values ​​ranging from 150 cycles to approximately 275 cycles, the effective throughput (1010) for the first transport layer scheduling mechanism closely tracks the effective throughput for an ideal periodic scheduler. In the example implementation, IPG values ​​from 150 cycles to approximately 275 cycles correspond to transmission rates from 100 Gbps to 50 Gbps.

[0138] For comparison, the second transport layer scheduling mechanism is the baseline mechanism. The second transport layer scheduling mechanism sends the next packet for the packet stream in iterations elapsed after the target next transmission time. For many IPG values, the effective throughput (1020) of the second transport layer scheduling mechanism is significantly worse than that of the first transport layer scheduling mechanism. For an IPG value of 200 cycles, which is an integer multiple of the iteration duration of 50 cycles (most iterations), packets are sent in the optimal iteration. Otherwise, as the IPG value deviates from the 200-cycle IPG value, performance deteriorates with delays in packet transmission. Figure 10The graph shows a sharp drop in performance after 200 cycles of IPG value. A similar "staircase" pattern is expected at 250 cycles of IPG value, but it is not shown because there are no data points between 220 and 260 cycles. E. Technical advantages.

[0139] Leveraging the innovations described in this paper, packet scheduling enables approximately fine-grained rate adjustment and periodic-granular inter-packet intervals for streams. In the example use case, the scheduler loop iterations of the transport layer scheduling mechanism have variable and coarse-grained durations. Even so, the sender can specify periodic-granular inter-packet intervals. The scheduling mechanism includes operations that schedule packets for transmission in a manner approximating the specified inter-packet intervals. This allows the sender to efficiently adjust the transmission rate in a fine-grained manner. F. Alternative.

[0140] In many of the examples described herein, the scheduler in the transport layer scheduling mechanism cyclically adjusts the transport layer packet rates and inter-packet intervals for the flow. The transport layer scheduling mechanism implements a reliable transport protocol. Alternatively, the methods described herein can be implemented for flow control operations in another network protocol, thereby allowing the scheduling mechanism to adjust the packet rates and inter-packet intervals for the flow of that network protocol.

[0141] Given that the principles of the disclosed invention can be applied to many possible embodiments, it should be understood that the illustrated embodiments are merely preferred examples of the invention and should not be considered as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. Therefore, this disclosure claims protection for all inventions falling within the scope and spirit of these claims as disclosed herein.

Claims

1. A group scheduling method in a computer system, the method comprising, during iteration of a scheduler loop: Determine (840) whether the proximity condition is satisfied for the next packet of a given packet stream, the proximity condition depending at least in part on how long after the target next transmission time is the current time; Determine (850) whether to schedule the next packet of the given packet stream for transmission, wherein the next packet of the given packet stream is scheduled for transmission if the next packet of the given packet stream expires or if the proximity condition for the next packet of the given packet stream is satisfied; as well as When the next packet of the given packet stream is scheduled for transmission, the next packet of the given packet stream is sent (852), and the target next transmission time is updated (854) at least in part based on the target transmission rate.

2. The method of claim 1, wherein the target transmission rate is a per-stream target packet gap value associated with the given packet stream.

3. The method according to claim 1 or 2, further comprising: Receive network feedback; The target transmission rate is set (822) based at least in part on the network feedback.

4. The method according to any one of claims 1 to 3, further comprising, in the iteration of the scheduler loop: Determine (830) whether the next packet of the given packet stream has expired, wherein the next packet of the given packet stream has expired if the target next transmission time is earlier than the current time.

5. The method of claim 4, further comprising, in the iteration of the scheduler loop: Determine whether any packet in the given packet stream has been sent, where if no packet in the given packet stream has been sent, then the next packet in the given packet stream has expired; or Determine whether the given packet stream has been newly added, wherein if the given packet stream has been newly added, the next packet of the given packet stream expires.

6. The method according to any one of claims 1 to 5, further comprising, in the iteration of the scheduler loop: Determine the difference between the next transmission time of the target and the current time; A threshold is determined, which depends on an estimate of the iteration duration for the scheduler loop; as well as The difference is compared with the threshold, wherein the satisfaction of the proximity condition depends on the result of the comparison between the difference and the threshold.

7. The method of claim 6, wherein determining the threshold comprises: Determine a random number between 0 and the estimate of the iteration duration for the scheduler loop; as well as Set the threshold to the random number.

8. The method of claim 7, wherein the target transmission rate is a target inter-packet gap value, and updating the target next transmission time comprises: Determine the transmission time; as well as Add the target inter-packet gap value to the transmission time.

9. The method of claim 6, wherein determining the threshold comprises: Set the threshold as the estimate of the iteration duration for the scheduler loop.

10. The method of claim 9, wherein the target transmission rate is a target inter-packet gap value, and wherein updating the target next transmission time comprises: Determine the transmission time; Determine whether the difference is greater than zero; If the difference is greater than zero, then the transmission time, the target inter-packet gap value, and the difference are combined. as well as Otherwise, the transmission time is combined with the target packet gap value.

11. The method according to any one of claims 1 to 10, wherein the scheduler loop is part of a transport layer scheduling mechanism, wherein the iteration of the scheduler loop has multiple phases, the multiple phases including: (a) The first stage for accepting new data (if any); (b) a second phase for receiving acknowledgment feedback (if present) and checking the conditions for the retransmission mechanism; (c) a third phase for bookkeeping tasks (if present); (d) a fourth phase for scheduling and sending packets (if present); and (e) a fifth phase for updating status information (if appropriate), wherein the iteration duration of the scheduler loop is variable due to the conditional path in at least one of the plurality of phases.

12. The method according to any one of claims 1 to 11, wherein the data structure tracks the characteristics of the given packet stream, the characteristics including one or more of the following: the target next transmission time, an indicator of whether the given packet stream has been newly added, and the target transmission rate.

13. The method according to any one of claims 1 to 12, further comprising: In the scheduler loop, multiple packet flows, including the given packet flow, are processed, including performing operations on the multiple packet flows in the iterations of the scheduler loop.

14. One or more computer-readable media having stored thereon computer-executable instructions, the computer-executable instructions being programmed to cause one or more processing units to perform operations for group scheduling, the operations including: Determine (840) whether the proximity condition is satisfied for the next packet of a given packet stream, the proximity condition depending at least in part on how long after the target next transmission time is the current time; Determine (850) whether to schedule the next packet of the given packet stream for transmission, wherein the next packet of the given packet stream is scheduled for transmission if the next packet of the given packet stream expires or if the proximity condition for the next packet of the given packet stream is satisfied; as well as When the next packet of the given packet stream is scheduled for transmission, the next packet of the given packet stream is sent (852), and the target next transmission time is updated (854) at least in part based on the target transmission rate.

15. A network interface device configured to perform operations for packet scheduling, the operations including: Determine (840) whether the proximity condition is satisfied for the next packet of a given packet stream, the proximity condition depending at least in part on how long after the target next transmission time is the current time; Determine (850) whether to schedule the next packet of the given packet stream for transmission, wherein the next packet of the given packet stream is scheduled for transmission if the next packet of the given packet stream expires or if the proximity condition for the next packet of the given packet stream is satisfied; as well as When the next packet of the given packet stream is scheduled for transmission, the next packet of the given packet stream is sent (852), and the target next transmission time is updated (854) at least in part based on the target transmission rate.

Images