MULTISTREAM PLANNING FOR TIME-SENSITIVE NETWORKING

Description

TECHNICAL FIELD

[0001] Embodiments described herein generally relate to data communication systems and in particular to a multi-stream scheduling for time sensitive networking.BACKGROUND

[0002] Currently, network cards transmit and receive data packets. As network use grows and additional systems come online to serve more data to more end users, data communication services need to become faster and more efficient. At the network card level, effective and deterministic packet processing is needed to increase throughput in a network.

[0003] US 2021 / 117353 A1 (KASICHAINULA KISHORE [US] ET AL) 22 April 2021 (2021-04-22) discloses a method to transmit and / or receive data streams with a network interface controller. An application 104 generates and transmits a descriptor. The descriptor includes a launch time and poll time offset (e.g. from a launch time). The application transmits data transmission information (e.g. reduction ratio, traffic class information) how the data streams should be scheduled. A descriptor cache 203 stores the descriptors.

[0004] US 2021 / 097019 A1 (KASICHAINULA KISHORE [US]) 1 April 2021 (2021-04-01) discloses computer networking components that use a local look-up table called enhanced gate control list thereby reducing latencies in data retrieval and transfer. The use of a local lookup table can also be split into multiple queues within the lookup table where each queue corresponds to a traffic class. Using the multiple traffic class queues allows precise per packet per traffic class scheduling and improved overall cycle jitter as offsets can be incorporated into transmission scheduling. The EGCL includes of memory address, size of payload, pre-fetch time and other control parameters to achieve low latency and determinism. Once the EGCL is programmed, the NIC can perform direct memory accesses (DMAs) of the data at predetermined time, called pre-fetch time, before the launch time of the packet. Using direct memory accesses (DMAs) allows the payload of the data packet to be directly fetched without going through descriptors or the CPU.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which: FIG. 1 is a schematic diagram illustrating an operating environment, according to an embodiment; FIG. 2 is a is a schematic diagram illustrating a network interface device (NID) for time sensitive networking (TSN), according to an embodiment; FIG. 3 is a schematic diagram illustrating transmit processing, according to an embodiment; FIG. 4 is a schematic diagram illustrating transmit processing of multiple streams, according to an embodiment; FIG. 5 is a schematic diagram illustrating transmit processing of multiple streams with a launch time offset, according to an embodiment; FIG. 6 is a block diagram illustrating a descriptor, according to an embodiment; FIG. 7 is a flowchart illustrating a method for multi-stream scheduling at a network interface device, according to an embodiment; and FIG. 8 is a block diagram illustrating an example machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform, according to an example embodiment. DETAILED DESCRIPTION

[0006] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of some example embodiments. It will be evident, however, to one skilled in the art that the present disclosure may be practiced without these specific details.

[0007] The emerging IEEE standards for deterministic networking referred to collectively as Time Sensitive Networking (TSN) provide extremely precise data transfer across the network. As a result, embedded designs in the industrial and automotive space are increasingly integrating TSN controllers. The inevitable convergence of information technology (IT) and operational technology (OT), workload consolidation, and demand for seamless communication across many connected devices necessitates transfer of various types of data traffic through a network interface card (NIC) without sacrificing the latencies for hard real-time applications.

[0008] To address these different types of traffic, the IEEE 802.1Q standard defined eight traffic classes with corresponding priorities. To support these eight traffic classes, the current generation of NICs implemented eight transmit (Tx) and eight receive (Rx) queues with each pair of transmit and receive queues dedicated to one of the eight traffic classes.

[0009] The IEEE standards, such as 802.1Qbv, defined a gate control list (GCL) for precisely scheduling streams based on traffic class. Hence, NICs designed based on these standards are capable of transmitting (or receiving) multiple streams with different traffic classes and priorities. However, such NICs can only transmit (or receive) one stream for a given traffic class and priority. In other words, only eight data streams, each belonging to a different traffic class, can be scheduled by the current NICs. This is because there is only gate per traffic class and the GCL is based on the traffic class and not based on the number of data streams.

[0010] In industrial applications, there may be multiple data streams belonging to same traffic class. In this situation, to schedule these multiple streams belonging to same traffic class, the application software or middleware must implement an additional software scheduler to merge and sequence all streams that belong to same traffic type and priority into a single stream, and then into a single descriptor ring so that the hardware can schedule these streams at appropriate launch times.

[0011] Though the current standards provide QoS based on traffic class, NICs designed to these standards are not capable of scheduling multiple streams per traffic class directly by the NIC. The additional merging and sequencing of multiple streams of same traffic class by the software is a huge overhead to the application and it steals a considerable amount of CPU cycles. It also contributes to software induced latency and jitter which is not desirable for hard real-time applications.

[0012] What is needed is a mechanism to schedule multiple streams per traffic class directly by the NIC. The systems and mechanisms described here provide multiple stream support with time-based scheduling and packet latencies less than 10µs. Descriptors are modified to add special parameters to achieve this. These parameters are added to precisely fetch data and transmit the packets. Variables like reduction ratios, phase offset, gating cycle number, and launch time are defined in addition to the traffic class of a particular stream to provide more fine-grained scheduling control.

[0013] The systems and methods described here eliminate software overhead, offload CPU for other tasks, and improve the latency and jitter. It also provides more area on a package or die, and is more power efficient. The architecture has dedicated DMA channels and prefetch cache for each data stream, but the packet buffers are still based on the traffic class. Only eight transmit and eight receive packet buffers are needed. The descriptor parser is enhanced and has an additional scheduler to arbitrate and sequence data streams before the data streams are pushed into transmit packet buffers, which then are scheduled using any IEEE standard-defined scheduling algorithm. The descriptors are modified to add three new fields called "launch time offset," "gating cycle offset," and "reduction ratio." The additional scheduler in the DMA decodes these newly defined fields in the descriptor and pushes the data packets into corresponding packet queues at the appropriate time based on the parameters.

[0014] A gating cycle is a time during which all entries in a gate control list are processed. The entries in the gate control list indicate which traffic class queue is to be transmitted and for how long. In the examples shown here, the duration of the gating cycle is divided equally amongst the traffic classes in the gate control list. However, it is understood that any allotment of time between traffic classes may be used. The launch time offset is used to delay a packet transmission within a gating cycle. The gating cycle offset is used to specify how many gating cycles to delay the initial packet from a data stream before the data stream is transmitted. The reduction ratio provides a way to control the repeat interval of a packet stream. A reduction ratio of " 1" indicates that packets from the stream are to be transmitted every gating cycle, "2" indicates that packets are to be transmitted every two gating cycles (e.g., every other gating cycle), "4" indicates that the packets are to be transmitted every four gating cycles, etc. Using these parameters, the packet streams can be scheduled with fine granularity. Additional details are provided below.

[0015] FIG. 1 is a schematic diagram illustrating an operating environment 100, according to an embodiment. The operating environment 100 may be a server computer, desktop computer, laptop, wearable device, hybrid device, onboard vehicle system, network switch, network router, or other compute device capable of receiving and processing network traffic. The operating environment 100 includes a network interface device (NID) 102. The NID 102 includes electronic circuity to support the data link layer with the physical layer. In particular, the NID 102 is able to receive data using an interconnect 104 or radio 106. The interconnect 104 is arranged to accept signals over a physical media, where the signals are arranged into some supported L2 framing, and interpret the incoming signal stream as a stream of bits organized into L2 units called "frames." The interconnect 104 may be an Ethernet port, for example. The radio 106 is able to send and receive radio frequency (RF) data and is used to communicate over wireless protocols, such as WiFi, Bluetooth, Zigbee, cellular communications, and the like. Other types of communication interfaces may be supported by NID 102, such as Gigabit Ethernet, ATM, HSSI, POS, FDDI, FTTH, and the like. In these cases, appropriate ports may be provided in the NID architecture.

[0016] The NID 102 includes circuitry, such as a packet parser 108 and a scheduler circuit 110. The packet parser 108 and the scheduler circuit 110 may use NID memory 112 or main memory 114 for various operations such as queuing packets, saving state data, storing historical data, supporting a neural network, or the like.

[0017] The NID 102 also includes a direct memory access (DMA) circuit 122 and media access control (MAC) circuit 124 (also referred to as medium access control (MAC)). The DMA circuit 122 may be used to access main memory 114 through a fabric (e.g., Intel On-Chip System Fabric (IOSF)). The DMA circuit 122 interfaces with the MAC circuit 124 to prepare frames for transmission. The MAC circuit 124 is able to perform: frame delimiting and recognition; addressing of destination stations (both as individual stations and as groups of stations), conveyance of source-station addressing information, provide transparent data transfer of LLC PDUs or of equivalent information in the Ethernet sublayer, protection against errors, generally by means of generating and checking frame check sequences, and control of access to the physical transmission medium. In the case of Ethernet, the functions required of a MAC circuit 124 is to: receive / transmit normal frames; provide half-duplex retransmission and backoff functions; append / check FCS (frame check sequence); enforce interframe gap; discard malformed frames; prepend(tx) / remove(rx) preamble, SFD (start frame delimiter), and padding; and provide half-duplex compatibility: append(tx) / remove(rx) MAC address.

[0018] The packet parser 108, scheduler circuit 110, DMA circuit 122, and MAC circuit 124 may be implemented using an on-NID CPU 111, an ASIC, an FPGA, or other type of computing unit on the NID 102. Further, portions of the packet parser 108, scheduler circuit 110, DMA circuit 122, and MAC circuit 124 may be incorporated into common circuitry, on a same die, or virtualized. It is understood that various arrangements of these components may be used according to available power, area, design, or other factors.

[0019] The operating environment 100 also includes central processing unit (CPU) cores 150A, 150B, 150C, and 150N (collectively referred to as 150A-N). Although four cores are illustrated in FIG. 1, it is understood that more or fewer cores may exist in particular CPU architectures. Additionally, there may be multiple CPUs logically grouped together to create a CPU complex. Mechanisms described herein may be used for a single-core CPU, a multi-core CPU, or multiple CPUs acting in concert.

[0020] The NID 102 may communicate with the cores 150A-N, main memory 114, or other portions of operating environment 100 via a suitable interconnect channel, such as Peripheral Component Interconnect Express (PCIe) connector 116. PCIe connector 116 may be of any width (e.g., x1, x4, x12, x16, or x32). Other interconnect channels include Intel On-Chip System Fabric (IOSF), QuickPath Interconnect (QPI), and Primary Scalable Fabric (PSF).

[0021] The NID 102 may communicate with cores 150A-N over a bus, such as a PCIe bus. A PCIe client 115 controls the bus and the PCIe connector 116 in the NID 102 that interfaces with a bus controller 118. The PCIe client 115 may perform additional functions, such as controlling allocation of internal resources to virtual domains, support various forms of I / O virtualization (e.g., single root input / output virtualization (SR-IOV)), and other functions. The PCIe bus controller 118 may be incorporated into the same die that includes the cores 150AN. A platform controller hub may include the PCIe bus controller 118, memory management unit (MMU) 120, Serial ATA controllers, Universal Serial Bus (USB) controllers, clock controller, trusted platform module (TPM), serial-peripheral interface (SPI), and other components in the processor die.

[0022] Modern processor architectures have multiple levels in the cache hierarchy before going to main memory. In many designs the outermost level of cache is shared by all cores on the same physical chip (e.g., in the same package) while the innermost cache levels are per core.

[0023] In the example illustrated in FIG. 1, each CPU core 150A-N includes a corresponding L1 cache, separated into an L1 instruction cache 152A, 152B, 152C, 152N (collectively referred to as 152A-N) and an L1 data cache 154A, 154B, 154C, 154N (collectively referred to as 154A-N). The cores 150A-N also each include an L2 cache 156A, 156B, 156C, 156N (collectively referred to as 156A-N). The size of the L1 caches and L2 caches vary depending on the processor design. Conventional sizes range from 32KB to 64KB for L1 cache size (e.g., 16KB instruction and 16KB data, or 32KB instruction and 32KB data), and 256KB to 512KB for L2 cache size. L3 cache size may vary from 8MB to 12MB or more.

[0024] FIG. 2 is a schematic diagram illustrating a NID 200 for time sensitive networking (TSN), according to an embodiment. The NID 200 includes a DMA circuitry 202 and MAC circuitry 204 to transmit packets in a time sensitive network environment. The DMA circuitry 202 includes a multichannel transmit DMA engine 210, a reduction ratio based scheduler 212, and a descriptor parsing circuitry 214. The MAC circuitry 204 includes a Precision Time Protocol (PTP) timer 220 and a gate control list (GCL) based scheduler 222.

[0025] A transmit descriptor cache 216 is used by the multichannel transmit DMA engine 210 and the descriptor parsing circuitry 214. The transmit descriptor cache 230 includes a separate queue for each channel. In an embodiment, 128 transmit channels and 128 receive channels supported by the DMA engine 210. It is understood that there may be more or fewer transmit and receive channels. Each DMA channel processes one ring buffer and each ring buffer is mapped to one of the eight traffic classes (TC0-TC7) as defined by a user.

[0026] The descriptors are prefetched and stored in a descriptors cache 230 to reduce end-to-end transmit latency. Each channel has an associated descriptor prefetch cache 232A, 232B,..., 232N (collectively referred to as 232A-N).

[0027] The DMA engine 210 prefetches all the available descriptors in a burst when the tail pointer 240 of a transmit packet buffer 242A, 242B,..., 242N (collectively referred to as 242A-N) is advanced. Each transmit buffer 242A-N has a corresponding tail pointer that points to the buffer entry after the last valid buffer entry. Once the descriptors are prefetched into the descriptor cache 230 , the descriptor parsing circuitry 214 decodes the fields in the descriptors and passes these values to the reduction ratio based scheduler 212. The descriptor fields include those described above: launch time offset, gating cycle offset, and reduction ratio.

[0028] The reduction ratio based scheduler 212 performs two functions for every packet that needs to be transmitted. First, it computes the gating cycle number and second, it computes the launch time. The gating cycle number specifies the cycle the MAC circuitry 204 should transmit a particular packet. The launch time specifies the precise time the packet should be transmitted within that gating cycle. The reduction ratio based scheduler 212 communicates the gating cycle number and launch time to the MAC circuitry 204.

[0029] The PTP timer 220 is used to synchronize clocks with the reduction ratio based scheduler 212. The MAC circuitry 204 provides the base time and cycle time to the reduction ratio based scheduler 212. The base time is the beginning time of the gating cycle. A gating cycle is a period and used to cycle through active classes in the gate control list 250. The cycle time is the duration of the gating cycle. The cycle time used in these examples is 100µs; however it is understood that any duration may be used as the cycle time.

[0030] The GCL-based scheduler 222 uses a gate control list 224 to determine from which ring buffer to read and transmit packets from. Because ring buffers are associated with traffic classes, the GCL-based scheduler 222 effectively schedules per traffic class from several ring buffers. This is described more in FIGS. 3-5 below.

[0031] FIG. 3 is a schematic diagram illustrating transmit processing, according to an embodiment. In FIG. 3, the NID is configured to transmit one stream per traffic class. These streams are transmitted every gating cycle. In current architectures, there is no concept of reduction ratios. Here, the reduction ratio may be considered to be one, such that every stream is transmitted on every gating cycle.

[0032] For the examples illustrated in FIGS. 3-5, the gating cycle duration is assumed to be 100µs. It is understood that the gating cycle may be set to a different duration.

[0033] In FIG. 3, there are only two streams set to be transmitted in the gate control list 300. A bit map is used to indicate which traffic class is active. In the example shown, each traffic class is given a substantially equal amount of time during the gating cycle to transmit packets of that class.

[0034] Ring buffers for traffic class seven (TC7) 302A and traffic class six (TC6) 302B are illustrated. Ring buffers are data structures in the application or middleware memory space and are used to store packets for the data stream. Each ring buffer uses a head pointer and a tail pointer to indicate the start and end of the queue. A tail pointer is used for each ring buffer 302A, 302B indicating the queue position for the next packet to be inserted into the ring buffer. As the application stores data in a ring buffer, it updates the tail pointer to point to the entry after the last valid buffer entry.

[0035] As packets are dequeued from the ring buffer, the head pointer is advanced. The NID fetches this data from the main memory into its local packet buffer, packetizes the data, and then transmits it in an appropriate gating cycle. As the NID fetches data from ring buffer, it advances the head pointer. The head pointer can be advanced until it reaches the tail pointer.

[0036] The packets are stored in the transmit packet buffer for the corresponding traffic class. Packets are transmitted during the gating cycle with the TC7 ring buffer 302A being processed first during the cycle as it has higher priority than the packets in the TC6 ring buffer 302B. Each ring buffer 302A, 302B has 50µs of transmit time.

[0037] As evident in FIG. 3, if multiple streams belonging to same traffic class need to be transmitted, then an application must pre-process streams before it creates the ring buffer. The application would have to put multiple streams of packets in the same ring buffer and sequence them appropriately. For example in an implementation, when there are multiple streams having the same reduction ratio, the NID can schedule these streams in a round-robin fashion.

[0038] FIG. 4 is a schematic diagram illustrating transmit processing of multiple streams, according to an embodiment. In contrast to the example illustrated in FIG. 3, FIG. 4 includes two data streams per traffic class (although more streams may be used). In FIG. 4, traffic class seven (TC7) has two data streams 402A, 402B and hence, two ring buffers 404A and 404B. Traffic class six (TC6) has two data streams 402C, 402D and corresponding ring buffers 404C, 404D. Note that there is no change to the gate control list. This is because the gate control list is based on the traffic class and not the data streams. The gate control list merely provides that a traffic class gate is open for a particular time period. The gate control list does not specify the stream that is transmitted during a particular traffic class period. Instead, the reduction ratio based scheduler is used to schedule the streams based on the packet descriptors.

[0039] As described above in FIG. 1, the reduction ratio based scheduler obtains the parameters from the packet descriptors including the gating cycle offset, the reduction ratio, and the launch time offset. For each data packet placed in a given ring buffer, the reduction ratio based scheduler calculates the cycle number using the following equation: Cycle n + 1 = Cycle n + Reduction Ratio + Gating Cycle Offset where n is a positive integer.

[0040] As can be understood, given a current cycle number n, the next cycle that a packet from the data stream will be transmitted is offset by the Reduction Ratio. The Gating Cycle Offset is used to offset the entire sequence by some number of gating cycles. In effect, the Gating Cycle Offset delays the first packet from the stream to a gating cycle that occurs later than the current gating cycle. The Gating Cycle Offset is only used to offset the initial transmission. The Reduction Ratio can be used to offset packet transmission using a periodic pattern after the first packet in the stream is transmitted.

[0041] The packets may be stored in the appropriate transmit packet buffer based on the traffic class of the packet. The transmit packet buffers may be stored, sorted, or populated by cycle number so that packets that are to be transmitted in a given cycle are consecutively ordered in the ring buffer. Streams from a ring buffer may not be repeated every gating cycle. Cyclic streams that are repeated over a periodic gating cycle pattern, such as every gating cycle, every other gating cycle, every third gating cycle, etc. Acyclic streams are those streams that are not repeated with a regular periodicity.

[0042] In the example illustrated in FIG. 4, during "gating cycle 1," packets from TC7 are transmitted from each of the ring buffers 404A and 404B for 50µs of the 100µs gating cycle period. During the next 50µs packets from TC6 are transmitted from each of the ring buffers 404C and 404D.

[0043] In the next "gating cycle 2," cyclic data stream packets from TC7 are transmitted from only ring buffer 404A. The ring buffer 404B has packets from a cyclic stream with a reduction ratio of "2," which means that packets from this stream are transmitted once every two cycles (alternating active cycles). Similarly, cyclic stream packets from TC6 are transmitted from only ring buffer 404C, which has a reduction ratio of "1." Those packets in ring buffer 404D are skipped in "gating cycle 2." In "gating cycle 3," packets from all of the ring buffers are transmitted. The gating cycle offset of all ring buffers is assumed to be zero.

[0044] FIG. 5 is a schematic diagram illustrating transmit processing of multiple streams with a launch time offset, according to an embodiment. The third parameter called "launch time offset" is specified with respective to each gating cycle. This parameter is obtained from the descriptor and is directly passed to the MAC circuitry for precise scheduling of a packet. FIG. 5 illustrates how the launch time offset can shift the packet launch time with respect to a base time. The launch time offset indicates that the packet is to be transmitted at a time equal to or later than the launch time offset from the base time in a gating cycle. In the example illustrated in FIG. 5, the launch time offset is different for each of TC7 and TC6 packets.

[0045] Base time is defined as the starting time of execution of entries in the gate control list. Each time the MAC circuitry executes the entire gate control list is called gate cycle time. The DMA reduction ratio based scheduler uses a PTP timestamp value to know the current time and compares it against the gating cycle base time. The reduction ratio based scheduler pushes the packets into the MAC packet queues accordingly. The base time is updated at the end of every gate cycle by the MAC circuitry.

[0046] FIG. 6 is a block diagram illustrating a descriptor 600, according to an embodiment. The descriptor 600 is a 32-bit double-word structure. Reserved fields are used to add parameters. The launch time offset is a 32-bit field and specifies a number of nanoseconds to offset a transmission within a gating cycle. The reduction ratio is an 8-bit field. This is specified in 2^n format (e.g., a bit map) with bits indicating a 1, 2, 4, 8, 16, 32, 64, or 128 periodic gating cycle. The gating cycle offset is a 8-bit field and is specified in terms of gating cycles, e.g., 0, 1, 2, etc.

[0047] FIG. 7 is a flowchart illustrating a method 700 for multi-stream scheduling at a network interface device, according to an embodiment. At 702, a packet descriptor is read from a descriptor cache. The packet descriptor includes at least one scheduling control parameter including: a launch time offset, a gate cycle offset, or a reduction ratio. The packet descriptor is associated with a packet stream having a traffic class.

[0048] At 704, packets from the packet stream are scheduled for transmission using the at least one scheduling control parameter.

[0049] In an embodiment, scheduling the packet includes storing the packet in a transmit queue for the traffic class. In a further embodiment, the network interface device includes one transmit queue for each traffic class of a plurality of traffic classes.

[0050] In an embodiment, the launch time offset indicates an offset from a base time of a gating cycle. When using this scheduling control parameter, scheduling packets from the packet stream includes scheduling packets to transmit at a time equal to or later than the offset from the base time during each gating cycle.

[0051] In an embodiment, the gate cycle offset indicates an offset from a current gating cycle. When using this scheduling control parameter, scheduling packets from the packet stream includes scheduling packets to begin transmitting at a gating cycle that occurs later than the current gating cycle.

[0052] In an embodiment, the reduction ratio indicates an gating cycle interval. When using this scheduling control parameter, scheduling packets from the packet stream includes scheduling packets to transmit only on gating cycles that correspond to the gating cycle interval.

[0053] In an embodiment, the method 700 includes transmitting packets from a transmit queue according to a gate control list where the packets are scheduled according to the scheduling control parameters in the packet descriptor. In a further embodiment, the gate control list includes a plurality of entries that control which traffic class of a plurality of traffic classes is able to transmit during a gating cycle.Hardware Platform

[0054] Embodiments may be implemented in one or a combination of hardware, firmware, and software. Embodiments may also be implemented as instructions stored on a machine-readable storage device, which may be read and executed by at least one processor to perform the operations described herein. A machine-readable storage device may include any non-transitory mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable storage device may include read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, and other storage devices and media.

[0055] A processor subsystem may be used to execute the instructions on the machine-readable medium. The processor subsystem may include one or more processors, each with one or more cores. Additionally, the processor subsystem may be disposed on one or more physical devices. The processor subsystem may include one or more specialized processors, such as a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or a fixed function processor.

[0056] Examples, as described herein, may include, or may operate on, logic or a number of components, modules, or mechanisms. Modules may be hardware, software, or firmware communicatively coupled to one or more processors in order to carry out the operations described herein. Modules may be hardware modules, and as such modules may be considered tangible entities capable of performing specified operations and may be configured or arranged in a certain manner. In an example, circuits may be arranged (e.g., internally or with respect to external entities such as other circuits) in a specified manner as a module. In an example, the whole or part of one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware processors may be configured by firmware or software (e.g., instructions, an application portion, or an application) as a module that operates to perform specified operations. In an example, the software may reside on a machine-readable medium. In an example, the software, when executed by the underlying hardware of the module, causes the hardware to perform the specified operations. Accordingly, the term hardware module is understood to encompass a tangible entity, be that an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., transitorily) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any operation described herein. Considering examples in which modules are temporarily configured, each of the modules need not be instantiated at any one moment in time. For example, where the modules comprise a general-purpose hardware processor configured using software; the general-purpose hardware processor may be configured as respective different modules at different times. Software may accordingly configure a hardware processor, for example, to constitute a particular module at one instance of time and to constitute a different module at a different instance of time. Modules may also be software or firmware modules, which operate to perform the methodologies described herein.

[0057] Circuitry or circuits, as used in this document, may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and / or firmware that stores instructions executed by programmable circuitry. The circuits, circuitry, or modules may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc.

[0058] FIG. 8 is a block diagram illustrating a machine in the example form of a computer system 800, within which a set or sequence of instructions may be executed to cause the machine to perform any one of the methodologies discussed herein, according to an example embodiment. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of either a server or a client machine in server-client network environments, or it may act as a peer machine in peer-to-peer (or distributed) network environments. The machine may be a wearable device, personal computer (PC), a tablet PC, a hybrid tablet, a personal digital assistant (PDA), a mobile telephone, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. Similarly, the term "processor-based system" shall be taken to include any set of one or more machines that are controlled by or operated by a processor (e.g., a computer) to individually or jointly execute instructions to perform any one or more of the methodologies discussed herein.

[0059] Example computer system 800 includes at least one processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both, processor cores, compute nodes, etc.), at least one co-processor 803(e.g., FPGA, specialized GPU, ASIC, etc.), a main memory 804 and a static memory 806, which communicate with each other via a link 808 (e.g., bus). The computer system 800 may further include a video display unit 810, an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In one embodiment, the video display unit 810, input device 812 and UI navigation device 814 are incorporated into a touch screen display. The computer system 800 may additionally include a storage device 816 (e.g., a drive unit), a signal generation device 818 (e.g., a speaker), a network interface device 820, and one or more sensors (not shown), such as a global positioning system (GPS) sensor, compass, accelerometer, gyrometer, magnetometer, or other sensor.

[0060] The storage device 816 includes a machine-readable medium 822 on which is stored one or more sets of data structures and instructions 824 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 824 may also reside, completely or at least partially, within the main memory 804, static memory 806, and / or within the processor 802 during execution thereof by the computer system 800, with the main memory 804, static memory 806, and the processor 802 also constituting machine-readable media.

[0061] While the machine-readable medium 822 is illustrated in an example embodiment to be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more instructions 824. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including but not limited to, by way of example, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks.

[0062] The instructions 824 may further be transmitted or received over a communications network 826 using a transmission medium via the network interface device 820 utilizing any one of a number of well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., Bluetooth, Wi-Fi, 3G, and 4G LTE / LTE-A or WiMAX networks). The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

[0063] Network interface device 820 may be configured or programmed to implement the methodologies described herein. In particular, the network interface device 820 may provide various aspects of packet inspection, aggregation, queuing, and processing. The network interface device 820 may also be configured or programmed to communicate with a memory management unit (MMU), processor 802, main memory 804, static memory 806, or other components of the system 800 over the link 808. The network interface device 820 may query or otherwise interface with various components of the system 800 to inspect cache memory; trigger or cease operations of a virtual machine, process, or other processing element; or otherwise interact with various computing units or processing elements that are in the system 800 or external from the system 800.

[0064] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments that may be practiced. These embodiments are also referred to herein as "examples." Such examples may include elements in addition to those shown or described. However, also contemplated are examples that include the elements shown or described. Moreover, also contemplated are examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0065] In this document, the terms "a" or "an" are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of "at least one" or "one or more." In this document, the term "or" is used to refer to a nonexclusive or, such that "A or B" includes "A but not B," "B but not A," and "A and B," unless otherwise indicated. In the appended claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein." Also, in the following claims, the terms "including" and "comprising" are open-ended, that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms "first," "second," and "third," etc. are used merely as labels, and are not intended to suggest a numerical order for their objects.

Claims

1. A method for multi-stream scheduling at a network interface device, the method including: reading a packet descriptor from a descriptor cache, wherein the packet descriptor includes the following scheduling control parameters: a launch time offset, a gate cycle offset, and a reduction ratio; wherein the packet descriptor is associated with a packet stream having a traffic class; and scheduling packets from the packet stream for transmission using the scheduling control parameters.

2. The method of claim 1, wherein the network interface device includes one transmit queue for each traffic class of a plurality of traffic classes.

3. The method of claim 2, wherein scheduling the packets includes storing the packet in a transmit queue for the traffic class.

4. The method of any of claims 1-3, wherein the launch time offset indicates an offset from a base time of a gating cycle, and wherein scheduling packets from the packet stream includes scheduling packets to transmit at a time equal to or later than the offset from the base time during each gating cycle.

5. The method of any of claims 1-4, wherein the gate cycle offset indicates an offset from a current gating cycle, and wherein scheduling packets from the packet stream includes scheduling packets to begin transmitting at a gating cycle that occurs later than the current gating cycle.

6. The method of any of claims 1-5, wherein the reduction ratio indicates an gating cycle interval, and wherein scheduling packets from the packet stream includes scheduling packets to transmit only on gating cycles that correspond to the gating cycle interval.

7. The method of any of claims 1-6, including: transmitting packets from a transmit queue according to a gate control list, wherein the packets are scheduled according to the scheduling control parameters in the packet descriptor.

8. The method of claim 7, wherein the gate control list includes a plurality of entries that control which traffic class of a plurality of traffic classes is able to transmit during a gating cycle.

9. At least one machine-readable medium including instructions, which when executed by a machine, cause the machine to perform operations of any of the methods of claims 1-8.

10. An apparatus comprising means for performing any of the methods of claims 1-8.