The quasi-output queue behavior of a packet switching device implemented using virtual output queue ordering determined independently for each output queue

Abstract

In one embodiment, the quasi-output-queue behavior of a packet switch is implemented using virtual output queue (VOQ) ordering determined independently for each particular output queue (OQ), including using maintained latency information for the VOQs of a particular OQ. In one embodiment, all packets from all VOQs with the same port priority destination experience similar latency within a particular time window, similar to the packet service provided by an output-queue switch fabric. In one embodiment, all input ports sending traffic to the same output port priority receive bandwidth proportional to their bandwidth demand divided by the total bandwidth. Existing methods to simulate the performance of an OQ switch fabric require complex and time-consuming scheduling determinations, and do not scale. Independently determining the order in which to send packets from the VOQs associated with each particular OQ provides quasi-output-queue behavior to a system that is scalable and implementable.

Description

Technical Field

[0001] This disclosure generally relates to using virtual output queues at an ingress platform to transmit packets within a packet switching device, wherein a virtual output scheduler independently determines the order of requests for each of one or more output queues at an egress platform. Background Technology

[0002] Packet switching devices have been implemented using different architectures. Output Queuing (OQ) switching architectures provide predictable and well-understood packet scheduling order. Essentially, each packet is submitted to its destination output queue upon arrival at an input port and scheduled from output queues to output ports based on its priority. However, the OQ architecture is not scalable because it requires each output queue to absorb packets from all its source input ports at potentially very high rates. To address the scalability issue, Virtual Output Queuing (VOQ) packet switching architectures were introduced. The VOQ architecture submits packets to a VOQ located on the ingress side of the switching matrix and transmits packets to an output queue located on the egress side (i.e., closer to the output port). However, various scheduling algorithms used to determine packet transmission between VOQs and OQs yield different results, and scheduling algorithms simulating the performance of OQ switch architectures require complex and time-consuming scheduling calculations and operations, and in particular, do not scale with increasing packet transmission and reception rates. Attached Figure Description

[0003] The appended claims specifically set forth the features of one or more embodiments. The embodiments and their advantages will be understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0004] Figure 1A A packet switching device according to one embodiment is shown;

[0005] Figure 1B A packet switching device according to one embodiment is shown;

[0006] Figure 1C A packet switching device according to one embodiment is shown;

[0007] Figure 2A A packet switching device according to one embodiment is shown;

[0008] Figure 2B An apparatus according to one embodiment is shown;

[0009] Figure 3A A process according to one embodiment is shown;

[0010] Figure 3B A process according to one embodiment is shown;

[0011] Figure 3C A process according to one embodiment is shown;

[0012] Figure 3D A process according to one embodiment is shown;

[0013] Figure 3E A process according to one embodiment is shown;

[0014] Figure 4A A process according to one embodiment is shown;

[0015] Figure 4B A process according to one embodiment is shown;

[0016] Figure 5A A process according to one embodiment is shown; and

[0017] Figure 5B A process according to one embodiment is shown. Detailed Implementation

[0018] 1. Overview

[0019] Various aspects of the invention are set forth in the independent claims, and preferred features are set forth in the dependent claims. A feature of one aspect may be applied to each aspect alone or in combination with other aspects.

[0020] Methods, apparatus, computer storage media, mechanisms, and modules are disclosed that relate to implementing quasi-output queue behavior of packet switching devices using virtual output queue ordering, which is determined independently for each output queue.

[0021] An embodiment of the apparatus (e.g., a packet switching device) includes: an egress interface group including a plurality of output queues, one or more virtual output queue (VOQ) schedulers, and one or more VOQ delay data structures stored in memory; a plurality of ingress interface groups, wherein each particular ingress interface group includes a particular virtual output queue (VOQ) for each particular output queue, the particular VOQ storing packet entities of packets received by the particular ingress interface group for use in the particular output queue, wherein the received packets are sent from the apparatus based on a corresponding dequeue order from the plurality of output queues; and one or more communication mechanisms providing data path communication between each ingress interface group and the egress interface group, including transmitting the packet entities dequeued from the VOQ to the egress interface group, the VOQ being identified by a next VOQ identifier determined by the VOQ scheduler. In one embodiment, for each specific output queue among a plurality of output queues, the VOQ scheduler repeatedly determines a specific next VOQ identifier for that specific output queue and maintains the latency information of the VOQ for that specific output queue in the VOQ latency data structure, independent of the latency information maintained for the VOQs of the other output queues in the output queue. In one embodiment, each VOQ in the VOQ and each of the output queues is a first-in-first-out (FIFO) queue.

[0022] In one embodiment, multiple specific packet entities among the transmitted packet entities include associated time values; and the VOQ delay data structure is updated based on the transmitted associated time values. In one embodiment, each specific packet entity among the transmitted packet entities includes a specific packet descriptor or a specific packet in the received packets, which is enqueued in a corresponding output queue in the output queue. In one embodiment, the transmitted associated time value identifies the time when the corresponding packet entity is enqueued in the VOQ or the time when the device receives the corresponding packet.

[0023] In one embodiment, the VOQ scheduler maintains a First-In-First-Out (FIFO) queue for each of a plurality of output queues, wherein entries in the FIFO queue for an identifiable output queue include a VOQ identifier for each non-empty VOQ associated with the identifiable output queue, wherein each next VOQ identifier determined for the identifiable output queue is a first VOQ identifier at the head of the identifiable output queue, wherein the first VOQ identifier is located at the head or tail of a particular output queue in response to an adaptive decision based on the maintained latency information of the VOQs in the identifiable output queue. In one embodiment, the adaptive decision includes determining whether the first VOQ identifier identifies a particular VOQ as having the longest latency associated with the identifiable output queue based on the maintained latency information. In one embodiment, the adaptive decision includes limiting the number of times the first VOQ identifier remains at the head of the identifiable output queue before being moved to the tail of the particular output queue. In one embodiment, the adaptive decision is based on a random value and the maintained latency information of the VOQs associated with the identifiable particular output queue.

[0024] In one embodiment, each specific next VOQ identifier among the multiple next VOQ identifiers is determined based on a random value from a non-empty VOQ in a given output queue. In another embodiment, each specific next VOQ identifier among the multiple next VOQ identifiers has the longest latency of the non-empty VOQ and is selected based on a weighted value and a random value, wherein the weighted value is determined based on the longest and shortest latency, which are identified in the maintained latency information of the non-empty VOQ in the corresponding output queue. In yet another embodiment, each specific next VOQ identifier among the multiple next VOQ identifiers does not have the longest latency of the non-empty VOQ and is selected based on a weighted value and a random value, wherein the weighted value is determined based on the longest and shortest latency, which are identified in the maintained latency information of the non-empty VOQ in the corresponding output queue.

[0025] One embodiment includes a method comprising: maintaining a Virtual Output Queue (VOQ) for each distinct ingress path among a plurality of ingress paths, wherein each VOQ is associated with an output queue; for each specific packet among a plurality of packets received on a specific ingress path among the ingress paths, causing an encapsulation entity to be enqueued in a specific VOQ of the VOQ of the specific ingress path of the specific packet, wherein the encapsulation entity includes a specific packet entity and a timestamp of the specific packet; for each of a plurality of internal communication cycles, and in response to a VOQ request generated by a VOQ scheduler identifying the specific VOQ, causing the specific encapsulation entity to be dequeued from the specific VOQ having the specific encapsulation entity, wherein the specific encapsulation entity includes a specific packet entity and a specific timestamp, and after the specific encapsulated packet entity is transmitted through one or more communication mechanisms, the specific packet entity is enqueued in an output queue, and a VOQ delay data structure is maintained based on the specific timestamp; and forwarding packets from the output queue through an egress path based on the order in which the packet entities are dequeued from the output queue. In one embodiment, the VOQ scheduler maintains the delay information of the VOQ in the VOQ delay data structure, independent of the delay information maintained by other VOQs in other output queues.

[0026] In one embodiment, each of a plurality of packets received on a particular ingress path is stored in a shared memory system; wherein each particular packet entity includes a packet descriptor; wherein the method includes: retrieving a corresponding packet from the shared memory system based on the packet descriptor. In one embodiment, retrieving the corresponding packet from the shared memory system is performed in response to a particular packet entity being dequeued from an output queue, wherein the particular packet entity includes a packet descriptor.

[0027] In one embodiment, the VOQ scheduler maintains a first-in, first-out (FIFO) queue, wherein the FIFO queue has entries including VOQ identifiers for each non-empty VOQ associated with the output queue. In one embodiment, after the VOQ scheduler determines a particular VOQ from a first VOQ identifier currently located at the head of the FIFO queue, the first VOQ identifier is located at the head or tail of the FIFO queue in response to the VOQ scheduler's adaptive decision, based on the current latency information associated with the VOQ of the output queue.

[0028] In one embodiment, the VOQ scheduler determines a specific VOQ based on the current latency information of the VOQs and a random value. In response to a first result of the ordering decision, a VOQ with the longest latency is selected as the specific VOQ based on a weighted value and the random value. In response to a second result of the ordering decision, a VOQ is randomly selected from the VOQs. In one embodiment, the weighted value is determined based on the longest and shortest latency identified in the VOQ latency data structure.

[0029] An embodiment of the apparatus (e.g., a packet switching device) includes: a plurality of egress interface groups, wherein each particular egress interface group includes a plurality of output queues, a virtual output queue (VOQ) scheduler, and one or more VOQ delay data structures stored in memory; a plurality of ingress interface groups, wherein each particular ingress interface group includes a particular VOQ storing a corresponding packet entity for each particular output queue of the plurality of egress interface groups, wherein each packet entity includes a timestamp and a packet descriptor of a corresponding packet received by the particular ingress interface group; and a communication mechanism for communicating between each ingress interface group and each egress interface group. The coupling includes selectively transmitting packet entities from the ingress interface group to the egress interface group; wherein, in response to a received packet entity, each egress interface group in the egress interface group maintains latency information in the corresponding VOQ latency data structure based on the timestamp of the received packet entity, and enqueues the packet descriptor of the received packet entity in the corresponding output queue of the output queue; communicatively coupled to each ingress interface group in the ingress interface group and each egress interface group in the egress interface group with a shared packet storage system, wherein each ingress interface group in the ingress interface group stores packets in the shared packet storage system, and wherein each egress interface group in the egress interface group retrieves packets from the shared storage system based on a corresponding packet descriptor. In one embodiment, the VOQ scheduler determines the next VOQ identifier for each output queue in the output queues of the same egress interface group, regardless of other output queues on the same egress interface group, wherein the order of the selectively transmitted packet entities is identified from the determined next VOQ identifier.

[0030] In one embodiment, the specific VOQ scheduler for each specific output queue in the output queues of the plurality of egress interface groups maintains a first-in-first-out (FIFO) queue for each specific output queue in the output queues of the specific egress interface group. The FIFO queue has an entry including an entry for each non-empty VOQ associated with the specific output queue. The specific VOQ scheduler determines the next VOQ identifier from a first VOQ identifier located at the head of the corresponding FIFO queue and adaptively leaves the first VOQ identifier at the head or moves it to the tail of the corresponding FIFO queue based on current latency information associated with the VOQ of the specific output queue.

[0031] In one embodiment, the scheduler for a specific VOQ in each of a plurality of egress interface groups determines a specific next VOQ identifier for each specific output queue in the specific output queue based on the current latency information of the non-empty VOQs associated with the specific output queue and a random value; in response to a first result of a sorting decision, the specific next VOQ identifier is the non-empty VOQ with the longest latency of the non-empty VOQs associated with the specific output queue based on a weighted value and a random value; and in response to a second result of a sorting decision, one non-empty VOQ is randomly selected from the non-empty VOQs associated with the specific output queue.

[0032] 2. Description

[0033] Methods, apparatus, computer storage media, mechanisms, and modules are disclosed that relate to implementing quasi-output queue behavior of packet switching devices using virtual output queue ordering, which is determined independently for each output queue.

[0034] As used herein, "forwarding information" includes, but is not limited to, information describing how to process (e.g., forward, send, manipulate, modify, alter, discard, copy, replicate, receive) a given packet. In one embodiment, determining forwarding information is performed via an ingress lookup operation and an egress lookup operation. Furthermore, the term "processing," when referring to the processing of a packet process, refers to a broad range of operations performed in response to a packet, such as, but not limited to, functions such as forwarding / sending, discarding, manipulating / modifying / altering, receiving, replicating, creating one or more services or applications, applying one or more services or applications to a packet or packet switching device (e.g., updating information), etc. Additionally, as used herein, the term "parallel" processing generally refers to at least a portion of two or more operations being performed concurrently in time. The term "interface," as used extensively herein, includes the interface infrastructure (e.g., buffers, memory locations, forwarding and / or other data structures, processing instructions) used by network nodes when performing packet-related processing. Furthermore, as used herein, in contrast to "physical interface," a "virtual interface" is an interface that is not directly connected to an external cable or other communication mechanism.

[0035] As described herein, embodiments include various elements and limitations, and no single element or limitation is considered critical. Each claim in the claims individually sets forth an overall aspect of the embodiments. Furthermore, some of the embodiments described may include, but are not limited to, systems, networks, integrated circuit chips, embedded processors, ASICs, methods, and computer-readable media containing instructions. One or more systems, devices, components, etc., may include one or more embodiments that may include some of the elements or limitations of the claims, performed by the same or different systems, devices, components, etc. Processing elements may be general-purpose processors, task-specific processors, cores of one or more processors, or other co-located resource-sharing implementations for performing corresponding processing. The embodiments described below embody various aspects and configurations, and the accompanying drawings illustrate exemplary and non-limiting configurations. Computer-readable media and modules (e.g., processors and memories or other means configured to perform such operations) for performing methods and processing block operations are disclosed, and these computer-readable media and modules are consistent with the scalable scope of the embodiments. The term "means" is used herein in accordance with its general definition of an appliance or device.

[0036] The steps, connections, and processing of signals and information shown in the accompanying drawings (including, but not limited to, any block diagrams, flowcharts, and message sequence diagrams) can generally be performed in the same or different serial or parallel order and / or by different components and / or processes, threads, etc., and / or by different connections, and in other embodiments in combination with other functions, unless this disables the embodiment or explicitly or implicitly requires a sequence (e.g., for a read value, the order in which the read value is processed must be such that the value is obtained before it is processed, but some of the associated processes can be performed before, simultaneously with, and / or after the read operation). Furthermore, unless explicitly stated otherwise, nothing described or referenced in this document is to be considered prior art of this application.

[0037] This document uses the term "an embodiment" to refer to a particular embodiment, wherein each reference to "an embodiment" may refer to a different embodiment, and the repeated use of the term in describing associated features, elements, and / or limitations does not establish a cumulative set of associated features, elements, and / or limitations that must be included in each embodiment, but embodiments may generally include all such features, elements, and / or limitations. Furthermore, the terms "first," "second," etc., and "specific," "concrete," and "identifiable" are generally used herein to denote different units (e.g., first component or operation, second component or operation, specific component or operation, concrete component or operation). The use of these terms herein does not necessarily indicate a sequence (e.g., one unit, operation, or event occurs or arrives before another unit, operation, or event or another feature), but rather provides a mechanism for distinguishing unit elements. Additionally, the phrases "based on x" and "in response to x" are used to denote the minimal set of items "x" from which something is derived or caused, where "x" is extensible and does not necessarily describe a complete list of items on which operations are performed, etc. The elements introduced by "multiple components" may subsequently be referred to by "the component." Similarly, the element introduced in “one or more parts” may subsequently be referred to as “the part” (referring to a single part or multiple parts). Additionally, the phrase “coupled to” is used to indicate some degree of direct or indirect connection between two elements or devices, where the coupled device may or may not modify the coupled signal or transmitted information. Furthermore, the term “or” is used herein to identify a selection of one or more (including all) of the combined items. Additionally, the transitional term “comprising,” synonymous with “including,” “containing,” or “characterized in,” is inclusive or open-ended and does not exclude additional, unmentioned elements or method steps. Finally, when referenced in a method claim for performing a step, the term “specific machine” refers to a specific machine within the statutory type of machine as defined in 35 USC § 101.

[0038] Figure 1AA packet switching device 100 according to one embodiment is shown. As shown, the packet switching device 100 includes a plurality of ingress interface groups 110-111, which are communicatively coupled to an egress interface group 120 via one or more communication mechanisms (e.g., switching matrix, structure) 104.

[0039] As used herein, an ingress interface group refers to a device (e.g., a line card, board, or switching platform) that includes at least one interface (for receiving packets) and at least one virtual output queue (for storing packet entities corresponding to the received packets); while an output interface group refers to a device (e.g., a line card, board, or switching platform) that includes at least one output queue (for storing packet entities received from the ingress interface group) and one or more interfaces (for sending corresponding packets from the egress interface group).

[0040] As used herein, a “packet entity” refers to a discrete unit that includes data for identifying a packet, such as, but not limited to, the packet itself, a packet descriptor (e.g., including information for retrieving the packet from a storage device and some data typically extracted from the packet header for forwarding the packet entity via a packet switching device), and possibly other information / data (e.g., internal packet switching data, including but not limited to timestamps, forwarding data, operational data, monitoring data, and / or other data or information).

[0041] In one embodiment, a packet entity (which typically contains some internal packet switching data (e.g., timestamps, forwarding information)) is enqueued in a virtual output queue and then transmitted to the appropriate output queue via a packet switching mechanism (e.g., a structure, a crossbar switch), wherein at least the packet is enqueued. In another embodiment, a packet entity in the form of a packet descriptor (which typically contains some internal packet switching data (e.g., timestamps, forwarding information)) is enqueued in a virtual output queue and then transmitted to the appropriate output queue via a packet switching mechanism (e.g., a structure, a crossbar switch), wherein at least the packet descriptor is enqueued. In yet another embodiment, the packet is stored in shared memory and retrieved from the egress interface group based on the packet descriptor; therefore, no resources of one or more communication mechanisms (e.g., excluding shared memory) are consumed when transmitting the entire packet.

[0042] In one embodiment, the egress interface group includes optimizations for at least a portion of the VOQs (e.g., all VOQs, VOQs storing higher-priority traffic) using Virtual Inbound Queues (VIQs), such that packet transmission cycles on communication mechanisms with handover contention are used to transfer packet entities from VOQs to their respective VIQs, and handover contention is not used to send packets based on the order of the requested VOQs. In one embodiment, the VOQ request ordering scheduler does not distinguish between packet entities in VOQs and VIQs when determining the appropriate request order.

[0043] like Figure 1A As shown, packet entities are enqueued in VOQ 118 and VOQ 119 for packets (101) received by the corresponding ingress interface group 110 and ingress interface group 111. In one embodiment, each of these packet entities includes a timestamp (e.g., reflecting the system time of packet reception or enqueueing), which the egress interface group 120 then uses to maintain one or more latency characteristics of the corresponding VOQ 118 and VOQ 119 in one or more data structures. In one embodiment, maintaining these one or more latency characteristics includes subtracting the received timestamp from the current system time and updating one or more fields in the latency data structure.

[0044] Based on the latency characteristics of VOQ 118 and VOQ 119 (e.g., disregarding the length, latency, or other characteristics of virtual output queues associated with other output queues), the fully local scheduler 125 on the egress interface group 120 generates an ordered sequence of VOQ identifier requests used by the packet switching device 100 in a corresponding order, causing packet entities to be dequeued from VOQ 118 and VOQ 119, and transmitting these dequeued packet entities (105) to the egress interface group 120 via one or more communication mechanisms 104. In one embodiment, the egress interface group 120 includes a virtual input queue (VIQ) 121 for each VOQ in VOQ 118 and VOQ 119.

[0045] In one embodiment, in response to a received requested packet entity (105), scheduler 125 maintains one or more delay data structures (“DS”) based on the timestamp of packet entity (105) (therefore maintaining the current delay information of VOQ 118, VOQ 119), egress interface group 120 enqueues the packet or packet descriptor of packet entity (105) in OQ 122, and in response to the dequeue of the corresponding packet or packet descriptor from OQ 122, the corresponding packet (109) is sent from egress interface group 120.

[0046] Figure 1BA packet switching device 130 according to one embodiment is illustrated, which includes a single OQ 140 (e.g., on an egress interface group in one embodiment) associated with a plurality of VOQs 136 (e.g., one VOQ 136 on each ingress interface group in one embodiment). Packets 131 are received and stored in a shared packet memory 132. Based on a forwarding decision for each of these packets (which is identified as being validly forwarded via OQ 130), a corresponding packet entity 133 (e.g., having a packet descriptor and timestamp) is enqueued in the corresponding first-in-first-out (FIFO) VOQ 136. In one embodiment, the VOQ 136 is located on the ingress interface group on which the corresponding packet 131 was received.

[0047] The VOQ request sequencer 148 identifies an ordered sequence of VOQ requests based on the latency data structure it maintains for each VOQ 136 in the VOQ 136, which are transmitted via one or more communication mechanisms 138 (e.g., control plane communication or onboard data plane communication) (141). In response, timestamped packet descriptors (133) are dequeued from the VOQ 136 in the corresponding order and transmitted as packet entities (143) via one or more communication mechanisms 138 (e.g., data plane communication). In one embodiment, the packet descriptors in the received packet entities (143) are enqueued in the FIFO OQ 140, and the scheduler 148 maintains the VOQ latency data structure based on the timestamps in the received packet entities (143) (e.g., directly identifying one or more latency characteristics or using one or more latency characteristics to determine the corresponding VOQ (136)).

[0048] In one embodiment, in response to a packet descriptor being dequeued from OQ 140, the corresponding packet (131) is retrieved (145) from the shared packet memory 132 and sent from the egress interface 146.

[0049] In response to receiving its packet entity (143) or its packet descriptor enqueued, one embodiment retrieves (145) the corresponding packet (131) from shared packet memory 132, and the retrieved packet 131 is stored in a buffer associated with egress interface 146. The packet 131 is then read from the buffer memory and sent from egress interface 146 in response to its packet dequeueing from OQ 140. In one embodiment, buffering its memory is faster than sharing memory 132. In one embodiment, multiple egress interfaces compete to read packets from packet memory 132, so reading and buffering packets a priori is more efficient than delaying read operations until their packet dequeues are dequeued.

[0050] In one embodiment, in response to its packet descriptor being received in packet entity (143), the corresponding packet (131) is retrieved (145) from shared packet memory 132, and the retrieved packet 131 is enqueued in FIFO OQ 140 (e.g., in the order of read requests for packets associated with OQ 140). In response to being dequeued from OQ 140, packet 131 is then transmitted from egress interface 146.

[0051] Figure 1C A packet switching device 150 according to one embodiment is shown. As shown, the packet switching device 150 includes two ingress interface groups (160 and 180) and two egress interface groups (170 and 190). One embodiment includes more input links, output links, ingress interface groups, and / or egress interface groups than... Figure 1C More details are shown below. In one embodiment, inlet interface group 1 (160) and outlet interface group 1 (170) are located on the same line card or other board. In one embodiment, inlet interface group 2 (180) and outlet interface group 2 (190) are located on the same line card or other board.

[0052] Each of the ingress interface groups 160 and 180 is communicatively coupled to each of the egress interface groups 170 and 190 via data plane communication (197). In one embodiment, the data plane communication (197) includes a switching matrix with connections for each packet communication cycle, such that M packet entities can be transmitted from each of the ingress interface groups 160 and 180, and that two packet entities can be received by each of the egress interface groups 170 and 190, where M is at least 1. In one embodiment, M is 2. In another embodiment, M is an integer greater than 2.

[0053] Each of the ingress interface groups 160 and 180, and the egress interface groups 170 and 190, is communicatively coupled to a shared packet memory subsystem (199). In one embodiment, for each memory cycle using the shared packet memory subsystem (199), each ingress interface group 160 and 180 may store K packets, and each egress interface group 170 and 190 may retrieve K packets, where K is at least 1. In one embodiment, K is 2. In one embodiment, K is greater than 2.

[0054] Each ingress interface group in ingress interface group 160 and ingress interface group 180 is communicatively coupled to each egress interface group in egress interface group 170 and egress interface group 190 via control plane communication (198). In one embodiment, control plane communication (198) provides communication of VOQ requests between each egress interface group in egress interface group 170 and egress interface group 190 and each ingress interface group in ingress interface group 160 and ingress interface group 180 during each packet communication cycle.

[0055] In one embodiment, the packet switching device exchanges packets according to N different priorities, where N is an integer of at least 1. In one embodiment, N is 2. In one embodiment, N is greater than 2.

[0056] like Figure 1C As shown, egress interface group 1 (170) includes a set of N output queues (171) and N output queues (172), where the N output queues (171) are associated with sending packets leaving egress link 1 (155), and the N output queues (172) are associated with sending packets leaving egress link 2 (156). Additionally, egress interface group 2 (190) includes a set of N output queues (193) and N output queues (194), where the N output queues (193) are associated with sending packets leaving egress link 3 (157), and the N output queues (194) are associated with sending packets leaving egress link 4 (158).

[0057] For example Figure 1C As shown, inlet interface group 1 (160) includes 4 groups of N VOQs (161 to 164), VOQ 161 is associated with OQ 171 of outlet interface group 1 (170), VOQ 162 is associated with OQ 172 of outlet interface group 1 (170), VOQ 163 is associated with OQ 193 of outlet interface group 2 (190), and VOQ 164 is associated with OQ 194 of outlet interface group 2 (190). Similarly, ingress interface group 2 (180) includes four groups of N VOQs (181 to 184), VOQ 181 is associated with OQ 171 of outgress interface group 1 (170), VOQ 182 is associated with OQ 172 of outgress interface group 1 (170), VOQ 183 is associated with OQ 193 of outgress interface group 2 (190), and VOQ 184 is associated with OQ 194 of outgress interface group 2 (190).

[0058] In one embodiment, each specific packet received on ingress links 151 to 154 is classified to determine which output link (155 to 158) the specific packet should be sent from and its corresponding priority (e.g., ranging from 1 to N). Specific packets are stored in a shared packet memory subsystem (199), and the corresponding packet entity is enqueued in the corresponding specific VOQ (161 to 164, 181–184) at a specific time. In one embodiment, each packet entity includes a packet descriptor and a timestamp (e.g., at a specific time) for the specific packet, which is used by the corresponding egress interface group (175, 195) to maintain one or more current latency characteristics of the specific VOQ (161 to 164, 181 to 184).

[0059] Based on the VOQ delay information stored in its (one or more) VOQ delay data structures, the egress interface group 1 scheduler 175 individually and independently determines the next VOQ (161, 162, 181, 182), and dequeues the packet entity from the next VOQ (161, 162, 181, 182) for each of the N OQs 171 and N OQs 172. Similarly, based on the VOQ delay information stored in its (one or more) VOQ delay data structures, the egress interface group 2 scheduler 195 individually and independently determines the next VOQ (163, 164, 183, 184), and dequeues the packet entity from the next VOQ (163, 164, 183, 184) for each of the N OQs 193 and N OQs 194.

[0060] Figures 2A to 2B The discussion herein provides a description of various network nodes according to one embodiment.

[0061] Figure 2AAn embodiment of a packet switching device 210 (e.g., a router, node, application device, gateway) according to one embodiment is shown. As shown, the packet switching device 210 includes a plurality of line cards (211 and 215), each line card having one or more network interfaces for sending and receiving packets over a communication link (e.g., possibly part of a link aggregation group), and one or more processing elements used in one embodiment, which is associated with implementing quasi-output queue behavior of the packet switching device using virtual output queue ordering, which is determined independently for each output queue. The packet switching device 210 also has a control plane having one or more processing elements (e.g., one or more routing processors 212) for managing the control plane and controlling plane packet processing, which is associated with implementing quasi-output queue behavior of the packet switching device using virtual output queue ordering, which is determined independently for each output queue. Packet switching device 210 also includes other cards 214 (e.g., service cards, blades) that, in one embodiment, include processing elements for processing (e.g., forwarding / sending, dropping, manipulating, altering, modifying, receiving, creating, copying, possibly in conjunction with shared memory and / or operating with one or more service functions, applying services according to one or more service functions) associated with implementing quasi-output queue behavior of the packet switching device using virtual output queue ordering (the virtual output queue ordering is determined independently for each output queue), and some hardware-based communication mechanisms 213 (e.g., buses, switching structures, and / or matrices, etc.) for allowing their different entities 211, 212, 214, and 215 to communicate. Line cards 211 and 215 typically perform actions that act as both ingress and egress line cards (e.g., including VOQ, OQ, and VOQ schedulers) involving multiple other specific packets and / or packet streams received by or sent from packet switching device 210.

[0062] Figure 2B This is a block diagram of an apparatus 220 (e.g., a host, router, node, destination, or part thereof) used in one embodiment, which is associated with implementing quasi-output queue behavior of a packet-switching device using virtual output queue ordering, which is determined independently for each output queue.

[0063] In one embodiment, the apparatus 220 performs one or more processes or portions thereof, the one or more processes or portions thereof corresponding to a flowchart shown or otherwise described herein and / or a flowchart shown or otherwise described herein in another figure.

[0064] In one embodiment, device 220 includes one or more processors 221 (typically having on-chip memory), memory 222 (possibly shared memory, VOQ, OQ), one or more storage devices 223 (possibly shared memory, VOQ, OQ), one or more dedicated components 225 (e.g., optimized hardware for performing lookup, packet processing; associative memory; binary and / or ternary content addressable memory; one or more application-specific integrated circuits; cryptographic hashing hardware, etc.), and one or more interfaces 227 for transmitting information (e.g., sending and receiving packets, user interfaces, displaying information, etc.), which are typically communicatively coupled to a communication path via one or more communication mechanisms 229 (e.g., bus, link, switching structure, matrix), which is typically tailored to meet the needs of a specific application.

[0065] Various embodiments of device 220 may include more or fewer elements. Operation of device 220 is typically controlled by processor(s) 221 using memory 222 and storage devices(s) 223 (e.g., including VOQ, OQ) to perform one or more tasks or processes. Memory 222 is a computer-readable / computer storage medium and typically includes random access memory (RAM), read-only memory (ROM), flash memory, integrated circuits, and / or other memory components. Memory 222 typically stores computer-executable instructions and / or data, which are executed by processor(s) 221, and the data is manipulated by processor(s) 221 to achieve the functionality according to embodiments. Storage devices(s) 223 are another type of computer-readable medium and typically include solid-state storage media, disk drives, disks, networking services, tape drives, and other storage devices. Storage devices(s) 223 typically store computer-executable instructions and / or data, which are executed by processor(s) 221, and the data is manipulated by processor(s) 221 to achieve the functionality according to embodiments.

[0066] Figure 3A A process according to one embodiment is illustrated. Processing of this data plane process begins at processing block 300. In processing block 302, packets are received at the ingress interface communicatively coupled to the ingress link. In processing block 304, the received packets are stored in a shared packet memory, and the corresponding packet entity (e.g., a timestamped packet descriptor) is enqueued in a VOQ, which is determined based on the packet and associated forwarding information (e.g., a VOQ associated with the corresponding OQ and priority). Processing returns to processing block 302 to receive and process more packets.

[0067] Figure 3BThe process according to one embodiment is illustrated. The data plane process begins at processing block 320. As determined in processing block 321, if it is time to send a packet from the packet switching device for a specific egress interface, the process proceeds to processing block 322; otherwise, the process remains at processing block 321. Processing continues in processing block 322, identifying a non-empty specific output queue among the N output queues (corresponding to each of the N priorities) associated with the specific egress interface, based on (e.g., bandwidth allocation between priorities or other scheduling algorithms). In processing block 324, a specific packet descriptor is dequeued from the head of the identified specific FIFO output queue. In processing block 326, the corresponding packet is retrieved from the shared packet memory subsystem based on the specific packet descriptor. In processing block 328, the specific packet is sent from the specific egress interface. Process returns to processing block 321.

[0068] Figure 3C A process according to one embodiment is illustrated. Processing of this control plane process begins at processing block 330. As determined in processing block 333, if it is time to update the VOQ scheduler on one or more egress interface groups, processing proceeds to processing block 334; otherwise, processing remains at processing block 333. Processing continues in processing block 334, updating the data structure in the VOQ scheduler of the egress interface group with the corresponding current VOQ length, including identifying new active VOQs (e.g., changing from empty to non-empty). In one embodiment, this information is transmitted between the ingress and egress interface groups via control plane communication. Processing returns to processing block 333.

[0069] Figure 3D The process according to one embodiment is illustrated. Processing begins at processing block 340. As determined in processing block 343, if the packet entity has been received by the egress interface group, processing proceeds to processing block 344; otherwise, processing remains at processing block 343. Processing continues in processing block 344, and the packet descriptor in the received packet entity is enqueued in the corresponding output queue. In processing block 346 of one embodiment, latency information, associated with the VOQ from which the packet entity was sent, is updated based on a timestamp retrieved from the received packet entity. In one embodiment, one or more features (e.g., the longest, shortest, or average latency of the VOQ or non-empty VOQ associated with the corresponding output queue) are updated based on the timestamp retrieved from the packet entity. Processing returns to processing block 343.

[0070] Figure 3EThe process performed for each specific egress interface group according to one embodiment is illustrated. Processing begins at processing block 360. As determined in processing block 363, when the next packet cycle for transmitting packets between the ingress interface group and the egress interface group via a restricted data path (e.g., a matrix) begins, processing proceeds to processing block 364; otherwise, processing remains at processing block 363.

[0071] Processing continues in processing block 364, determining the next interface for a particular group of egress interfaces based on the link rate and flow control of the egress interface (e.g., proportionally allocating egress interface bandwidth in one embodiment). In processing block 366, a specific OQ associated with the determined next interface is identified based on the link rate and flow control (e.g., proportionally allocating bandwidth in one embodiment) associated with the OQ of the particular group of egress interfaces.

[0072] In processing block 368, the egress VOQ scheduler determines a specific VOQ associated with a particular OQ in a manner independent of (e.g., disregarding) the VOQs of other OQs. In one embodiment, this determination is based on maintained latency information associated with the VOQ of a particular output queue (e.g., longest, shortest, average latency), one or more random numbers, a weighted average of one or more VOQs with specific latency characteristics, and the selection of the next VOQ identifier in a loop. In processing block 370, the next VOQ identifier corresponding to the determined specific VOQ (e.g., via a control path) is transmitted to the appropriate ingress interface group. In processing block 372, the packet entity (including a packet descriptor and a timestamp associated with the latency) is dequeued from the specific VOQ and sent to the egress interface group via a restricted data path. Processing returns to processing block 363.

[0073] Therefore, in one embodiment, all packets from all VOQs with the same port priority destination experience similar latency within a specific time window. This is similar to the packet service provided by an OQ switch architecture, since all packets with the same port priority destination are submitted to the same OQ; thus, they will experience similar latency within a specific time window. Additionally, in one embodiment, all input ports sending traffic to the same output port priority receive bandwidth proportional to their bandwidth requirement divided by the total bandwidth.

[0074] Determining the next VOQ independently for each output queue in the output queue provides packet switching equipment with practical, faster, and more efficient scheduling determination (e.g., in one embodiment, VOQs of other OQs are not classified and / or are not considered), and these determinations, based on the current VOQ latency, provide expected behavior that approximates the output queue behavior of the packet switching equipment.

[0075] In this way, one embodiment provides latency fairness between virtual output queues of each output queue when compared to the corresponding ordering of the ideal output queue.

[0076] Existing packet-switching systems attempting to simulate output queue behavior typically require complex and time-consuming computations (e.g., sorting the time values ​​of all VOQs across all OQs) and dedicated hardware (e.g., non-FIFO queues), making their use impractical in high-performance switches and routers. While the VOQ architecture is highly scalable, each VOQ-to-OQ scheduler delivers different scheduling outcomes based on its algorithm. For example, packet-level VOQ round-robin scheduling can cause input ports transmitting large packets to consume more bandwidth than input ports sending small packets to the same output port, even if both input ports send packets of the same bandwidth to the output port. In a second example, VOQ round-robin scheduling can result in input ports requiring low packet bandwidth to the output port and input ports requiring high packet bandwidth to the same output port receiving the same bandwidth.

[0077] Figure 4A and Figure 4B The process for identifying the next VOQ, executed in one embodiment, is illustrated. In one embodiment, it is executed in parallel. Figure 4A and Figure 4B The process updates the maintained latency information based on the time values ​​in the grouped entities (e.g., in...). Figure 4A (in the middle), the group entity is received based on a VOQ-ID request that identifies a specific VOQ (e.g., in Figure 4B (It is determined in processing box 442).

[0078] In one embodiment, the time value directly identifies the latency of the corresponding VOQ. In another embodiment, the time value indirectly identifies the latency of the corresponding VOQ, for example, by subtracting it from the current system time, but not limited to.

[0079] Although about Figure 4A and Figure 4B One embodiment described uses the longest latency information, but one embodiment maintains one or more latency characters (e.g., longest, shortest, average and / or other values) and uses one or more of these latency characteristics when determining the next VOQ-ID (e.g., whether the selected VOQ-ID is kept at the head / first position of the queue / list or placed at the tail / last position of the queue / list).

[0080] Figure 4A The maintenance of delay information for each VOQ is illustrated in one embodiment (e.g., in...). Figure 3D In the processing box 346), each VOQ includes the longest latency information.

[0081] Processing begins at processing block 400. In processing block 402, the VOQ latency is determined by subtracting the received timestamp from the current system time. As determined in processing block 405, if the VOQ identified by the packet entity (including the timestamp) has the longest latency, the maintained longest latency value is updated to the determined latency value in processing block 406, and processing proceeds directly to processing block 419.

[0082] Otherwise, proceed to processing box 409, where it is determined that if the determined delay is greater than the previous longest delay, the processing proceeds to processing box 410; otherwise, the processing proceeds directly to processing box 419.

[0083] Continuing with block 410, update the identifier of the maintained VOQ with the longest delay. In block 412, update the longest maintained delay of the VOQ associated with the OQ. In block 414, an adaptive continuous maximum value is initialized to limit the number of times the same VOQ can be repeatedly sent for the OQ (e.g., used when at least two VOQs are active). Continuing with block 419, complete the process. Figure 4A The processing of flowcharts.

[0084] Figure 4B A process executed in one embodiment is illustrated, which is used to independently determine the next VOQ (e.g., in) the identified OQ of the requesting grouping entity. Figure 3E (Executed in processing block 368 in one embodiment). One embodiment maintains an ordered list (also known as a FIFO queue) of active VOQs for each OQ. The list is sorted as the next VOQ is identified, while determining the position that may remain in the list based on the longest latency, without exceeding an adaptive continuous maximum value.

[0085] Therefore, one embodiment allocates more bandwidth to a particular VOQ (e.g., based on the current adaptive consecutive maximum value) when the maintained latency information identifies the particular VOQ as having the longest latency of the VOQ associated with the corresponding OQ (e.g., as determined in processing block 447), while preventing abnormal behavior by limiting the number of consecutive VOQ-ID requests sent for the same particular VOQ (e.g., as determined in adaptive decision processing block 453). In this way, one embodiment compensates for VOQs that receive fewer packets than other VOQs (which are associated with the same OQ). Furthermore, by increasing the adaptive consecutive maximum value in response to invoking this limit (e.g., in processing block 458), more bandwidth adaptively becomes available (e.g., for more consecutive iterations) for the next VOQ identified as having the longest latency (e.g., as determined in processing block 447).

[0086] Processing begins at processing box 440. In processing box 442, a specific VOQ-ID is dequeued from the head of the active VOQ-IDS list (causing a packet descriptor request to be requested and transmitted from the corresponding VOQ, e.g., according to...). Figure 3E In processing blocks 370 to 372), the number of groups in the corresponding VOQ (e.g., each variable maintained in the exit interface group) is reduced by 1. As determined in processing block 445, if the corresponding VOQ is now identified as empty, the processing proceeds directly to processing block 459; otherwise, the processing proceeds to processing block 447.

[0087] Continue processing block 447, where, if the particular VOQ-ID is the VOQ with the longest delay that is identified as an OQ in the maintained delay information (e.g., maintained in a variable by the VOQ scheduler), then processing proceeds to processing block 450; otherwise, processing proceeds to processing block 448.

[0088] Continue processing box 448, placing the specific VOQ-ID at the end of the active VOQ list / queue in the OQ. Processing proceeds directly to processing box 459.

[0089] Continue processing block 450, incrementing the adaptive continuity counter by 1 (which was initially set to 0 in the previous iteration in processing block 458). If, as determined in processing block 453, the adaptive continuity counter equals the adaptive continuity maximum value, the process proceeds to processing block 456; otherwise, the process proceeds to processing block 454.

[0090] Continuing with box 454, place the specific VOQ-ID at the beginning of the active VOQ list in the OQ (therefore) Figure 4BIn the next iteration of the loop shown, the same VOQ-ID will be used in processing block 442. Processing proceeds directly to processing block 459.

[0091] Continuing with processing box 456, the specific VOQ-ID is placed at the end of the OQ's active VOQ list. In processing box 458, the adaptive continuous maximum value is incremented to provide future increments for the specific VOQ (e.g., to compensate for smaller packets sent via the specific VOQ), and its adaptive continuous counter is reset to zero. Processing proceeds directly to processing box 459.

[0092] Continue processing box 459 to complete the process. Figure 4B The processing of flowcharts.

[0093] Figure 5A and Figure 5B The process for identifying the next VOQ, executed in one embodiment, is illustrated. In one embodiment, it is executed in parallel. Figure 5A and Figure 5B The process updates the maintained latency information based on the time values ​​in the grouped entities (e.g., in...). Figure 5A (in the middle), the group entity is received based on a VOQ-ID request that identifies a specific VOQ (e.g., in Figure 5B (As determined in processing box 548).

[0094] In one embodiment, the time value directly identifies the latency of the corresponding VOQ. In another embodiment, the time value indirectly identifies the latency of the corresponding VOQ, for example, by subtracting it from the current system time, but not limited to.

[0095] Although about Figure 5A and Figure 5B One embodiment described uses the longest latency information and the shortest latency information, but one embodiment maintains one or more latency characters (e.g., longest, shortest, average and / or other values) and uses one or more of these latency characteristics when determining the next VOQ-ID (e.g., weighting one or more of these characteristics and comparing the value with a random number generated when selecting the next VOQ-ID).

[0096] Figure 5A The maintenance of delay information for each VOQ is illustrated in one embodiment (e.g., in...). Figure 3D In the processing box 346, each VOQ includes the longest latency information and the shortest latency information.

[0097] Processing begins at processing block 500. In processing block 502, the VOQ latency is determined by subtracting the received timestamp from the current system time. As determined in processing block 505, if the packet entity (including the timestamp) has the longest latency from the VOQ from which it identifies the VOQ, the maintained longest latency value is updated to the determined latency value in processing block 506, and processing proceeds directly to processing block 513.

[0098] Otherwise, proceed to processing box 509, where it is determined that if the determined delay is greater than the previous longest delay, the processing proceeds to processing box 510; otherwise, the processing proceeds directly to processing box 513.

[0099] Continue processing in block 510, update the maintained identifier of the VOQ with the longest latency, and update the maintained longest latency of the VOQ associated with the OQ. Continue processing in block 513.

[0100] Continuing with processing box 513, as determined therein, if the grouped entity (including the timestamp) has the shortest latency from its VOQ of identification, the maintained shortest latency value is updated to the determined latency value in processing box 514, and processing proceeds directly to processing box 519.

[0101] Otherwise, proceed to processing box 517, where it is determined that if the determined delay is less than the previous shortest delay, the processing proceeds to processing box 518; otherwise, the processing proceeds directly to processing box 519.

[0102] Continue processing block 518, update the maintained identifier of the VOQ with the shortest latency, and update the maintained shortest latency of the VOQ associated with the OQ. Processing continues to processing block 519.

[0103] Continue processing box 519 to complete the process. Figure 5A The processing of flowcharts.

[0104] Figure 5B The process executed in one embodiment is illustrated, which is used to independently determine the next VOQ (e.g., VOQ-ID) of the identified OQ from which the requesting group entity originates (e.g., in...). Figure 3E (Executed in processing block 368 in one embodiment). One embodiment maintains a set of VOQ-IDs corresponding to the active VOQ for each OQ. The longest and shortest latency information maintained for the VOQs of each OQ (e.g., based on...) Figure 5A The next VOQ-ID is selected from the VOQ-ID set of OQ using one or more weighted values. In one embodiment, the weighting is also a function of the number of active VOQ-IDs in the VOQ-ID set.

[0105] Processing begins at processing block 540. In processing block 542, an adaptive probability is determined for selecting a specific VOQ-ID, which is identified as having the longest latency among the maintained latency information. Because the maintained latency is updated based on timestamps in a packet entity that requests reception based on its corresponding VOQ-ID, one embodiment limits this adaptive probability to less than 1 to allow any active VOQ-ID in the active VOQs of the OQ to be selected. Therefore, abnormal behavior is prevented by allowing updates to the maintained latency information for all active VOQs.

[0106] In one embodiment, the adaptive probability is determined based on a weighted average of the difference between the longest and shortest sustained latency of the VOQs in the OQ's VOQ-IDs set (e.g., the number of active VOQs), given the number of VOQ-IDs in the OQ's VOQ-IDs set. In one embodiment, when the difference is zero, all VOQ-IDs are given the same probability of being selected. In one embodiment, for larger differences, the adaptive probability of selecting a particular VOQ-ID is relatively high, while for smaller differences, the adaptive probability of selecting a particular VOQ-ID is relatively low. In one embodiment, the adaptive probability is adjusted based on a lower value when the number of VOQ-IDs in the VOQ-IDs set is relatively high, and based on a higher value when the number of VOQ-IDs in the VOQ-IDs set is relatively low.

[0107] As determined in processing block 545, if the specific VOQ-ID identified as having the longest latency is scheduled based on a comparison operation between the determined adaptive probability and the generated random number, then the processing proceeds to processing block 546; otherwise, the processing proceeds to processing block 548.

[0108] Continuing in processing block 546, a specific VOQ-ID is set to the currently maintained VOQ-ID with the longest latency. Processing proceeds directly to processing block 550.

[0109] Continuing in processing block 548, a weighted random selection based on the maintained latency information of the OQ-VOQ may be used, where an active VOQ-ID from the active VOQ-IDs of the OQ is randomly selected as a specific VOQ-ID. Processing proceeds to processing block 550.

[0110] Continue in processing box 550, for example, according to Figure 3EIn processing blocks 370 to 372, a packet entity corresponding to a specific VOQ-ID is requested and sent from the corresponding VOQ. As determined in processing block 555, if the VOQ corresponding to the specific VOQ-ID is empty (e.g., as a result of processing block 550), the processing proceeds to processing block 556; otherwise, the processing proceeds directly to processing block 559.

[0111] Continue processing box 556, where a specific VOQ-ID is removed from the list of active VOQ-IDs in the OQ, and some latency information may be updated (e.g., the VOQ-ID identified as corresponding to the longest and / or shortest VOQ in the OQ).

[0112] Continue processing box 559 to complete the process. Figure 5B The processing of flowcharts.

[0113] In summary, in one embodiment, quasi-output queuing behavior of a packet-switching device is achieved using virtual output queue (VOQ) ordering determined independently for each specific output queue (OQ), including the latency information maintained by the VOQs using a specific OQ. In one embodiment, all packets from all VOQs with destinations having the same port priority experience similar latency within a specific time window, similar to the packet service provided by an output queue switching architecture. In one embodiment, all input ports sending traffic to the same output port priority receive bandwidth proportional to their bandwidth requirement divided by the total bandwidth. Existing methods for simulating the performance of OQ switch architectures require complex and time-consuming scheduling determination and are not scalable. Independently determining the order in which packets are sent from the VOQs associated with each specific OQ provides quasi-output queuing behavior to scalable and implementable systems.

[0114] Given the many possible embodiments to which the principles of this disclosure can be applied, it should be understood that the embodiments and aspects thereof described herein with reference to the accompanying drawings are merely illustrative and should not be construed as limiting the scope of this disclosure. For example, it will be apparent to those skilled in the art that many processing block operations within a processing block operation can be reordered to be performed before, after, or substantially simultaneously with other operations. Moreover, many different forms of data structures can be used in various embodiments. The disclosure described herein contemplates all such embodiments that are within the scope of the following claims and their equivalents.

Claims

1. An apparatus for data communication, comprising: The egress interface group includes: multiple output queues, one or more virtual output queue (VOQ) schedulers, and one or more VOQ delay data structures stored in memory; Multiple ingress interface groups, wherein each specific ingress interface group includes a specific virtual output queue (VOQ) for each specific output queue among the multiple output queues, the specific VOQ storing packet entities of packets received by the specific ingress interface group for use in the specific output queue, wherein the received packets are sent from the device based on a corresponding dequeue order from the multiple output queues; and One or more communication mechanisms provide data path communication between each ingress interface group and the egress interface group in the ingress interface group, including transmitting the packet entity dequeued from the VOQ to the egress interface group, the VOQ being identified by the next VOQ identifier determined by the VOQ scheduler; Specifically, for each specific output queue among the plurality of output queues, the VOQ scheduler repeatedly determines a specific next VOQ identifier for that specific output queue and maintains the VOQ delay information of that specific output queue in the VOQ delay data structure, regardless of the maintained delay information of the VOQs of other output queues in the output queue.

2. The apparatus according to claim 1, wherein, Each of the VOQ and the output queue is a first-in, first-out (FIFO) queue.

3. The apparatus according to claim 1 or 2, wherein, Multiple specific packet entities in the transmitted packet entities include associated time values; and the VOQ delay data structure is updated based on the associated time values.

4. The apparatus according to claim 3, wherein, Each specific packet entity in the transmitted packet entity includes a specific packet descriptor or a specific packet in the received packets, and the specific packet descriptor or the specific packet is enqueued in the corresponding output queue in the output queue.

5. The apparatus according to claim 4, wherein, The transmitted associated time value identifies the time when the corresponding group entity is enqueued in the VOQ or the time when the device receives the corresponding group.

6. The apparatus according to claim 1, wherein, The VOQ scheduler maintains a first-in-first-out (FIFO) queue for each of the plurality of output queues, wherein the entries for the FIFO queue of the identifiable output queue include a VOQ identifier for each non-empty VOQ associated with the identifiable output queue, wherein each next VOQ identifier determined for the identifiable output queue is a first VOQ identifier at the head of the identifiable output queue, wherein the first VOQ identifier is located at the head or tail of the particular output queue in response to an adaptive decision based on the maintained latency information of the VOQ of the identifiable output queue.

7. The apparatus according to claim 6, wherein, The adaptive decision includes: determining, based on the maintained latency information, whether the first VOQ identifier identifies a particular VOQ as having the longest latency associated with the identifiable output queue.

8. The apparatus according to claim 6 or 7, wherein, The adaptive decision includes limiting the number of times the first VOQ identifier remains at the head of the identifiable output queue before being moved to the tail of the specific output queue.

9. The apparatus according to claim 6 or 7, wherein, The adaptive decision is based on random values ​​and the maintained delay information of the VOQ associated with the identifiable output queue.

10. The apparatus according to claim 1, wherein, Each specific next VOQ identifier in the plurality of next VOQ identifiers is determined based on a random value from a non-empty VOQ in the corresponding output queue of the plurality of output queues.

11. The apparatus according to claim 10, wherein, Each of the plurality of specific next VOQ identifiers has the longest delay of the non-empty VOQ and is selected based on a weighted value and the random value, wherein the weighted value is determined based on the longest delay and the shortest delay, which are identified in the maintained delay information of the non-empty VOQ in the corresponding output queue.

12. The apparatus according to claim 10 or 11, wherein, Each of the plurality of specific next VOQ identifiers does not have the longest delay of the non-empty VOQ and is selected based on a weighted value and the random value, wherein the weighted value is determined based on the longest delay and the shortest delay, which are identified in the maintained delay information of the non-empty VOQ in the corresponding output queue.

13. A method for data communication, comprising: A virtual output queue (VOQ) is maintained for each distinct ingress path among multiple ingress paths, wherein each VOQ is associated with an output queue; For each specific packet among multiple packets received on a specific ingress path in the ingress path, an encapsulation entity is enqueued in a specific VOQ in the specific ingress path of the specific packet, wherein the encapsulation entity includes a specific packet entity of the specific packet and a timestamp; For each of a plurality of internal communication cycles, and in response to a VOQ request generated by the VOQ scheduler that identifies a particular VOQ, a particular encapsulation entity is dequeued from the particular VOQ having the particular encapsulation entity, wherein the particular encapsulation entity includes a particular packet entity and a particular timestamp, and after the particular encapsulation packet entity is transmitted through one or more communication mechanisms, the particular packet entity is enqueued in the output queue, and the VOQ delay data structure is maintained based on the particular timestamp; as well as Based on the order of the packet entities dequeued from the output queue, packets are forwarded from the output queue via the exit path; The VOQ scheduler maintains the delay information of the VOQ in the VOQ delay data structure, which is independent of the delay information maintained by other VOQs in other output queues.

14. The method according to claim 13, wherein, Each of the plurality of packets received on a specific ingress path is stored in a shared memory system; wherein each specific packet entity in the specific packet entity includes a packet descriptor; and wherein the method includes: retrieving the corresponding packet from the shared memory system based on the packet descriptor.

15. The method according to claim 14, wherein, The retrieval of the corresponding packet from the received packets from the shared memory system is performed in response to a specific packet entity being dequeued from the output queue, wherein the specific packet entity includes the packet descriptor.

16. The method according to any one of claims 13 to 15, wherein, The VOQ scheduler maintains a first-in, first-out (FIFO) queue, wherein the FIFO queue has entries including VOQ identifiers for each non-empty VOQ associated with the output queue; and Wherein, after the VOQ scheduler determines the specific VOQ from the first VOQ identifier currently located at the head of the FIFO queue, the first VOQ identifier is located at the head or tail of the FIFO queue in response to the adaptive decision of the VOQ scheduler, based on the current latency information associated with the VOQ of the output queue.

17. The method according to any one of claims 13 to 15, wherein, The VOQ scheduler determines the specific VOQ based on the current latency information of the VOQ and a random value, wherein, in response to a first result of the ordering decision, based on a weighted value and a random value, one of the VOQs with the longest latency is selected as the specific VOQ, and in response to a second result of the ordering decision, one of the VOQs is randomly selected; and The weighting value is determined based on the longest and shortest delays identified in the VOQ delay data structure.

18. An apparatus for data communication, comprising: Multiple egress interface groups, wherein each particular egress interface group in the multiple egress interface groups includes: multiple output queues, a virtual output queue (VOQ) scheduler, and one or more VOQ delay data structures stored in memory; Multiple ingress interface groups, wherein each specific ingress interface group includes a specific VOQ for storing a corresponding packet entity for each specific output queue of the multiple egress interface groups, wherein each packet entity includes a timestamp and a packet descriptor of a corresponding packet received by the specific ingress interface group; A communication mechanism communicatively couples each ingress interface group in the ingress interface group to each egress interface group in the egress interface group, including selectively transmitting packet entities from the ingress interface group to the egress interface group; wherein, in response to a received packet entity, each egress interface group maintains delay information in the corresponding VOQ delay data structure based on the timestamp of the received packet entity, and enqueues the packet descriptor of the received packet entity in the corresponding output queue of the output queue; and A shared packet storage system communicatively coupled to each ingress interface group in the ingress interface group and each egress interface group in the egress interface group, wherein each ingress interface group stores packets in the shared packet storage system, and wherein each egress interface group retrieves packets from the shared storage system based on a corresponding packet descriptor; The VOQ scheduler determines the next VOQ identifier for each output queue in the same output interface group of the same output interface group, regardless of other output queues in the same output interface group, wherein the order of the selectively transmitted packet entities is identified from the determined next VOQ identifier.

19. The apparatus according to claim 18, wherein, The specific VOQ scheduler for each specific output queue in the plurality of egress interface groups maintains a first-in-first-out (FIFO) queue for each specific output queue in the output queues of the specific egress interface group. The FIFO queue has an entry including an entry for each non-empty VOQ associated with the specific output queue. The specific VOQ scheduler determines the next VOQ identifier from a first VOQ identifier located at the head of the corresponding FIFO queue and adaptively leaves the first VOQ identifier at the head or moves it to the tail of the corresponding FIFO queue based on the current latency information associated with the VOQ of the specific output queue.

20. The apparatus according to claim 18 or 19, wherein, The specific VOQ scheduler for each specific output queue in the plurality of egress interface groups determines each specific next VOQ identifier for each specific output queue in the specific output queue on the specific egress interface group based on a random value and current latency information of the non-empty VOQ associated with the specific output queue; and wherein, in response to a first result of the sorting decision, based on a weighted value and a random value, the specific next VOQ identifier is the non-empty VOQ with the longest latency of the non-empty VOQ associated with the specific output queue; In response to a second result of the sorting decision, one of the non-empty VOQs associated with the particular output queue is randomly selected.

21. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 13 to 17.

22. A computer-readable medium having instructions stored thereon, which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 13 to 17.

Images