Multi-data path support for low latency traffic manager

Abstract

Embodiments of the present disclosure relate to multiple data path support for low latency traffic managers. Techniques described herein may be implemented to support handling of CT and SAF traffic. A common packet data buffer is allocated to store incoming CT and SAF packet data. The SAF packet control data is directed onto a control data path having a first processing engine to a scheduler at a first latency. After being processed in a second control path by a second processing engine that bypasses a subset of the first processing engines, the CT packet control data is directed onto the second control data path to the scheduler at a second latency that is less than the first latency. And using the CT and SAF data packet control data to generate CT and SAF data packet dequeue requests for the CT and SAF data packets, respectively, and merging the CT and SAF data packet dequeue requests into a merged dequeue request sequence to retrieve corresponding data packet data from the common data packet data buffer based on the merged dequeue request sequence.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 620,414, filed on January 12, 2024, and the corresponding U.S. non-provisional application, which are hereby incorporated by reference. Technical Field

[0003] Embodiments relate generally to computer network communications, and more particularly, to processing cut-through (CT) and store-and-forward (SAF) traffic. Background Art

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art merely by virtue of its inclusion in this section.

[0005] Cut-through services can be supported by a network or (multiple) network switching devices therein to reduce latency and increase data transmission speed, which is particularly beneficial in environments where speed or low latency is critical, such as high-performance computing, real-time applications, data transmission within or between data centers, or time-sensitive services.

[0006] While cut-through switching can provide lower latency and faster packet forwarding, it presents significant challenges. Frequent handoffs and a mix of cut-through and store-and-forward traffic can lead to packet reordering, increased latency, or inefficiencies. Dedicated cut-through networks or network paths can avoid some of these issues, but such a setup can be impractical in large or complex network environments, especially in high-throughput networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The subject matter of the invention is illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements and in which:

[0008] Figure 1 An example framework for processing and forwarding CT traffic and SAF traffic is shown;

[0009] Figure 2A Example aspects of an example network system are shown; Figure 2B Example aspects of a network device are shown;

[0010] Figure 3A Example operations for processing and forwarding CT traffic and SAF traffic are shown;

[0011] Figure 3B An example packet control datapath merge operation is shown; and

[0012] Figure 4 An example process flow is shown. DETAILED DESCRIPTION

[0013] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the subject matter of the present invention. However, it is apparent that the subject matter of the present invention can be implemented without these specific details. In other examples, known structures and equipment are shown in block diagram form to avoid unnecessarily obscuring the subject matter of the present invention.

[0014] 1.0 General Overview

[0015] The techniques described herein can be implemented or used with network devices or nodes such as network (e.g., Ethernet, etc.) switches or (e.g., IP, etc.) routers in a computer communication network to support both cut-through (CT) and store-and-forward (SAF) services that share the common resources of the network device or node. These techniques can ensure relatively low or lowest possible latency for CT services while still maintaining relatively high performance for SAF services.

[0016] In some operational scenarios, multiple link structures may be used in the packet control datapath to support queuing and dequeuing of SAF packets by a network device / node. These multiple link structures in the packet control datapath may be specifically designed or implemented to relatively efficiently store SAF packets to be received and forwarded by a network device / node. As used herein, the term "operation" may refer to one or more actions taken or performed by a corresponding specific device, device component, logic, logic component, processing engine, etc.

[0017] In some approaches, the same or similar link structures and / or the same or similar control data paths and / or the same or similar operations on the control data paths may be used in queuing and dequeuing CT packets. Delay matching may need to be implemented in these approaches for dequeuing CT and SAF packets from a common buffer at an egress port, resulting in additional latency for forwarding CT packets due to the matched CT and SAF delays.

[0018] In contrast, under the techniques described herein, a dedicated CT control data path, separate from the SAF control data path, is created to support queuing and dequeuing of CT packets using a separate link structure that is also separate from the multiple link structures used in the SAF control data path. Additionally, optionally or alternatively, in some operational scenarios, the CT and SAF paths may utilize components selected from a superset that includes the same processing components. While path-specific templates, such as path-specific control data structures (e.g., inter-packet and intra-packet link data structures, etc.), path-specific path control data field values, etc., are used to select the specific composition of components used in the respective CT or SAF paths, there may be overlap with some of the same components used for both CT and SAF paths.

[0019] As a result, even though common components may be used in both the CT and SAF paths, the entire CT dequeue pipeline or control data path may exhibit or generate different latency than the entire SAF dequeue pipeline or control data path without requiring latency matching to be performed in the packet dequeue operation.

[0020] To support sharing of common resources of a network switch / node, such as packet data buffers of an egress port for both CT and SAF packets, Dequeue Request Path Merger (DRPM) logic may be used to manage or avoid conflicts or contention, such as buffer access conflicts or contention, between dequeuing CT and SAF packets to be forwarded out through the egress port.

[0021] Methods, techniques, and mechanisms for handling cut-through (CT) and store-and-forward (SAF) traffic are disclosed. In one embodiment, a common packet data buffer is allocated for an egress port to store incoming packet data, including both CT packets and SAF packets. The CT packets and SAF packets are forwarded out through the same egress port (e.g., to the same or different destination addresses, etc.). Upon receipt, SAF packet control data for the SAF packets is directed onto a control data path defined by a first plurality of processing engines. The SAF control data arrives at a scheduling logic engine with a first latency after being processed by the first plurality of processing engines. Upon receipt, the CT packet control data for the CT packets is directed onto a second control data path. After being processed in the second control path by a second plurality of processing engines that bypasses at least one or more of the first plurality of processing engines, the CT control data arrives at the scheduling logic engine with a second latency less than the first latency. A CT packet dequeue request is generated for the CT packet using the CT packet control data, while a SAF dequeue request is generated for the SAF packet using the SAF packet control data. The CT packet dequeue request and the SAF dequeue request are merged into a merged dequeue request sequence. Packet data is retrieved from the common packet data buffer based on the merged sequence of dequeue requests.

[0022] In other aspects, the present subject matter includes computer devices and / or computer-readable media configured to perform the foregoing techniques.

[0023] 2.0. Structural Overview

[0024] Figure 1 An exemplary framework for processing and forwarding CT traffic and SAF traffic that share common resources of network devices / nodes in a communication network as described herein is shown. For example, a network device / node (e.g., Figure 2A 110, etc.) can be a single networked computing device (or network device), such as a router or switch, in which some or all of the processing components described herein are implemented in an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or (multiple) other integrated circuits. As another example, a network device / node can include: one or more memories storing instructions for implementing the various components described herein; one or more hardware processors configured to execute the instructions stored in the one or more memories; and various data repositories in the one or more memories for storing data structures utilized and manipulated by the various components.

[0025] like Figure 1 As shown, a network device / node may include a traffic manager that operates with other packet processing components and / or resources in the network device / node to process and forward CT and SAF traffic.

[0026] In response to receiving an input data packet (for forwarding to a next hop toward a destination (address)) or a data unit or cell in a group of cells that constitute a data packet, the service manager may, for example, generate or access input data packet control data or data packet metadata for the input data packet based at least in part on a packet data field of the input data packet.

[0027] The traffic manager, or a processing component operating in conjunction with the traffic manager, then determines whether the incoming data packet qualifies as a CT data packet.

[0028] The common or shared processing components of the CT path and the SAF path may include (common and shared) egress ports, (common or shared) packet data buffers for the (common or shared) egress ports, (common and shared) buffer logic or managers, (common and shared) schedulers, (common and shared) path mergers, etc. These processing components may be used to operate different (path-specific) queues and FIFOs, different (path-specific) scheduling algorithms for different (path-specific) packet reception, enqueuing, dequeuing, etc., as described below.

[0029] In response to determining that the incoming packet qualifies as a CT packet, the service manager directs the incoming packet control data for the CT packet (referred to as CT packet control data for simplicity) to a dedicated CT packet control data path. Alternatively, in response to determining that the incoming packet does not qualify as a CT packet but is a SAF packet, the service manager directs the incoming packet control data for the SAF packet (referred to as SAF packet control data for simplicity) to a dedicated SAF packet control data path.

[0030] If the input data packet is determined to be a CT data packet, the CT data packet control data path for the CT data packet may include performing enqueue and dequeue operations for the CT data packet. The CT data packet enqueue operation may generate (multiple) data write requests to request the buffer allocation logic (or buffer manager) to buffer some or all input data in the CT data packet in a data buffer shared by the CT and SAF services through the corresponding egress port.

[0031] If the input packet is determined to be a SAF packet, the SAF packet control data path for the SAF packet may include performing enqueue and dequeue operations for the SAF packet, as well as SAF-specific or SAF-only operations ( Figure 1 Not shown; see e.g. Figure 2B and Figure 3AThe SAF packet enqueue operation may generate (multiple) data write requests to request the buffer allocation logic (or buffer manager) to buffer some or all of the input data of the SAF packet in a data buffer shared by the CT and SAF services through the corresponding egress port. SAF-specific operations or SAF-only operations are not performed on CT packets on the CT packet control data path, but are specific to or performed only on the SAF packet control data path.

[0032] The traffic manager may include a scheduler or operate in conjunction with a scheduler for an egress port to manage how data packets are processed and forwarded when multiple input data packets are waiting to be sent through the egress port.

[0033] In some operational scenarios, a single CT queue may be established by the service manager or scheduler to schedule the dequeueing of incoming CT packets for downstream processing, including but not limited to packet transmission operations. In contrast, multiple SAF queues may be established by the service manager or scheduler to schedule the dequeueing of incoming SAF packets for downstream processing.

[0034] The incoming packet control data or corresponding queuing / linking data or reference pointers can be enqueued into different queues established by the service manager or scheduler (e.g., CT or SAF, different QoS SAFs, different priority SAFs, different service class / type SAFs, etc.). The packet control data or queuing / linking data can include or correspond to path-specific templates. An example path-specific template can be a set of path-specific packet-related or packet-specific data structures and / or path-specific data field values ​​maintained, for example, in an inter-packet linked list, an inter-cell linked list, an intra-packet linked list, etc.

[0035] The scheduler implements a CT dequeue algorithm (such as a first-come, first-served dequeue algorithm) to dequeue an element from a CT queue (e.g., the head of a queue) or generate a CT packet dequeue request within a clock cycle. Alternatively or alternatively, the scheduler may implement an optimal scheduling algorithm to dequeue any element (e.g., the head of a queue) present in the CT queue without waiting.

[0036] The scheduler implements a SAF dequeue algorithm, such as one or more of the following algorithms: a first-come, first-served (FCFS) algorithm, in which SAF packets are forwarded in the order in which they arrive at the SAF queue(s); a weighted round-robin (WRR), in which a fixed round-robin time slot is assigned to each of some or all SAF queues, but larger time slots may be assigned to (multiple) SAF queues with higher priorities; a priority scheduling, in which packets in (multiple) high-priority SAF queues are prioritized and (multiple) low-priority SAF queues may be preempted; a loss-making round-robin (DRR), in which fairness is ensured between some or all SAF queues while still maintaining priority for time-sensitive services; and so on.

[0037] In order to arbitrate or allocate shared resources (such as bandwidth of the same egress port and / or read access to the same data buffer at the egress) between CT and SAF services and minimize inter-cell jitter, CT and SAF packet dequeue requests from the CT and SAF packet control data paths (or channels) will be merged with or at the dequeue request (path) merger or DRPM. As used herein, some or all packet scheduling operations such as CT or SAF packet scheduling performed by the service manager or scheduler and / or the DRPM therein may refer to (packet-based or cell-based) scheduling operations on one or more subdivided data units in a packet, such as scheduling a single cell or individual cells in a cell group in a CT or SAF packet. In addition, optionally or alternatively, in addition to scheduling operations, other operations such as buffer storage operations, buffer retrieval operations, queuing operations, dequeue operations, merging operations, etc. may be performed on a packet or cell basis.

[0038] To support relatively low latency arbitration (e.g., 1 to 3 clock cycle latency, etc.), the merger can establish, maintain, or use a store-and-forward request (SRF) FIFO and a cut-through request (CRF) FIFO, which can be sized specifically or separately for CT and SAF traffic. The merger can be implemented with relatively simple arbitration logic to select the earliest arriving SRF and CRF headers (e.g., current or upcoming) from the SRF and CRF FIFOs, respectively.

[0039] Although the same or common scheduler and merger are used for both the CT and SAF packet control datapaths (e.g., along with buffer allocation logic and data buffers, etc.), the CT packet control path (which is a simplified path compared to the (full) SAF packet control datapath) results in lower latency from the scheduler to the DRPM. This is at least in part due to the use of a relatively simple CT connection structure compared to the SAF connection structure. Furthermore, the relatively low (CT control path) latency is due to the fact that SAF-specific or SAF-only operations are excluded from execution on the CT packet control datapath.

[0040] In some operating scenarios, (e.g., at most) one CT packet dequeue request from a CT queue maintained by the scheduler may arrive at or appear in a CT-specific FIFO maintained by a consolidator (DRPM) per clock cycle. Additionally, optionally or alternatively, at most one SAF packet dequeue request from some or all SAF queues maintained by the scheduler may arrive at or appear in a SAF-specific FIFO maintained by the DRPM per clock cycle. As used herein, the term "consolidator" or "DRPM" may refer to a processing component that may also be implemented as logic (e.g., hardware, etc.). The DRPM may maintain a CRF FIFO and an SRF FIFO, each of which is specifically and separately sized or optimized to absorb intermittent bursts caused at least in part by the difference in latency between the SAF and CT packet control data paths. The scheduler assigns a (DRPM) arrival timestamp to each CT or SAF packet dequeue request that arrives at or appears in the DRPM so that it enters at the tail (end) of the CT or SAF FIFO.

[0041] The DRPM can implement an earliest-first arbiter that prioritizes either the SAF or the CT in controlling the departure from the CRF and SRF FIFOs maintained by the DRPM in the event that both SAF and CT dequeue requests arrive simultaneously in their respective FIFOs maintained by the DRPM. The DRPM arrival timestamps of the CRF and SRF headers of the SRF and SRF FIFOs are compared during the dequeue request merging operation.

[0042] If both CRF and SRF headers are present in the CRF and SRF FIFOs, then the earliest of the CRF and SRF headers indicated by the corresponding DRPM arrival timestamps is dequeued or selected by the DRPM in a common or merged sequence of packet dequeue requests sent or provided by the DRPM to the buffer allocation logic.

[0043] If only one of the CRF and SRF FIFOs has data or entries, its header is dequeued and included in the common packet dequeue request sequence.

[0044] Therefore, the DRPM enforces a constraint (e.g., I / O resources, timing control, etc.) by which at most one dequeue request can leave the DRPM for each (e.g., read, etc.) clock cycle to a buffer allocation. In some operational scenarios, a dequeue request leaves if either the SRF or CRF FIFO (or both) has data or entries.

[0045] The dequeue (request) path merging operation described herein allows pending CT packets to use or occupy any bandwidth not used by any SAF packets that arrived at the DRPM from the scheduler before the CT packets, when the opportunity arises. At the same time, this allows pending SAF packets that arrived at the DRPM earlier than some CT packets to continue to use egress (port) bandwidth with a target / intended / optimized bandwidth allocation for the SAF traffic (e.g., no or little impact from the CT traffic, depending on the volume / volume of the CT traffic; no or little inter-cell jitter that may be introduced when a CT packet preempts an earlier arriving SAF packet from the egress port), depending on the scheduling / dequeue algorithm implemented by the scheduler and / or DRPM.

[0046] After a dequeue request corresponding to a CRF or SRF header (entry) in a CRF or SRF FIFO leaves the DRPM, goes to the buffer allocation logic or buffer manager, or is dispatched by the DRPM to the buffer allocation logic or buffer manager, the CT and SAF packet data control paths merge into the same or common packet control path or subpath, in which the same or common packet processing operations can be performed—generating output packet control data, obtaining input packet data using (multiple) data read requests, generating output packet data, forwarding output network / data packets corresponding to input network / data packets, etc.

[0047] In some operational scenarios, for a given (e.g., CT, SAF, etc.) data packet or cell thereof, when the data packet or cell is received by the ingress processor, it only results in a single data write request to the data buffer, and when the data packet or cell is to be sent or forwarded from the egress port, it only results in a single read request to the same data buffer.

[0048] 3.0. Packet Communication Network

[0049] Figure 2AExample aspects of an example networked system 100 (also referred to as a network) in which the techniques described herein may be practiced are shown, according to an embodiment. The network system 100 includes a plurality of interconnected nodes 110a-110n (collectively, nodes 110), each of which is implemented by a different computing device. For example, the node 110 may be a single networked computing device (or network device), such as a router or switch, in which some or all of the processing components described herein are implemented in an application specific integrated circuit (ASIC, field programmable gate array (FPGA), or other integrated circuit. As another example, the node 110 may include one or more memories (e.g., non-transitory computer-readable media, etc.) that store instructions for implementing the various components described herein, one or more hardware processors configured to execute the instructions stored in the one or more memories, and various data repositories in the one or more memories that store data structures utilized and manipulated by the various components.

[0050] Each node 110 is connected to one or more other nodes 110 in the network 100 via one or more communication links. The communication links can be any suitable wired cables or wireless links. Note that the system 100 illustrates only one of many possible arrangements of nodes within the network. Other networks may include fewer or more nodes 110 with any number of links between them.

[0051] While each node 110 may or may not have various other functionalities, in an embodiment, each node 110 is configured to send, receive, and / or relay data to one or more other nodes 110 via these links. Typically, data is transmitted as a series of discrete units or data structures represented by signals transmitted over the communication links. Figure 2A As shown, some or all nodes, including but not necessarily limited to node 100c, may implement some or all of the template-based CT / SAF path selection techniques described herein.

[0052] 3.1. Packets and other data units

[0053] Different nodes 110 within the network 100 may send, receive, and / or relay data units at different communication levels or layers. For example, a first node 110 may send a data unit (e.g., a TCP segment, an IP packet, etc.) at the network layer to a second node 110 via a path that includes intermediate nodes 110. Before the data unit is sent from the first node 110, the data unit will be divided into smaller data units at various sub-levels. These smaller data units may be referred to as "sub-units" or "portions" of the larger data unit.

[0054] For example, the data unit may be sent in one or more of the following: a data packet, a cell, a collection of signal encoding bits, etc. to the intermediate node 110. Depending on the network type and / or device type of the intermediate node 110, the intermediate node 110 may reconstruct the entire original data unit before routing the information to the second node 110, or the intermediate node 110 may simply reconstruct certain sub-units of the data (e.g., frames and / or cells) and route these sub-units to the second node 110 without reconstructing the entire original data unit.

[0055] When a node 110 receives a data unit, it typically examines the addressing information within the data unit (and / or other information within the data unit) to determine how to process the data unit. The addressing information may be, for example, an Internet Protocol (IP) address, an MPLS label, or any other suitable information. If the addressing information indicates that the receiving node 110 is not the destination of the data unit, the receiving node 110 may look up the destination node 110 within the receiving node's routing information and, based on the forwarding instructions associated with the destination node 110 (or the address group to which the destination node belongs), route the data unit to another node 110 connected to the receiving node 110. The forwarding instructions may indicate, for example, the egress port through which the data unit is to be sent, a label to which the data unit is to be attached, the next hop, etc. In the event that multiple (e.g., equal cost, unequal cost, etc.) paths to the destination node 110 are possible, the forwarding instructions may include information indicating an appropriate method for selecting one of these paths, or a path that is considered the best path may already be defined.

[0056] The addressing information, flags, labels, and other metadata used to determine how to process a data unit are typically embedded in a portion of the data unit called a header. The header is typically located at the beginning of the data unit and is followed by the data unit's payload, which is the information actually sent in the data unit. The header typically includes different types of fields, such as a destination address field, a source address field, a destination port field, a source port field, and so on. In some protocols, the number and arrangement of fields can be fixed. Other protocols allow for an arbitrary number of fields, with some or all of the fields preceded by type information that explains the meaning of the field to the node.

[0057] A traffic flow is a sequence of data units, such as data packets, having a common attribute that generally travels from the same source to the same destination. In an embodiment, the source of the traffic flow can mark each data unit in the sequence as a member of a flow using a label, tag, or other suitable identifier within the data unit. In another embodiment, a flow is identified by deriving an identifier from other fields in the data unit (e.g., a "five-tuple" or "5-tuple" combination of source address, source port, destination address, destination port, and protocol). Flows are typically intended to be sent sequentially, and network devices can therefore be configured to send all data units in a given flow along the same path to ensure that the flow is received sequentially.

[0058] A data unit may be single-destination or multi-destination. A single-destination data unit is typically a unicast data unit that specifies only a single destination address. A multi-destination data unit is typically a multicast data unit that specifies multiple destination addresses or an address shared by multiple destinations. However, a given node may treat a unicast data unit as having multiple destinations in certain circumstances. For example, the node may be configured to mirror the data unit to another port such as an enforcement port or a debug port, copy the data unit to a central processing unit for diagnostic purposes or suspicious activity, re-circulate the data unit, or take other actions that cause the unicast data unit to be sent to multiple destinations. A given data unit may treat a multicast data unit as a single-destination data unit in certain circumstances, through the same token, such as if all destinations targeted by the data unit are reachable through the same egress port.

[0059] For convenience, many of the techniques described in this disclosure are described with respect to routing data units that are IP packets in an L3 (level / layer 3) network, or routing constituent cells and frames thereof in an L2 (level / layer 2) network, where the techniques have particular advantages in that context. However, it should be noted that these techniques may also be applied to achieve advantages in routing other types of data units conforming to other protocols and / or at other communication layers within a network. Therefore, unless otherwise stated or apparent, the techniques described herein should also be understood to be applicable in contexts where a "data unit" is any other type of data structure (e.g., a segment or datagram) transmitted over a network. That is, in these contexts, other types of data structures may be used in place of packets, cells, frames, etc.

[0060] Note that the actual physical representation of a data unit can change due to the processes described herein. For example, when a data unit is moved from one component to another within a network device or even between network devices, the data unit can be converted from a physical representation at a specific location in one memory to a signal-based representation and back to a physical representation at a different location in a potentially different memory. This movement may technically involve deleting, converting, and / or copying some or all of the data units any number of times. However, for simplicity, even if the physical representation of a data unit changes, it is logically considered that the data unit remains the same data unit during transmission in the device. Similarly, the content and / or structure of a data unit can change as it is processed, such as by adding or deleting header information, adjusting cell boundaries, or even modifying payload data. However, even after changing its content and / or structure, the modified data unit is still referred to as the same data unit.

[0061] 3.2. Network Path

[0062] Any node in the illustrated network 100 can communicate with any other node in the network 100 by sending data units through a series of nodes 110 and links (referred to as paths). For example, node B (110b) can send a data unit to node H (110h) via a path from node B to node D to node E to node H. There may be a large number of valid paths between two nodes. For example, another path from node B to node H is from node B to node D to node G to node H.

[0063] In an embodiment, a node 110 does not actually need to specify a complete path for the data units it sends. Instead, a node 110 can simply be configured to calculate the best path for a data unit to exit the device (e.g., which egress port it should send the data unit to, etc.). When a node 110 receives a data unit that is not directly addressed to the node 110, based on header information associated with the data unit, such as path and / or destination information, the node 110 relays the data unit to the destination node 110, or the node 110 calculates that a "next hop" node 110 is in a better position to relay the data unit to the destination node 110. In this way, the actual path of the data unit is the product of each node 110 along the path making routing decisions about how best to move the data unit to the destination node 110 identified by the data unit.

[0064] 4.0. Network Equipment

[0065] Figure 2BExample aspects of an example network device 200 in accordance with an embodiment are shown, in which the techniques described herein may be practiced. The network device 200 is a computing device comprising any combination of hardware and software that is configured to implement the various logical components described herein, including components 210-290. For example, the device may be a single networked computing device, such as a router or switch, in which application-specific integrated circuits (ASICs) are used to implement some or all of the components 210-290 described herein. As another example, the implementing device may include: one or more memories storing instructions for implementing the various components described herein; one or more hardware processors configured to execute the instructions stored in the one or more memories; and various data repositories in the one or more memories for storing data structures utilized and manipulated by the various components 210-290.

[0066] Device 200 is generally configured to receive data unit 205 and forward it to other devices in a network (e.g., network 100) through a series of operations performed at various components within device 200. Note that in embodiments, some or all of nodes 110 in system 100 may each be or include a separate network device 200. In embodiments, a node 110 may include more than one device 200. In embodiments, device 200 itself may be one of multiple components within node 110. For example, network device 200 may be an integrated circuit or "chip" dedicated to performing switching and / or routing functions within a network switch or router. In embodiments, the network switch or router also includes one or more central processing units, storage units, memory, physical interfaces, LED displays, or other components external to the chip, some or all of which may be in communication with the chip.

[0067] A non-limiting example flow of data unit 205 through the various subcomponents of the forwarding logic of device 200 is as follows. After being received via port 210, data unit 205 may be buffered in ingress buffer 224 and queued in ingress queue 225 by ingress arbiter 220 until data unit 205 can be processed by ingress packet processor 230 and then delivered to an interconnect (or cross-connect) such as a switch fabric. Data unit 205 may be forwarded from the interconnect to traffic manager 240. Traffic manager 240 may store data unit 205 in egress buffer 244 and assign data unit 205 to egress queue 245. Traffic manager 240 manages traffic of data unit 205 through egress queue 245 until data unit 205 is released to egress packet processor 250. Depending on the process, traffic manager 240 may then assign data unit 205 to another queue so that it can be processed by another egress processor 250, or egress packet processor 250 may send data unit 205 to egress arbiter 260, which temporally stores or buffers data unit 205 in a transmit buffer and ultimately forwards the data unit out via another port 290. Of course, depending on the embodiment, the forwarding logic may omit some of these subcomponents and / or include other subcomponents in a different arrangement.

[0068] Example components of device 200 are now described in more detail.

[0069] 4.1.Port

[0070] Network device 200 includes ports 210 / 290. Ports 210 (including ports 210-1 through 210-N) are inbound ("ingress") ports through which data units 205 are received on a network, such as network 110. Ports 290, including ports 290-1 through 290-N, are outbound ("egress") ports through which at least some data units 205, after being processed by network device 200, are sent to other destinations within the network.

[0071] The egress port 290 may operate in conjunction with a corresponding transmit buffer to store data units or sub-units (e.g., packets, cells, frames, transmission units, etc.) divided therefrom to be transmitted through the port 290. The transmit buffer may have a one-to-one correspondence with the port 290, a many-to-one correspondence with the port 290, etc. The egress processor 250 or the egress arbiter 260 operating in conjunction with the egress processor 250 may output these data units or sub-units to the transmit buffer before transmitting them out of the port 290.

[0072] The data unit 205 can be any suitable PDU type, such as a data packet, a cell, a frame, a transmission unit, etc. In an embodiment, the data unit 205 is a data packet. However, the individual atomic data units on which the described components can operate can actually be sub-units of the data unit 205. For example, the data unit 205 can be received, acted upon, and sent at the cell or frame level. These cells or frames can be logically linked together as the data units 205 to which they belong (e.g., data packets, etc.) to determine how to process these cells or frames. However, the sub-units may not actually be spliced ​​into the data unit 205 within the device 200, particularly if the sub-units are being forwarded to another destination through the device 200.

[0073] For purposes of illustration, ports 210 / 290 are depicted as separate ports, but may actually correspond to the same physical hardware port (e.g., a network jack or interface, etc.) on network device 210. That is, network device 200 can receive data units 205 and transmit data units 205 through a single physical port, and the single physical port can therefore function as both an ingress port 210 (e.g., one of 210a, 210b, 210c, ..., 210n, etc.) and an egress port 290. However, for various functional purposes, certain logic of network device 200 may treat the single physical port as a separate ingress port 210 and a separate egress port 290. Furthermore, for various functional purposes, certain logic of network device 200 may subdivide a single physical ingress port or egress port into multiple ingress ports 210 or egress ports 290, or aggregate multiple physical ingress ports or egress ports into a single ingress port 210 or egress port 290. Therefore, in some operational scenarios, ports 210 and 290 should be understood as different logical constructs that map to physical ports, rather than simply as different physical constructs.

[0074] In some embodiments, the ports 210 / 290 of the device 200 can be coupled to one or more transceivers, such as serializer / deserializer ("SerDes") blocks. For example, the port 210 can provide a parallel input of received data units to a SerDes block, which then serially outputs the data units to the ingress packet processor 230. On the other end, the egress packet processor 250 can serially input the data units to another SerDes block, which then outputs the data units in parallel to the port 290.

[0075] 4.2. Packet Processor

[0076] The device 200 includes one or more packet processing components that collectively implement forwarding logic by which the device 200 determines how to process each data unit 205 received at the device 200. These packet processing components can be any suitable combination of fixed circuitry and / or software-based logic, such as specific logic components implemented by one or more field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), or a general-purpose processor that executes software instructions.

[0077] Different packet processors 230 and 250 can be configured to perform different packet processing tasks. These tasks can include, for example, identifying a path along which to forward data unit 205, forwarding data unit 205 to egress port 290, implementing flow control and / or other policies, manipulating packets, performing statistics or debugging operations, etc. Device 200 can include any number of packet processors 230 and 250 configured to perform any number of processing tasks.

[0078] In an embodiment, the packet processors 230 and 250 within the device 200 can be arranged so that the output of one packet processor 230 or 250 can ultimately be input to another packet processor 230 or 250 by passing the data unit 205 from one packet processor 230 and / or 250 to the other packet processors 230 and / or 250 in a sequence of stages until the data unit 205 is ultimately disposed of (e.g., by sending the data unit 205 out of the egress port 290, "dropping" the data unit 205, etc.). In some embodiments, the exact set and / or sequence of packet processors 230 and / or 250 that process a given data unit 205 can vary depending on the attributes of the data unit 205 and / or the state of the device 200. There is no limit to the number of packet processors 230 and / or 250 that can be chained together in this manner.

[0079] Based on the decisions made while processing data unit 205, in some embodiments and / or for certain processing tasks, packet processor 230 or 250 may directly manipulate data unit 205. For example, packet processor 230 or 250 may add, delete, or modify information in a data unit header or payload. In other embodiments and / or for other processing tasks, packet processor 230 or 250 may generate control information that accompanies or is merged with data unit 205 as data unit 205 continues through device 200. This control information may then be used by other components of device 200 to implement the decisions made by packet processor 230 or 250. In some operational scenarios, the data unit that is actually processed by the processing pipeline (while the original payload and header are stored in memory) may be referred to as a descriptor (or template).

[0080] In an embodiment, the packet processor 230 or 250 does not have to process the entire data unit 205, but can only receive and process a sub-unit of the data unit 205 including the header information of the data unit. For example, if the data unit 205 is a data packet including multiple cells, the first cell or a first subset of cells can be forwarded to the packet processor 230 or 250, while the remaining cells of the data packet (and possibly the first cell(s)) are forwarded in parallel to the merging component, where they await processing results.

[0081] In embodiments, packet processors can be generally categorized as either ingress packet processors 230 or egress packet processors 250. Typically, an ingress processor 230 parses the destination of a traffic manager 240 to determine which egress port 290 (e.g., one of 290a, 290b, 290c, ..., 290n, etc.) and / or queue a data unit 205 should exit from. There can be any number of ingress processors 230, including only a single ingress processor 230.

[0082] In an embodiment, the ingress processor 230 performs certain receive tasks on the data units 205 as they arrive. These receive tasks may include, for example, but not limited to, parsing the data units 205, performing routing-related lookup operations, blocking the data units 205 categorically with certain attributes and / or when the device 200 is in certain states, duplicating certain types of data units 205, performing initial classification of the data units 205, etc. Once the appropriate receive task(s) have been performed, the data units 205 are forwarded to the appropriate traffic manager 240, to which the ingress processor 230 may be coupled directly or via various other components, such as interconnect components.

[0083] In contrast, the egress packet processor(s) 250 of the device 200 may be configured to perform non-receiving tasks necessary to implement the forwarding logic of the device 200. These tasks may include, for example, identifying paths along which data units 205 are forwarded, implementing flow control and / or other policies, manipulating data units, performing statistical or debugging operations, etc. In an embodiment, different egress packet processor(s) 250 may be assigned to different flows or other types of traffic, such that not all data units 205 will be processed by the same egress packet processor 250.

[0084] In an embodiment, each egress processor 250 is coupled to a different set of egress ports 290, which can send data units 205 processed by the egress processor 250 to the egress ports 290. In an embodiment, access to a set of ports 290 or corresponding transmit buffers of ports 290 can be regulated via an egress arbiter 260 coupled to the egress packet processor 250. In some embodiments, the egress processor 250 can also or instead be coupled to other potential destinations, such as an internal central processing unit, a storage subsystem, or a traffic manager 240.

[0085] Buffer

[0086] Because not all data units 205 received by the device 200 can be processed simultaneously by component(s) such as packet processors 230 and / or 250 and / or port 290, various components of the device 200 may temporarily store data units 205 in memory structures referred to as (e.g., ingress, egress, etc.) buffers while the data units 205 wait to be processed. For example, a particular packet processor 230 or 250 or port 290 may only be able to process a certain number of data units 205, such as a certain number of data units 205 or portions of data units 205, in a given clock cycle, which means that other data units 205 or portions of data units 205 destined for the packet processor 230 or 250 or port 290 must be ignored (e.g., discarded, etc.) or stored. At any given time, depending on network traffic conditions, a large number of data units 205 may be stored in the buffers of the device 200.

[0087] Device 200 may include various buffers, each for a different purpose and / or component. Typically, a data unit 205 awaiting processing by a component is held in a buffer associated with the component until the data unit 205 is "released" to the component for processing.

[0088] The buffer can be implemented using any number of different banks. Each bank can be a part of any type of memory, including volatile memory and / or non-volatile memory. In an embodiment, each bank includes many addressable "entries" (e.g., rows, columns, etc.) in which data units 205, subunits, link data, or other types of data can be stored. The size of each entry in a given bank is referred to as the "width" of the bank, and the number of entries in the bank is referred to as the "depth" of the bank. The number of banks can vary according to the embodiment.

[0089] Each memory bank may have associated access restrictions. For example, a memory bank may be implemented using a single-port memory that is only accessible once in a given time slot (e.g., a clock cycle, etc.). Thus, the device 200 may be configured to ensure that no more than one entry is read from or written to the memory bank in a given time slot. Alternatively, a memory bank may be implemented in a multi-port memory to support two or more accesses in a given time slot. However, in many cases, a single-port memory may be desirable for higher operating frequencies and / or to reduce costs.

[0090] In embodiments, in addition to buffer banks, the device may be configured to group certain banks together into logical banks that support additional reads or writes in time slots and / or higher write bandwidth. In embodiments, each bank (whether logical, physical, or another (e.g., addressable, hierarchical, multi-level, sub-bank, etc.) organizational structure) is capable of being accessed simultaneously with every other bank in the same clock cycle, although full implementation of this capability is not required.

[0091] Some or all components of device 200 that utilize one or more buffers may include a buffer manager configured to manage the use of these buffers. Among other processing tasks, the buffer manager may, for example, maintain a mapping of data units 205 to the buffer entries in which data for those data units 205 is stored, determine when a data unit 205 must be discarded because it cannot be stored in a buffer, perform garbage collection on buffer entries for data units 205 (or portions thereof) that are no longer needed, and so on.

[0092] The buffer manager may include buffer allocation logic. The buffer allocation logic is configured to identify which buffer entry or entries should be utilized to store a given data unit 205 or portion thereof. In some embodiments, each data unit 205 is stored in a single entry. In other embodiments, a data unit 205 is received as a component data unit portion for storage purposes, or is divided into component data unit portions for storage purposes. The buffer may store these component portions separately (e.g., not at the same address location or even within the same memory bank, etc.). One or more buffer entries storing a data unit 205 are marked as used (e.g., in a "free" list, free or available (if not marked as used), etc.) to prevent a newly received data unit 205 from overwriting an already buffered data unit 205. After a data unit 205 is released from the buffer, one or more entries in which the buffered data unit 205 was buffered may then be marked as available for storing a new data unit 205.

[0093] In some embodiments, the buffer allocation logic is relatively simple, as data units 205 or portions of data units are assigned to memory banks and / or specific entries within those memory banks, either randomly or using a round-robin approach. In some embodiments, data units 205 are assigned to buffers based at least in part on characteristics of those data units 205, such as their corresponding traffic flows, destination addresses, source addresses, ingress ports, and / or other metadata. For example, different memory banks may be utilized to store data units 205 received from different ports 210 or groups of ports 210. In some embodiments, the buffer allocation logic also or alternatively utilizes buffer state information (e.g., utilization metrics) to determine which memory bank and / or buffer entry to assign to a data unit 205 or portion thereof. Other allocation considerations may include buffer allocation rules (e.g., not writing two consecutive cells from the same memory bank to the same memory bank, etc.) and I / O scheduling conflicts, e.g., to avoid assigning a data unit to a memory bank when no write operations are available to that memory bank due to other components reading content already in the memory bank.

[0094] 4.4. Queues

[0095] In an embodiment, various components of the device 200 may implement queuing logic to manage the order in which data units 205 are processed from the buffers. For example, the flow of data units through the ingress buffer 224 may be managed using the ingress queue 225, while the flow of data units through the egress buffer 244 may be managed using the egress queue 245.

[0096] Each data unit 205 or buffer location(s) storing data units 205 is said to belong to one or more constructs called queues. In general, a queue is a set of memory locations (e.g., in buffers 224 and / or 244, etc.) arranged in some order by metadata describing the queue. The memory locations can (and typically are) non-contiguous with respect to their addressing scheme and / or physical or logical arrangement. For example, the metadata for a queue may indicate that the queue consists of, in order, entry addresses 2, 50, 3, and 82 in a certain buffer.

[0097] In various embodiments, the order in which a queue arranges its constituent data units 205 generally corresponds to the order in which the data units 205 or data unit portions in the queue will be released and processed. Such a queue is referred to as a first-in, first-out ("FIFO") queue, although other types of queues may be used in other embodiments. In some embodiments, the number of data units 205 or data unit portions allocated to a given queue at a given time may be limited globally or on a per-queue basis, and this limit may change over time.

[0098] 4.5. Business Manager

[0099] According to an embodiment, the device 200 also includes one or more traffic managers 240 configured to control the flow of data units to one or more packet processors 230 and / or 250. For example, a buffer manager (or buffer allocation logic) within the traffic manager 240 may temporarily store the data unit 205 in a buffer 244 while the data unit 205 awaits processing by the egress processor(s) 250. The traffic manager 240 may receive the data unit 205 directly from the port 210, from the ingress processor 230, and / or other appropriate components of the device 200. In an embodiment, the traffic manager 240 receives one TDU from each possible source (e.g., each port 210, etc.) during each clock cycle or other time slot.

[0100] The traffic manager 240 may include or be coupled to an egress buffer 244 for buffering the data units 205 before sending them to their respective egress processors 250. While the data units 205 are awaiting processing by the egress processors 250, a buffer manager within the traffic manager 240 may temporarily store the data units 205 in the egress buffers 244. The number of egress buffers 244 may vary depending on the embodiment. By reading the data units 205 from the (e.g., egress, etc.) buffers 244 and sending the data units 205 to the egress processors 250, the data units 205 or portions of data units in the egress buffers 244 may ultimately be "released" to one or more egress processors 250 for processing. In an embodiment, the traffic manager 240 may release up to a certain number of data units 205 from the buffers 244 to the egress processors 250 during each clock cycle or other defined time slot.

[0101] In addition to managing the use of buffers 244 to store data units 205 (or copies thereof), traffic manager 240 may include queue management logic configured to assign buffer entries to queues and manage the flow of data units 205 through the queues. For example, upon receiving a data unit 205, traffic manager 240 may identify a particular queue for a given data unit 205. Traffic manager 240 may also determine when to release (also referred to as "dequeue") a data unit 205 (or portion thereof) from a queue and provide these data units 205 to a particular packet processor 250. Buffer management logic in traffic manager 240 may also "deallocate" entries in buffers 244 that store data units 205 that are no longer linked to a queue of the traffic manager. These entries are then reclaimed through a garbage collection process to store new data.

[0102] In embodiments, different queues may exist for different destinations. For example, each port 210 and / or port 290 may have its own set of queues. For example, the queue to which an incoming data unit 205 is assigned and linked may be selected based on forwarding information indicating which port 290 the data unit 205 should exit. In embodiments, a different egress processor 250 may be associated with each different set of one or more queues. In embodiments, the current processing context of a data unit 205 may be used to select which queue the data unit 205 should be assigned.

[0103] In an embodiment, different queues may also or alternatively exist for different flows or sets of flows. That is, each identifiable service flow or service flow group is assigned its own set of queues, to which data units 205 are respectively assigned.

[0104] Device 200 may include any number (e.g., one or more, etc.) of packet processors 230 and / or 250 and traffic managers 240. For example, different groups of ports 210 and / or ports 290 may have their own traffic managers 240 and packet processors 230 and / or 250. As another example, in an embodiment, traffic managers 240 may be replicated for some or all stages of processing a data unit. For example, system 200 may include a traffic manager 240 and an egress packet processor 250 for the egress stage executed when a data unit 205 exits system 200, and / or traffic managers 240 and packet processors 230 or 250 for any number of intermediate stages. Thus, a data unit 205 may pass through any number of traffic managers 240 and / or packet processors 230 and / or 250 before exiting system 200.

[0105] In an embodiment, traffic manager 240 is coupled to ingress packet processor 230 such that data unit 205 (or portion thereof) is allocated to a buffer only upon initial processing by ingress packet processor 230. Once in egress buffer 244, data unit 205 (or portion thereof) may be "released" to one or more egress packet processors 250 for processing, either by traffic manager 240 sending link or other suitable addressing information for the corresponding buffer 244 to the egress packet processor 250, or by sending the data unit 205 directly.

[0106] During the processing of data unit 205, device 200 may replicate data unit 205 one or more times for purposes such as, but not limited to, multicasting, mirroring, debugging, etc. For example, a single data unit 205 may be replicated to multiple egress queues 245. Under the techniques described herein, any given replica of a data unit can be treated as a received packet to be routed or forwarded with a multipath group. For example, data unit 205 may be linked to a separate queue for each of ports 1, 3, and 5. As another example, data unit 205 may be replicated multiple times (e.g., to different egress processors 250, etc.) after it reaches the head of a queue. Therefore, while certain techniques described herein may refer to the original data unit 205 received by device 200, it should be noted that these techniques are equally applicable to replicas of data unit 205 generated for various purposes. A replica of data unit 205 may be partial or complete. Furthermore, actual replicas of data unit 205 may exist in a buffer, or a single replica of data unit 205 may be linked to multiple queues simultaneously from a single buffer location.

[0107] The service manager may implement a dedicated CT packet control data path (for CT services) that is separate from the SAF packet control data path for SAF services. The SAF packet control data path may include SAF-specific or SAF-only packet processing operations that are performed with SAF packets but not with CT packets.

[0108] To coordinate the use and access of common resources for CT and SAF traffic, such as common data buffers for both CT and SAF traffic passing through corresponding egress ports, the traffic manager may enhance or implement a scheduler that maintains separate queues for CT and SAF traffic and a dequeue request (path) merger or DRPM that merges CT and SAF packet dequeue requests transmitted from the egress port into a common packet dequeue request sequence.

[0109] Common dequeue request sequence for both CT and SAF services (in Figure 2B The CT or SAF packet data stored in the common data buffer of the egress port may be transformed or used to generate outgoing packet data to be included in a corresponding outgoing packet to be transmitted or forwarded through the egress port.

[0110] 4.6. Forwarding Logic

[0111] The logic of device 200 that determines how to process data unit 205 (such as where and whether to send data unit 205, whether to perform additional processing on data unit 205, etc.) is referred to as the forwarding logic of device 200. As described above, this forwarding logic is collectively implemented by various components of device 200. For example, ingress packet processor 230 may be responsible for parsing the destination of data unit 205 and determining a set of actions / edits to be performed on data unit 205, and egress packet processor 250 may perform the edits. Alternatively, in some cases, egress packet processor 250 may also determine the actions and parse the destination. Furthermore, embodiments may exist in which ingress packet processor 230 performs the edits.

[0112] Depending on the embodiment, the forwarding logic may be hard-coded and / or configurable. For example, in some cases, the forwarding logic of device 200, or portions thereof, may be at least partially hard-coded into one or more ingress processors 230 and / or egress processors 250. As another example, the forwarding logic or elements thereof may also be configurable in that the logic changes over time in response to analysis of state information collected from various components of device 200 and / or other nodes in the network in which device 200 is located, or instructions received from various components of device 200 and / or other nodes in the network in which device 200 is located.

[0113] In an embodiment, the device 200 typically stores one or more forwarding tables (or equivalent structures) in its memory that map certain data unit attributes or characteristics to actions to be taken with respect to a data unit 205 having those attributes or characteristics, such as sending the data unit 205 to a selected path or using a specified internal component to process the data unit 205. For example, such attributes or characteristics may include a quality of service level specified by the data unit 205 or associated with another characteristic of the data unit 205, a flow control group, an ingress port 210 through which the data unit 205 is received, a label or tag in a packet header, a source address, a destination address, a packet type, or any other suitable distinguishing characteristic. The traffic manager 240 may, for example, implement logic that reads such a table, determines one or more ports 290 to send the data unit 205 based on the table, and sends the data unit 205 to an egress processor 250 coupled to the one or more ports 290.

[0114] According to an embodiment, a forwarding table describes a group of one or more addresses, such as a subnet of IPv4 or IPv6 addresses. Each address is the address of a network device on the network, although a network device can have more than one address. Each group is associated with a potentially different set of one or more actions to be performed with respect to data units that are resolved to (e.g., directed to, etc.) the addresses within the group. Any suitable set of one or more actions can be associated with a group of addresses, including but not limited to forwarding a message to a specified "next hop", copying the message, changing the destination of the message, discarding the message, performing debugging or statistical operations, applying quality of service policies or flow control policies, etc.

[0115] For the purpose of illustration, these tables are described as "forwarding tables", but it should be noted that the extent of the action(s) described by these tables can be much greater than simply where to forward a message. For example, in an embodiment, a table can be a basic forwarding table that simply specifies the next hop for each group. In other embodiments, a table can describe one or more complex policies for each group. In addition, different types of tables can be used for different purposes. For example, one table can be a basic forwarding table that is compared with the destination address of each data packet, while another table can specify the policy to be applied to a data packet upon entry based on the destination (or source) group of the data packet, etc.

[0116] In an embodiment, the forwarding logic may read port state data for ports 210 / 290. The port state data may include, for example, flow control state information describing various service flows and associated service flow control rules or policies, link state information indicating an uplink or downlink, and port utilization information indicating how the port is utilized (e.g., utilization percentage, utilization status, etc.). The forwarding logic may be configured to implement associated rules or policies associated with the flow to which a given packet belongs.

[0117] As data units 205 are routed through different nodes in a network, nodes may sometimes drop, fail to send, or fail to receive certain data units 205, resulting in data units 205 not reaching their intended destination. The act of dropping a data unit 205 or failing to deliver a data unit 205 is generally referred to as "dropping" the data unit. Instances of dropping a data unit 205 (referred to herein as "dropping" or "packet loss") may occur for a variety of reasons, such as resource limitations, errors, or intentional policies. Different components of the device 200 may make the decision to drop a data unit 205 for a variety of reasons. For example, the traffic manager 240 may determine to drop a data unit 205 because, among other reasons, a buffer is overutilized, a queue exceeds a certain size, and / or the data unit 205 has a certain characteristic.

[0118] 5.0.CT and SAF Business Management

[0119] Figure 3A Example (relatively detailed) operations for processing and forwarding CT traffic and SAF traffic that share common resources of a network device / node in a communication network as described herein are shown. For example, a network device / node (e.g., Figure 2A 110, etc.) can be a single networked computing device (or network device), such as a router or switch, in which some or all of the processing components described herein are implemented in an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other integrated circuit. As another example, a network device / node may include: one or more memories storing instructions for implementing the various components described herein; one or more hardware processors configured to execute the instructions stored in the one or more memories; and various data repositories in the one or more memories for storing data structures utilized and manipulated by the various components.

[0120] like Figure 3A As shown, in response to receiving an input network / data packet (or unit) for forwarding to a next hop toward a destination (address), a network device / node or one or more packet processing components such as an ingress (packet) processor, an ingress (packet) arbitrator, a traffic manager, etc., may generate input packet control data (which may also be referred to as metadata for packet processing and forwarding) corresponding to the input network / data packet. Some or all of the input packet control data (in one or more packet data fields (e.g., in one or more header portions, one or more payload portions, etc.) may be extracted or obtained. Figure 3A denoted as “input control” in the ).

[0121] Some or all of the input packet control data of the input packet may be verified, validated or checked (in Figure 3A denoted as “input check” operation 302, such as error detection, checksum or CRC code verification, etc., to ensure that the input data packet is a valid data packet to be further processed or forwarded by the network device / node.

[0122] Based at least in part on the input packet control data and / or the results of the input inspection, the network device / node then determines (in Figure 3A 304) determines whether the input packet qualifies as a pass-through packet, which can be accelerated in subsequent packet queuing, dequeuing and forwarding operations.

[0123] In response to determining that an incoming packet is or qualifies as a CT packet, the network device / node or a service manager therein directs the CT packet to a dedicated CT packet control datapath. As used herein, a CT packet includes an actual CT packet as well as any other packet that qualifies as a CT packet. Alternatively, in response to determining that an incoming packet is not or does not qualify as a CT packet, the network device / node or a service manager therein directs the incoming (or SAF) packet to a dedicated SAF packet control datapath.

[0124] In the case where the input data packet is a CT data packet, the CT data packet control data path of the CT data packet may include performing enqueue and dequeue operations for the CT data packet. Figure 3A ” in the example 100) may generate a data write request (or req) to request that the buffer allocation logic 310 (e.g., a buffer manager, etc.) write some or all of the received data of the input (CT) packet (in the example 100). Figure 3A Incoming (CT) data packets (represented as "incoming data") are buffered in a data buffer (e.g., shared by an egress port, etc.) 312. Furthermore, (e.g., relatively lightweight, etc.) CT queuing data (including, but not limited to, CT (e.g., intra-packet, inter-packet, etc.) linking data 314) may be generated based at least in part on the incoming (CT) data packet control data and used to queue the incoming (CT) data packet into a dedicated CT packet (control data) queue established for the egress port or for a data buffer allocated for forwarding data packets through the egress port on the CT packet control data path.

[0125] In the case where the input data packet is a SAF data packet, the SAF data packet control data path for the SAF data packet may include performing enqueue and dequeue operations for the SAF data packet and (other or additional) SAF-specific or SAF-only operations. These SAF-specific or SAF-only operations are not performed on the CT data packet control data path for CT data packets, but are specific to or performed only on the SAF data packet control data path. For example, the SAF data packet enqueue operation (on Figure 3A 308). In addition, the SAF packet enqueue operation may generate a data write request (or req) to request that the buffer allocation logic 310 write some or all of the received data of the incoming (SAF) packet (in the SAF packet queue). Figure 3AIn some embodiments, the SAF queued data 316 may be buffered in a data buffer 312 (e.g., common to the egress ports). Furthermore, SAF queued data, including but not limited to SAF (e.g., intra-packet, inter-packet, etc.) chaining data 316, may be generated based at least in part on the inbound (SAF) packet control data and used to queue the inbound (SAF) packet into one or more SAF packet (control data) queues established for the same egress port or for the same data buffer allocated for forwarding packets through an egress port on the SAF packet control datapath. Compared to the CT queued data 314 for CT packets, SAF queued data, such as SAF chaining data 316, may be relatively heavy or relatively large in size.

[0126] As used herein, SAF-specific or SAF-only active queue management may refer to management operations performed by a network device / node, including but not limited to: proactively managing SAF queues before they become full, avoiding congestion, improving overall performance, etc. These operations may use (e.g., early detection, etc.) algorithms or logic to monitor the state of the queues (e.g., size, currently used capacity, etc.) and take actions to avoid overfilling or overflowing buffers / queues and reducing network congestion (e.g., packet dropping and marking, explicit congestion notification or ECN, avoiding long delays and / or excessive retransmissions, etc.).

[0127] The SAF-specific or SAF-only admission check described herein may refer to operations performed in conjunction with an incoming SAF packet before resources (such as buffer space, bandwidth, or processing power) are allocated for the SAF packet in a network device / node. These operations may be performed to determine whether the incoming SAF packet can be accepted without violating system constraints such as quality of service (QoS), available buffer space, network capacity, etc. (e.g., no available buffer / queue space, resulting in congestion, packet loss or excessive latency, fairness or QoS violations, etc.). In response to determining that the admission check for the incoming SAF packet fails, the network device / node may reject or drop the packet and / or send a signal back to the sender of the incoming SAF packet to indicate the admission check failure or packet rejection.

[0128] The network device / node or the traffic manager therein may include a scheduler 318 (maintained and operated in conjunction with a packet queue for an egress port or an egress packet queue) to manage how to process and forward packets when multiple input packets are waiting to be transmitted through the egress port. The scheduler 318 may be implemented or used to determine a specific time order (e.g., to prevent packet reordering issues, etc.) in which input packets from different queues are forwarded through the egress port, which helps optimize performance and fairness in packet forwarding or transmission operations.

[0129] In some operational scenarios, a single CT queue may be established by a service manager or a scheduler operating in conjunction with the service manager to schedule the transmission of incoming CT packets. In contrast, one or more SAF queues may be established by the service manager or scheduler to schedule the transmission of incoming SAF packets.

[0130] Once an input packet arrives or is received, the traffic manager directs the input packet or input packet control data to be processed in the CT packet control data path or the SAF packet control data path. The input packet control data or corresponding queuing / linking data or reference pointer can be enqueued into different queues (e.g., CT or SAF, different QoS SAFs, different priority SAFs, different traffic class / type SAFs, etc.) established by the traffic manager or scheduler 318. For each received input packet, whether it is an SAF or CT packet, the traffic manager or scheduler 318 can assign an arrival timestamp or arrival timing information that indicates when the input packet was received or queued / enqueued into a designated SAF queue of the CT queue or one or more SAF queues.

[0131] In the CT packet control data path, for a CT queue, the scheduler 318 may implement one or more CT dequeue algorithms to dequeue a CT queue element representing a corresponding CT packet. The CT queue element may represent or include a CT packet reference / pointer for accessing or retrieving a CT queue / link data portion specific to the CT packet (in Figure 3A denoted as "CT link" in the example. The portion of CT queued / linked data accessed or retrieved using a CT queue element may represent or include a CT packet dequeue, which may be used to cause or instruct the buffer allocation logic 310 to retrieve some or all of the input CT packet data (for a particular CT packet) stored in the data buffer 312 of the egress port. The input (or incoming) CT packet data for the particular CT packet may be used to generate output (or outgoing) CT packet data for the particular CT packet. The particular CT packet with the output CT packet data may be transmitted or forwarded by the network device / node via the egress port.

[0132] In the SAF packet control data path, for a SAF queue, the scheduler 318 may implement one or more SAF dequeue algorithms to dequeue a SAF queue element representing a corresponding SAF packet. The SAF queue element may represent or include a SAF queue / link data portion (in Figure 3A310. The SAF queue element 310 may be a SAF packet reference / pointer to a SAF packet (denoted as "SAF link" in the example). The portion of the SAF queued / linked data accessed or retrieved using the SAF queue element may represent or include a SAF packet dequeue request, which is used to cause or instruct the buffer allocation logic 310 to retrieve some or all of the input SAF packet data (for a particular SAF packet) stored in the data buffer 312 of the egress port. The input (or incoming) SAF packet data of the particular SAF packet may be used to generate the output (or outgoing) SAF packet data of the particular SAF packet. The particular SAF packet with the output SAF packet data may be transmitted or forwarded by the network device / node through the egress port.

[0133] The CT dequeue algorithm implemented by the scheduler may implement a first-come, first-served dequeue algorithm to dequeue an element (e.g., a queue head) or generate a CT packet dequeue request from the CT queue within a clock cycle. Alternatively or alternatively, the scheduler may implement an optimal scheduling algorithm to dequeue any element (e.g., a queue head) present in the CT queue without waiting.

[0134] The SAF dequeueing algorithm implemented by the scheduler may include one or more of the following: a first-come, first-served (FCFS) algorithm, in which SAF packets are forwarded in the order in which they arrive at (multiple) SAF queues; a weighted round-robin (WRR), in which a fixed round-robin time slot is assigned to each of some or all SAF queues, but larger time slots may be assigned to (multiple) SAF queues with higher priorities; a priority scheduling, in which packets in (multiple) high-priority SAF queues are given priority, and (multiple) low-priority SAF queues may be preempted; a loss-making round-robin (DRR), in which fairness is ensured between some or all SAF queues while still maintaining priority for time-sensitive services; and the like.

[0135] When shared resources such as the same egress port and the same data buffer 312 of the egress are processing and forwarding CT and SAF network / data packets, CT and SAF packet dequeue requests from the CT and SAF packet control datapath will be processed with Figure 3A The dequeued request (path) merger or DRPM 320 merger.

[0136] Figure 3BAn example packet control data path merging operation performed by the dequeue request (path) merger 320 is shown. In some operational scenarios, to support relatively low latency arbitration (e.g., one to three clock cycle latency, etc.), the merger can establish, maintain, or use a store-and-forward request FIFO (SRF) and a pass-through request FIFO (CRF). The merger 320 can be implemented with relatively simple arbitration logic to select the earliest arriving SRF and CRF headers (e.g., current or upcoming) from the SRF and CRF FIFOs, respectively.

[0137] The packet dequeue request (path) merger 320 can be implemented to manage the contention between SAF and CT packet dequeue requests so that the egress bandwidth (of the egress port) maintains a specific bandwidth distribution defined or implemented by the scheduler, the latency for forwarding through packets is minimized, and the per-port inter-cell jitter is minimized, thereby avoiding or reducing the possibility or occurrence of packet corruption (e.g., underrun, etc.).

[0138] like Figure 3B As shown, each CRF or SRF entry in the CRF or SRF FIFO maintained by the DRPM 320 may include (e.g., include only, include at least, etc.): a buffer address to be used by the buffer allocation logic 310 to access the packet data maintained in the data buffer 312 of the egress port for the corresponding CT or SAF network / data packet (or its cell); and a (DRPM) arrival timestamp captured or assigned by the scheduler when a dequeue request for the CT or SAF packet (or its cell) leaves the CT or SAF queue maintained by the scheduler 318 and enters the CRF or SRF FIFO maintained by the DRPM 320.

[0139] Although the same or common scheduler and merger are used for both the CT and SAF packet control data paths (e.g., along with buffer allocation logic and data buffers, etc.), the CT packet control path incurs lower latency from the scheduler 318 to the DRPM 320. This is at least in part due to the use of a relatively simple CT connection structure compared to the SAF connection structure.

[0140] In some operating scenarios, on average (e.g., at most, at least, etc.), one packet dequeue request arrives per clock cycle, e.g., from each or both of the CT and SAF queues maintained by scheduler 318 to DRPM 320. The CRF and SRF FIFOs maintained by DRPM 320 may be specifically sized or optimized to absorb intermittent bursts due at least in part to the difference between SAF and CT packet control data path latencies.

[0141] The DRPM 320 may implement an earliest-first arbiter to control departures from the CRF and SRF FIFOs maintained by the DRPM 320. The DRPM arrival timestamps of the CRF and SRF headers of the SRF and SRF FIFOs (e.g., timestamps assigned by the scheduler when transmitting CT and SAF dequeue requests to the DRPM 320; CT and SAF entries corresponding to the CT and SAF dequeue requests from the scheduler (enqueued by the DRPM 320 at the tail of the CRF and SRF FIFOs and maintained therein) are compared.

[0142] If both CRF and SRF headers are present in the CRF and SRF FIFOs, the earliest of the CRF and SRF headers indicated by the respective DRPM arrival timestamps is dequeued or selected by DRPM 320 in a common sequence of packet dequeue requests sent or provided by DRPM 320 to the buffer allocation logic.

[0143] If only one of the CRF and SRF FIFOs has data or entries, its header is dequeued and included in the common packet dequeue request sequence.

[0144] In some operating scenarios, DRPM 320 enforces (eg, I / O resources, timing controls, etc.) constraints whereby at most one dequeue request may leave DRPM 320 to buffer allocation 310 per (eg, read, etc.) clock cycle.

[0145] In some operational scenarios, a dequeue request leaves if either the SRF or CRF FIFO (or both) has data or entries.

[0146] After a dequeue request corresponding to a CRF or SRF header (entry) in a CRF or SRF FIFO leaves the DRPM 320 to the buffer allocation logic 310 or is dispatched by the DRPM 320 to the buffer allocation logic 310, the CT and SAF packet data control paths merge into the same or common packet control path or subpath, in which the same or common packet processing operations, such as ( Figure 3B ) dequeue control (ctrl) processing 322.

[0147] These common packet processing operations may include generating output packet control data, fetching input packet data, generating output packet data, forwarding output network / data packets corresponding to input network / data packets, etc. For example, based on the common sequence of the merged dequeue requests, the buffer allocation logic 310 may issue a read request ( Figure 3A) to access or retrieve the stored packet data referenced by the address in the dequeued header entry from the CRF or SRF FIFO. Exemplary outgoing packet control data may include, but is not necessarily limited to, control data for performing packet header modification (e.g., updating the frame check sequence or FCS, updating the VLAN tag, etc.), address resolution (to determine the next hop for forwarding), encapsulation or decapsulation, etc.

[0148] Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 3A and Figure 3B Representative examples of many possible alternative arrangements of devices configured to provide the functionality described herein are shown. Other arrangements may include fewer, additional, or different components, and the division of work between components may vary depending on the arrangement. Furthermore, in embodiments, the techniques described herein may be utilized in various computing contexts other than within network 100 or network device 200.

[0149] Furthermore, the figures herein illustrate only a few of the various arrangements of memory that may be used to implement the described buffering techniques. Other arrangements may include fewer or additional elements in varying arrangements.

[0150] 6.0 Example Embodiments

[0151] Various example method flows for implementing various features of the systems and system components described herein are described in this section. The example method flows are non-exhaustive. Alternative method flows and flows for implementing other features will be apparent from this disclosure.

[0152] The various elements of the process flows described below can be executed in various systems, including in one or more computing or networking devices that utilize some or all of the load balancing or traffic distribution mechanisms described herein. In an embodiment, each process described in conjunction with the functional blocks described below can be implemented using one or more integrated circuits, logic components, computer programs, other software elements, and / or digital logic in any of general-purpose or special-purpose computers while performing data retrieval, transformation, and storage operations involving interaction with and transformation of the physical state of the computer's memory.

[0153] Figure 4An example process flow according to an embodiment is shown. Various elements of the process described below can be performed by one or more network devices (or processing engines therein) implemented using one or more computing devices. In block 402, a network device as described herein, or a traffic manager therein, allocates a common packet data buffer for an egress port to store incoming packet data, including both CT packets and SAF packets. The CT packets and SAF packets are to be forwarded out of the same egress port.

[0154] In block 404, the traffic manager directs the SAF packet control data received from the SAF packet to the control data path defined by the first plurality of processing engines. The SAF control data will arrive at the scheduling logic engine with a first latency after being processed by the first plurality of processing engines.

[0155] In block 406, the traffic manager directs the CT packet control data of the CT packet upon receipt thereof onto the second control data path. After being processed in the second control path by the second plurality of processing engines that bypass at least one or more processing engines of the first plurality of processing engines, the CT control data arrives at the scheduling logic engine at a second latency that is less than the first latency.

[0156] In block 408, the traffic manager generates a CT packet dequeue request for the CT packet using the CT packet control data and generates a SAF dequeue request for the SAF packet using the SAF packet control data.

[0157] In block 410, the traffic manager merges the CT packet dequeue requests and the SAF dequeue requests into a merged dequeue request sequence.

[0158] In block 412, the traffic manager retrieves packet data from the common packet data buffer based on the merged sequence of dequeue requests.

[0159] In an embodiment, the control data path includes, and the second control data path does not include, performing one or more of: active queue management operations with respect to one or more SAF queues, or SAF admission checking operations.

[0160] In an embodiment, the traffic manager further performs: in response to receiving the incoming data packet, determining whether the incoming data packet qualifies as a CT data packet.

[0161] In an embodiment, the scheduling logic engine assigns a first arrival timestamp of the CT packet control data of the CT packet to the CT packet, and assigns a second arrival timestamp of the SAF packet control data of the SAF packet to the SAF packet.

[0162] In an embodiment, the scheduling logic engine compares a first arrival timestamp of a CT packet control data portion of a CT packet enqueued in a single CT queue with a second arrival timestamp of a selected SAF packet control data portion of a selected SAF packet enqueued in one or more SAF queues to select one of a CT dequeue request or a SAF dequeue request to generate a read request during a given read clock cycle.

[0163] In an embodiment, the merged sequence of dequeue requests results in a single data read request to the common packet buffer for each data unit in a CT packet or a SAF packet to be forwarded out through the egress port.

[0164] In an embodiment, the scheduling logic engine and the merging logic engine are implemented with a traffic manager of a networked device.

[0165] In an embodiment, a computing device such as a switch, a router, a line card in a chassis, a network device, etc. is configured to perform any of the aforementioned methods. In an embodiment, an apparatus includes a processor and is configured to perform any of the aforementioned methods. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause the performance of any of the aforementioned methods.

[0166] In an embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause performance of any of the aforementioned methods.

[0167] Note that although separate embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form further embodiments.

[0168] 7.0. Extensions and Alternatives

[0169] As used herein, the terms "first," "second," "certain," and "particular" are used as naming conventions to distinguish queries, plans, representations, steps, objects, devices, or other items from one another so that these items can be referenced after they are introduced. Unless otherwise specified herein, the use of these terms does not imply an ordering, timing, or any other characteristic of the referenced items.

[0170] In the accompanying drawings, various components are depicted as being communicatively coupled to various other components via arrows. These arrows merely illustrate certain examples of information flow between components. The direction of arrows or the lack of arrow lines between certain components should not be interpreted as indicating the presence or absence of communication between certain components themselves. In fact, each component can feature an appropriate communication interface through which it can be communicatively coupled to other components as needed to implement any of the functionality described herein.

[0171] In the foregoing description, embodiments of the inventive subject matter have been described with reference to a number of specific details that may vary from implementation to implementation. Therefore, the sole and exclusive indicator of the inventive subject matter, and what the applicant considers to be the inventive subject matter, is the set of claims issued from this application, in the specific form in which such claims are issued, including any subsequent amendments. In this regard, although specific claim dependencies are set forth in the claims of this application, it should be noted that features of the dependent claims of this application may be appropriately combined with features of other dependent claims and with features of the independent claims of this application, and not merely in accordance with the specific dependent claims recited in the claim group. Furthermore, although individual embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form additional embodiments.

[0172] Any definitions expressly set forth herein for terms contained in these claims shall govern the meaning of such terms as used in the claims. Accordingly, no limitation, element, property, feature, advantage, or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for processing direct CT and store-and-forward (SAF) services, the method comprising: allocating a common packet data buffer for an egress port to store incoming packet data including both CT packets and SAF packets, wherein the CT packets and the SAF packets are to be forwarded out through the same egress port; directing SAF packet control data of the SAF packet upon receipt onto a control data path defined by a first plurality of processing engines, the SAF control data arriving at a scheduling logic engine with a first latency after being processed by the first plurality of processing engines; directing CT packet control data of the CT packet onto a second control data path upon receipt, wherein the CT control data is processed by a second plurality of processing engines in the second control path and arrives at the scheduling logic engine with a second latency, the second latency being less than the first latency, the second plurality of processing engines bypassing at least one or more processing engines among the first plurality of processing engines; generating a CT packet dequeue request for the CT packet using the CT packet control data, and generating a SAF dequeue request for the SAF packet using the SAF packet control data; Merging the CT packet dequeue request and the SAF dequeue request into a merged dequeue request sequence; and Packet data is retrieved from the common packet data buffer based on the merged sequence of dequeue requests.

2. The method of claim 1 , wherein the control data path comprises performing one or more of the following and the second control data path does not comprise performing one or more of the following: active queue management operations related to the one or more SAF queues, or SAF admission checking operations.

3. The method according to claim 1, further comprising: In response to receiving an incoming data packet, a determination is made as to whether the incoming data packet qualifies as a CT data packet.

4. The method of claim 1 , wherein the scheduling logic engine assigns a first arrival timestamp of the CT packet control data of the CT packet to the CT packet, and assigns a second arrival timestamp of the SAF packet control data of the SAF packet to the SAF packet.

5. The method of claim 1 , wherein the scheduling logic engine compares a first arrival timestamp of a CT packet control data portion of a CT packet enqueued in the single CT queue with a second arrival timestamp of a selected SAF packet control data portion of a selected SAF packet enqueued in the one or more SAF queues, and selects one of a CT dequeue request or a SAF dequeue request in response to the comparison, and generates a read request during a given read clock cycle.

6. The method of claim 1, wherein the merged sequence of dequeue requests results in a single data read request to the common packet buffer for each data unit in a CT packet or a SAF packet to be forwarded out through the egress port.

7. The method of claim 1 , wherein the traffic manager comprises the scheduling logic engine and a merging logic engine, the scheduling logic engine being configured to perform enqueue operations and dequeue operations on both CT incoming packets and SAF incoming packets for forwarding, and the merging logic engine being configured to receive CT dequeue requests and SAF dequeue requests from the scheduling logic engine and merge the CT dequeue requests and SAF dequeue requests into the common dequeue request sequence.

8. A network switching system comprising: a buffer manager configured to allocate a common packet data buffer for an egress port to store incoming packet data including both CT packets and SAF packets, wherein the CT packets and the SAF packets are to be forwarded out through the same egress port, and retrieve packet data from the common packet data buffer based on a merged dequeue request sequence; an ingress packet processor configured to, upon receipt, direct SAF packet control data of the SAF packet onto a control data path defined by a first plurality of processing engines, the SAF control data arriving at the scheduling logic engine with a first latency after being processed by the first plurality of processing engines; wherein the ingress packet processor is further configured to direct the CT packet control data of the CT packet onto a second control data path upon receipt, the CT control data being processed by a second plurality of processing engines in the second control path and arriving at the scheduling logic engine with a second latency, the second latency being less than the first latency, the second plurality of processing engines bypassing at least one or more processing engines among the first plurality of processing engines; a scheduling logic engine configured to generate a CT packet dequeue request for the CT packet using the CT packet control data, and to generate a SAF dequeue request for the SAF packet using the SAF packet control data; and The merging logic engine is configured to merge the CT packet dequeue request and the SAF dequeue request into the merged dequeue request sequence.

9. The system of claim 8, wherein the ingress packet processor is configured to perform one or more of: active queue management operations related to the one or more SAF queues, or SAF admission checking operations, the operations being included in the data path but not included in the second data path.

10. The system of claim 8, wherein the instructions, when executed by the one or more computing devices, further cause: in response to receiving an incoming data packet, determining whether the incoming data packet qualifies as a CT data packet.

11. The system of claim 8, wherein the scheduling logic engine is configured to assign a first arrival timestamp of the CT packet control data of the CT packet to the CT packet, and assign a second arrival timestamp of the SAF packet control data of the SAF packet to the SAF packet.

12. The system of claim 8 , wherein the scheduling logic engine is configured to compare a first arrival timestamp of a CT packet control data portion of a CT packet enqueued in the single CT queue with a second arrival timestamp of a selected SAF packet control data portion of a selected SAF packet enqueued in the one or more SAF queues, and in response to the comparison, select one of a CT dequeue request or a SAF dequeue request, and generate a read request based on the selected CT dequeue request or the selected SAF dequeue request during a given read clock cycle.

13. The system of claim 8, wherein the buffer manager is configured to process the merged sequence of dequeue requests, the merged sequence of dequeue requests resulting in a single data read request to the common packet buffer, each data unit in a CT packet or a SAF packet to be forwarded out through the egress port.

14. The system of claim 8 , wherein the system further comprises a traffic manager, the traffic manager comprising the scheduling logic engine and the merging logic engine, the scheduling logic engine being configured to perform enqueue operations and dequeue operations on both CT incoming packets and SAF incoming packets for forwarding, and the merging logic engine being configured to receive CT dequeue requests and SAF dequeue requests from the scheduling logic engine and merge the CT dequeue requests and SAF dequeue requests into the common dequeue request sequence.

Images