Automated flow management

Abstract

Packet switching operations in network devices are managed based on detection of excess rate traffic flows. A network device receives data units, determines the traffic flow to which the data units belong, and updates flow tracking information for the flow. The network device uses the tracking information to determine when the rate at which the network device is receiving data belonging to the flow exceeds an excess rate threshold and is therefore an excess rate flow. The network device can enable one or more excess rate policies for excess rate traffic flows. Such policies can include any number of features affecting how the device handles data units belonging to the flow, such as excess rate notification, differential discard, differential congestion notification, and reprioritization. Memory and other resource optimizations for such flow tracking and management are also described.

Description

TECHNICAL FIELD

[0001] Embodiments relate generally to computer networks, and more particularly, to techniques for automatically identifying and / or managing network traffic flows. BACKGROUND

[0002] The approaches described in this section are approaches that can be taken, but need not be taken, to achieve the application. Accordingly, nothing that is described in this section should be deemed critical or essential to the application unless explicitly so identified.

[0003] A computer network is a collection of computing components interconnected by communication links. Each computing component can be a separate computing device, such as, but not limited to, a hub, switch, bridge, router, server, gateway, or personal computer, or a component thereof. Each computing component, or "network device," is considered a node in the network. A communication link is a mechanism that connects at least two nodes such that each node can send data to and receive data from the other node. Such data can be transmitted in the form of signals over a transmission medium, such as, but not limited to, a cable, optical fiber, or wireless medium.

[0004] Data structures and transmissions between nodes are governed by a number of different protocols. There can be multiple protocol layers, often starting with the lowest layer, such as a "physical" layer that manages the transmission and reception of raw bit streams as signals over the transmission medium. Each layer defines a unit of data (a protocol data unit, or "PDU"), where multiple data units on one layer are often combined into a single data unit in another layer. Additional examples of layers can include, for example, a data link layer, where bits defined by the physical layer are combined to form frames or cells; a network layer, where frames or cells defined by the data link layer are combined to form packets; and a transport layer, where packets defined by the network layer are combined to form TCP segments or UDP datagrams. The Open Systems Interconnection communications model describes these and other layers of communication. However, other models that define layered information in other ways can also be used. The Internet Protocol Suite, or "TCP / IP stack," is one example of a common set of protocol groups that can be used together over multiple layers to deliver information. However, the techniques described herein can be applied to other protocols beyond the TCP / IP stack.

[0005] A given node in a network does not necessarily have a link to every other node in the network, especially in more complex networks. For example, in a wired network, each node can have only a limited number of physical ports into which cables can be plugged to create a link. Certain "terminal" nodes, often servers or end-user devices, can have only one or a few ports. Other nodes, such as switches, hubs, or routers, can have more ports, and are often used to relay information between terminal nodes. The arrangement of nodes and links in a network is referred to as the topology of the network, and is often visualized as a network graph or tree.

[0006] A given node in a network can communicate with another node in the network by sending data units along one or more different paths in the network that lead to the other node, each path including any number of intermediate nodes. Data transfer across a computing network often involves sending data units such as packets, cells, or frames along paths through intermediate network devices such as switches or routers, which direct or redirect each data unit to a corresponding destination.

[0007] When a data unit is passing through an intermediate network device - a period of time conceptually defined as an "access" or "hop" - the device can perform any of a variety of actions or processing steps on the data unit. The exact set of actions taken will depend on various characteristics of the data unit, such as metadata found in a header of the data unit, and in many cases on the context or state of the network device. For example, address information specified by or otherwise associated with a data unit, such as a source address, a destination address, or path information, is often used to determine how to process the data unit (e.g., what actions to take on the data unit). For example, an Internet Protocol ("IP") data packet can include a destination IP address field within a header of the IP data packet, based on which the network device can determine one or more other network devices to which to forward the IP data packet among a plurality of possible other network devices. The logic within a network device that controls the particular set of actions performed with respect to a given data unit is often referred to as "packet switching" logic.

[0008] A traffic flow is a set of data units having some common property or properties. These properties can indicate to the packet-switching logic that the data units have similar functionality or purpose and thus should be handled in a similar manner. For example, in one embodiment, a traffic flow is a sequence of data units sent from the same source device to the same destination device. Flows can or can not be further defined by the context in which the data units are sent (e.g., the particular protocol used, the class of service, etc.). In some protocols, it can be intended that flows be sent in a particular order, and thus network devices are typically configured to send all data units within a given flow along the same path to ensure that the flow is received in that order.

[0009] While in some embodiments, the source of a traffic flow can mark each data unit in the set as a member of the flow (e.g., using a tag, label, or other suitable identifier within the data unit), in other embodiments, intermediate network devices must determine for themselves which data units they receive that constitute a traffic flow. In some embodiments, data units are identified as belonging to a flow by deriving an identifier from a header field in the data unit. For example, a "five-tuple" combination of source address, source port, destination address, destination port, and protocol is often used to derive an identifier for a traffic flow, although any other suitable combination of elements within a data unit can also be used.

[0010] Network devices can include any number of internal hardware and / or software components that are configured to handle the movement of data units between processing components within the device and ultimately out of the device. It is desirable for these components to quickly determine where to send and / or store data for processing and, once determined, to conveniently send and / or store that data to the appropriate destination. In addition, it is desirable for these components to process network traffic in a manner that will best utilize the available network resources throughout the network in which the device is located. BRIEF DESCRIPTION OF DRAWINGS

[0011] The subject matter of the present application is illustrated in the accompanying drawings, in which:

[0012] Figure 1 A method flow for processing high-rate traffic flows in a network device is shown;

[0013] Figure 2 An example method for managing flow tracking containers is shown;

[0014] Figure 3 A schematic diagram of various components of an example system configured for traffic tracking and management in accordance with the techniques described herein;

[0015] Figure 4 An example flow tracker component is shown;

[0016] Figure 5 is a schematic diagram of various aspects of an example network system in which the technology described herein can be practiced;

[0017] Figure 6 is a schematic diagram of various aspects of an example network device in which the technology described herein can be practiced;

[0018] Figure 7 shows an example of a network device having multiple packet processing pipelines; and

[0019] Figure 8 is a block diagram showing an example computer system that can be used to implement the technology described above. DETAILED DESCRIPTION

[0020] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the inventive subject matter. It will be apparent, however, that the inventive subject matter can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the inventive subject matter.

[0021] 1.0. OVERVIEW

[0022] Methods, techniques, and mechanisms are disclosed for improving the performance of a switch or other network device by detecting and taking action on over-rate traffic flows within the device. When a network device receives a data unit, it uses information within the data unit to determine the traffic flow to which the data unit belongs. Based on this information, the network device updates flow tracking information for the traffic flow, such as a rate counter or a log. The network device uses the tracking information to determine when the rate at which the network device is receiving and / or transmitting data belonging to a traffic flow exceeds an over-rate threshold. When the rate exceeds the threshold, the traffic flow is considered an over-rate flow. When a traffic flow becomes an over-rate flow, the network device can enable one or more over-rate policies for the traffic flow.

[0023] Generally, the "rate" of a flow is the amount of data determined to belong to the flow that a device or other measuring entity receives (or transmits) in a period of time. In some embodiments, the amount of data can be expressed in terms of the number of data units (e.g., packets) or sub-units (e.g., cells) that are received. In other embodiments where the size of data units can vary significantly, the amount of data can optionally (but not necessarily) be expressed in terms of the total size of data units belonging to the flow, such as the total number of bytes received, the total number of bits received, the total number of bytes allocated for storing data units, and so on. The period of time over which the rate is measured and updated can likewise vary from embodiment to embodiment (e.g., 1000 data units per second, 700 bytes per millisecond, etc.).

[0024] When an over-speed policy is enabled for a traffic flow that has become an over-speed flow, the packet switching logic (e.g., forwarding logic, traffic manager, packet processor, etc.) of the network device processes data units identified as belonging to the traffic flow in a manner different from normal processing. For example, the over-speed policy can include an over-speed notification feature that causes the network device to notify a designated collection entity that the flow has become an over-speed flow. Such a policy can cause the device to clone data units belonging to the over-speed flow and forward the cloned data units to a collector, for example, that would not normally occur for that data unit.

[0025] As another example, the over-speed policy can include a reprioritization feature that causes the device to reprioritize data units belonging to the over-speed flow. Reprioritization can involve sending the data units to a designated queue (e.g., an egress queue, an ingress queue, etc.) that is different from the queue to which the data units would normally be sent when the traffic flow is not an over-speed flow, for example. In this example and the over-speed notification example, the over-speed policy can cause the device to process all data units belonging to the over-speed flow in the same manner, or can only process a particular subset of the data units (e.g., a random sample, every other one, etc.) in this manner.

[0026] As another example, the over-speed policy can include a differentiated drop rate feature that causes data units belonging to the over-speed flow to be dropped at a higher rate than when the traffic flow is not an over-speed flow. For example, a traffic manager can be configured to drop one of every ten data units belonging to a queue once the queue reaches a first size under normal circumstances, while the over-speed policy can cause the traffic manager to drop one of every five data units upon reaching the first size, and / or cause the traffic manager to start dropping data units once the queue reaches a second size instead of the first size.

[0027] In one embodiment, a weighted random early detection ("WRED") curve can be utilized to determine a drop probability for dropping data units upon enqueuing. The over-speed policy can cause the network device to use a different curve for data units belonging to the over-speed flow. If the flow is determined to be an over-speed flow, a more aggressive curve can be selected. Such a curve will drop data units more aggressively, thereby not having a severe impact on compliant flows, or allowing a less aggressive curve to be used for compliant flows.

[0028] As yet another example, an over-rate policy can include a differentiated congestion notification feature that causes the notification logic to begin notifying entities of congestion earlier or later than would otherwise be implemented. For example, a network device can implement an explicit congestion notification according to which the network device marks data units in their headers with an identifier to signal to the recipient of the data unit that congestion or impending congestion. For data units belonging to over-rate flows that have an over-rate policy enabled, the network device can begin such marking at a lower or higher congestion threshold. In one embodiment, an ECN curve can be used to determine the ECN marking probability (e.g., how likely it is that a data unit will be marked for ECN purposes). The over-rate policy can cause the device to use a different curve for data units belonging to over-rate flows that have an over-rate policy enabled in a manner similar to a WRED curve.

[0029] In some embodiments, the memory and / or other resource cost of tracking the actual traffic rate of all possible traffic flows can be undesirable, even prohibitive, due to the large number of possible traffic flows. For these and other reasons, in one embodiment, rather than continuously calculating the actual rate at which the device receives and / or transmits data units belonging to a traffic flow, a proxy is used to indicate the rate of the flow using a corresponding rate counter. Each tracked flow has a rate counter that is incremented in response to receiving (or transmitting) a data unit belonging to the flow, and then decremented periodically or at other intervals using a background process. The amount and / or frequency of the decrementing of the counter can be based on a threshold rate and / or an associated target rate. Whenever the value of a traffic flow's rate counter exceeds some threshold counter value, the rate of the traffic flow is determined to be above the threshold rate (and thus the flow is referred to as an over-rate flow).

[0030] In one embodiment, the rate or traffic flow can be checked each time a data unit belonging to the flow arrives at the device. However, in other embodiments, the actual testing of whether a flow exceeds its over-rate threshold can only be repeated at various intervals (e.g., in a background process). The results of the testing are cached in a state indicator accessible to the over-rate policy management logic of the device, and the device need not repeat the testing each time a data unit is received. Thus, the device continues to behave as an over-rate (or under-rate) even if the results of the testing would have been different during this time, before the testing is performed again.

[0031] Further, in some embodiments, tracking information is stored for only a subset of the possible traffic flows, such that there need not be a rate counter associated with every traffic flow for which the network device has received data. The traffic flows included in the subset can be determined, for example, based on which traffic flows have received the greatest amount of data over a given time period, which traffic flows are currently active (as opposed to idle), and / or for which traffic flows the network device has recently received data units.

[0032] According to one embodiment, tracking information for a traffic flow, such as a traffic rate counter, is stored in a flow tracking container. One or more memories, collectively referred to as flow tracking memory, can be dedicated to storing flow tracking containers. A limited number of flow tracking containers can be stored in the flow tracking memory. Thus, tracking containers for low-rate and / or idle traffic flows can be reassigned from time to time for different traffic flows.

[0033] In one embodiment, rather than searching the entire flow tracking memory for a tracking container belonging to a certain traffic flow, the memory can be partitioned into indexed rows. One or more hash values output by one or more hash functions of an identifier of the traffic flow can be used to locate one or more rows in which a flow tracking container for the traffic flow can be located. The set of row(s) located by the one or more hash values is considered to be the storage space allocated to the traffic flow. If a flow tracking container for the traffic flow is not found in its storage space, a new flow tracking container for the traffic flow can be created within that region, or an existing flow tracking container can be reassigned to the traffic flow if the flow tracking container qualifies for reassignment (e.g., is associated with a low-rate and / or idle traffic flow).

[0034] In other aspects, the inventive subject matter includes computer devices and computer readable media configured to perform the foregoing techniques.

[0035] 2.0. FUNCTIONAL OVERVIEW

[0036] This section describes example method flows for implementing various features of the systems and system components described herein. The example method flows are not exhaustive. Alternative method flows and flows for implementing other features will be apparent from this disclosure.

[0037] Various elements of the process flows described below can be performed in various systems, including the switches described in other sections and / or in other switching devices. In one embodiment, each of the processes described in connection with the functional blocks described below can be implemented using one or more integrated circuits, logic components, computer programs, other software elements, and / or digital logic in any of a general purpose computer or a special purpose computer, while performing data retrieval, transformation, and storage operations involving interaction with and transformation of the physical state of the memory of the computer.

[0038] 2.1. Managing Excessive-Rate Traffic Flows

[0039] Figure 1 A method flow 100 for handling excessive-rate traffic flows in a network device is shown, according to one embodiment. The network device can be a switch or any other network device, including the example devices described elsewhere herein.

[0040] Block 110 includes receiving a data unit. According to embodiments, the data unit can be a packet, cell, frame, message, or any other suitable data unit described herein. The data unit is received through a communication interface of the device, which can be directly connected to the sender ("source") of the data unit, or indirectly connected through one or more intermediate devices in a network.

[0041] Block 120 includes identifying the traffic flow to which the data unit belongs. According to embodiments, the identification process can include various sub-steps. For example, in one embodiment, flow tracking logic within the network device can perform the identification by extracting certain data from the data unit, such as certain header fields of the data unit. This data can be used as a flow identifier. For example, these fields can be, but are not limited to, source address, destination address, source port, destination port, and protocol type. In another embodiment, the resulting values of this extraction can be input into a function or functional block to yield a flow identifier. For example, the function can be a hash function, and the resulting hash value output from the hash function can be used as the flow identifier. In other embodiments, other functions and steps can also be used to yield a flow identifier for the data unit. For example, a user can specify a flow identifier directly in the data unit, and this identifier can be directly parsed from the data unit.

[0042] Block 130 includes updating tracking information to reflect the receipt of the data unit. This can include, for example, flow tracking logic within the network device incrementing a rate counter associated with the flow. In embodiments that track the rate of a flow according to the number of data units received for the flow, or in embodiments that track the rate of a flow in, for example, bytes, the counter can be incremented by one. In another embodiment, block 130 can also or instead include the flow tracking logic updating a log to show that a data unit belonging to the traffic flow was received.

[0043] Block 140 includes determining whether the rate at which the network device receives data belonging to the particular traffic flow exceeds a threshold rate. For example, the threshold rate can be set by a network administrator based on various factors. In one embodiment, the threshold rate is selected based at least in part on a desired target rate for the particular traffic flow. For example, it can be desirable for the particular traffic flow to have a target rate of 10 Gbps. The threshold rate can be set automatically to twice that target rate (20 Gbps), three times that target rate, or some other function of that target rate. In other embodiments, the threshold rate can be set independently of any target rate.

[0044] Likewise, the particular sub-steps involved in the determination of block 140 can vary from embodiment to embodiment. For example, in some embodiments, the over-rate flow monitoring logic within the network device can be configured to actually calculate the flow rate of the traffic flow and compare that rate to the threshold rate. For example, the logic can use a log to count the amount of data received in a recent period of time and divide that amount by the length of the period of time to calculate the flow rate.

[0045] In other embodiments, the logic can use a periodically decremented rate counter as a proxy indicator of whether the current flow rate exceeds the threshold rate, rather than actually calculating the flow rate. Generally, the rate counter for a traffic flow is incremented each time a data unit of the traffic flow arrives (e.g., by the number of bytes in the data unit). The threshold is applied to the byte counter to determine whether the traffic flow is over-rate. Then, a background process iterates through each counter once per update period, reducing the counter by an amount that reflects the reduction needed to reach the threshold rate and / or target rate.

[0046] Under this mechanism, the rate counter that is incremented each time block 130 is executed is periodically decremented by an amount in block 150, described below. The over-rate flow monitoring logic within the network device interprets a rate counter value that is higher than the threshold counter value as an indication that the current flow rate exceeds the threshold rate.

[0047] For example, in one embodiment, the amount removed from the rate counter each time block 150 is executed is a reduction amount that can be selected based on the threshold rate. For example, if the flow rate is not greater than the threshold rate, the removal amount can be the maximum amount of data that can be added to the rate counter since the last time the rate counter was reduced. Thus, when the rate counter is higher than some amount (e.g., the amount that would normally be removed the next time the rate counter is decremented), it can be said that the flow rate has recently exceeded the threshold rate and is therefore over-rate. Note that this technique only approximates any given actual flow rate, and the threshold can be selected to account for temporary fluctuations due to isolated bursts of traffic and other factors.

[0048] In another embodiment, the reduction amount can be selected based on the target rate, while the threshold counter value reflects the threshold rate. In other embodiments, the reduction amount and / or threshold counter value can be selected based on a function(s) of both the target rate and the threshold rate, or can even be set independently without regard to any desired target rate or threshold rate. In the latter case, the threshold rate at which the over-speed flow monitoring mechanism is configured to monitor can not necessarily have been explicitly specified, but is still a function of the selected reduction amount and threshold counter value.

[0049] In some embodiments, the threshold rate, threshold counter value, reduction amount, and / or target rate can vary according to various properties of the respective traffic flows. For example, there can be different reduction amounts that apply to traffic flows that are handling data units destined for or coming from particular ports, addresses, protocols, etc. In one embodiment, traffic flows can be classified according to their characteristics as belonging to different profiles, and the threshold rate, counter threshold, amount to decrement per period, etc. can vary based on the profile. Or, in other embodiments, there can be a single global threshold rate.

[0050] For example, bandwidth for control traffic typically has a relatively low rate compared to other traffic types. For example, one can expect the peak flow rate for control traffic to be 1 Gbps under normal circumstances, while the peak flow rate for other types of traffic can be significantly higher (e.g., 100 Gbps). Thus, an over-speed rate threshold for control traffic can be used to flag any flows observed above, e.g., 5 Gbps, which is 5 times the expected rate, while the threshold for other traffic flows can be much greater.

[0051] In some embodiments, block 140 can be performed in response to block 130. For example, in the former case, the over-speed flow monitoring logic within the network device can perform block 140 each time a data unit is received.

[0052] In other embodiments, to reduce the number of computations required each time a data unit is received, block 140 is performed asynchronously with respect to block 130. That is, the over-speed flow monitoring logic can be implemented as a background process that performs block 140 only at specified times. Additional block 145 will also be performed at these times, which includes updating the over-speed status indicator to reflect the determination of block 140. That is, when it is determined that the rate at which the network device is receiving data belonging to a particular traffic flow exceeds the threshold rate, the status indicator will be set to "over-speed," otherwise it will be set to "normal." The processing of data units received between these two specified times will be based on the status that the over-speed status indicator is currently indicating, even if during this time the actual traffic can have fallen below the threshold.

[0053] Block 150 is also optional according to embodiments, including decrementing the rate counter if the counter refresh period has elapsed. Block 150 will be performed only for embodiments in which the rate counter is used as a proxy indicator of whether the flow rate exceeds a relevant threshold, and is described in more detail above. In one embodiment, the over- rate flow monitoring logic can perform block 150 asynchronously with respect to the receipt of data units, such as part of a background update process. In another embodiment, the over-rate flow monitoring logic within the network device can test whether to perform block 150 in response to block 110, such that the elapse of the refresh period is checked each time a new data unit is received.

[0054] Block 160 includes determining whether the over-rate policy should be used for the flow to which the data unit received in block 110 belongs. According to embodiments, block 160 can involve several determinations. First, block 160 can include determining whether the over-rate policy is enabled for the traffic flow. The over-rate policy can be enabled on a per-flow basis and / or on other contexts such as per-port. In some embodiments, the over-rate policy can be enabled for all traffic flows and contexts, and the determination can be skipped.

[0055] Second, block 160 can include determining whether the flow is currently considered an over-rate flow. In embodiments in which block 140 is performed each time a data unit is processed, this determination is effectively synonymous with block 140. In other embodiments, block 160 can include accessing the state indicator for the flow that was updated in the last iteration of block 145, and determining to use the over-rate policy if the state indicator indicates that the flow is currently an over-rate flow.

[0056] If the over-rate policy is not to be used for the traffic flow, flow 100 proceeds to block 170. Block 170 includes the device processing the data unit according to its normal packet switching logic. For example, the forwarding logic of the device can identify the destination of the data unit and forward the data unit to a queue associated with an egress port associated with the destination, from which downstream packet switching logic can continue to process the data unit normally.

[0057] If the over-rate policy is to be used for the traffic flow, flow 100 proceeds to block 175. Block 175 includes marking the data unit as belonging to an over-rate flow. The marking can include, for example, marking the data unit with in-band or sideband data having a marker (e.g., special bits or other metadata) that indicates that it is part of an over-rate flow. The marking of the data unit signals to downstream logic such as downstream packet processors and / or traffic managers to process the data unit according to the over-rate policy.

[0058] Block 180 includes features that determine the over-rate policy for the traffic flow. For example, over-rate flow policy logic within the network device can determine one or more features of the over-rate policy to use according to the configuration data, and provide instructions to downstream logic to implement those feature(s). The instructions can take the form of accompanying data units downstream or in-band or sideband data, or flow state information sent separately downstream (e.g., once for the entire flow, rather than each data unit). Alternatively, the determination of block 180 can be made separately at each component of the downstream logic responsible for implementing the over-rate flow policy features. The over-rate policy can include one or more features that are not part of the normal policy for the traffic flow. In some embodiments, different traffic flows can have different over-rate policy features.

[0059] In one embodiment, block 180 can include sending a separate state indicator for each possible feature of the over-rate policy. For any given traffic flow at any given time, certain over-rate policy features can be enabled, such as forwarding to a designated queue, or using a different drop rate or WRED curve, according to the rules associated with the traffic flow and / or the particular attributes of the traffic flow, while other features can not be enabled. For example, a certain rule can indicate that the over-rate WRED curve feature should be enabled for any flow to a particular egress port that becomes over-rate, but the over-rate cloning and collection features should not be enabled at the same time when those flows become over-rate. Thus, when over-rate is detected for a given traffic flow from that particular egress port, a state indicator for the over-rate WRED curve feature can be enabled for that traffic flow, but a state indicator for the over-rate cloning and collection features can not be disabled.

[0060] Flow 100 proceeds from block 180 to block 185, which includes downstream packet switching logic that processes the data unit according to the over-rate policy. Different components of the device, such as described in other sections, can take one or more forwarding actions indicated by the over-rate policy, such as generating and sending a notification, cloning the data unit, forwarding a copy of the data unit to a collector, applying a different drop rate, applying a different WRED curve, re-determining the priority of the data unit, forwarding the data unit to a designated queue, etc. These actions can be in addition to or in place of actions that the device would perform under the normal logic in block 170, according to the features of the policy.

[0061] Flow 100 is merely one example flow for handling over-rate traffic flows in a network device. Other flows can include fewer or more elements, arranged in different ways. For example, in one embodiment, there can be different threshold rates - and thus different counters, decrement amounts, and / or thresholds - for enabling different over-rate policy features. As another example, in some embodiments, block 130 can be performed only after data units are transmitted, and thus the tracked flow rate will reflect the rate of transmitting rather than receiving data units for the flow.

[0062] In one embodiment, flow 100 can also include determining whether the traffic flow is currently enabled for over-rate tracking based on various attributes of the traffic flow. For example, a flow management configuration setting of the network device can disable over-rate tracking for traffic flows from a particular ingress port or destined for a particular egress port. If the traffic flow is not enabled for over-rate management, the network device can skip blocks 130-160 and 180-185 entirely.

[0063] In one embodiment, the counting mechanism can be reversed. Each time a data unit is received (or transmitted), the rate counter for the flow is decremented by a corresponding amount. The counter is periodically incremented by an amount based on the threshold rate or target rate. If the counter reaches zero, the flow is considered an over-rate flow.

[0064] Flow 100 is repeated for any number of data units received by the network device. Depending on the device bandwidth and processing resources, steps from some iterations of flow 100 can be performed concurrently with steps in other iterations of flow 100. In embodiments where blocks 140-150 are performed via a background process, blocks 140-150 are not necessarily performed once per iteration of flow 100, but can be performed once per given time period across all iterations of flow 100.

[0065] 2.2. Flow tracking containers

[0066] According to one embodiment, flow tracking information can be stored in structures within the flow tracking memory referred to as flow tracking containers. In at least some embodiments, it can be undesirable or impractical to allow all traffic flows to always have a flow tracking container from a resource utilization perspective. Thus, there can be a flow tracking container management mechanism to allocate and de-allocate flow tracking containers to and from traffic flows as needed. Figure 2 An example method flow 200 for managing flow tracking containers is shown according to one embodiment.

[0067] Block 210 includes identifying a traffic flow whose rate tracking information should be updated. Block 210 can include, for example, performing a search of the flow tracking containers to identify a flow tracking container that is associated with the traffic flow. Figure 1of FIG. 1, or similar steps, to prepare for performing block 130.

[0068] Block 220 includes identifying a storage space in the flow tracking memory in which to search for a flow tracking container for the identified traffic flow. The storage space can include, for example, a particular storage unit, a particular row or other portion of a storage unit, multiple rows from multiple storage units, etc.

[0069] In one embodiment, locating the storage space can include determining, within the flow tracking memory, one or more index addresses to which the identified traffic flow is mapped. This can include, for example, inputting the traffic flow identifier into a mapping mechanism such as a hash function, a modulo function, a mapping table, etc. In one embodiment, multiple mapping mechanisms can be used to locate multiple different portions of the allocated storage space, as described below.

[0070] In one embodiment, there can be a single storage space for all flow tracking containers, in which case block 220 can be skipped.

[0071] Block 230 includes searching the storage space identified in block 220 for a flow tracking container allocated to the identified traffic flow. Each flow tracking container can include, for example, an identifier of the flow with which it is currently associated. Block 230 will thus include comparing the flow identifier determined in block 210 to each flow tracking container found in the storage space. Note that no other storage spaces need be searched.

[0072] If the allocated flow tracking container is found, flow 200 proceeds to block 240. Block 240 includes updating the flow tracking container with flow tracking information, such as described with respect to block 130 of FIG. 1. This can include, for example, incrementing a rate counter within the container, resetting a timeout value, etc. In one embodiment, this can also or instead include updating one or more over-rate status indicators or other tracking information within the flow tracking container. Figure 1

[0073] If the allocated flow tracking container is not found, flow 200 proceeds to block 250. Block 250 includes determining whether a flow tracking container can be allocated to the identified flow within the storage space. For example, if there is free space in the storage space that has not been allocated to another flow tracking container, a flow tracking container can be allocated. In one embodiment, a background process can have marked certain existing flow tracking containers as inactive or deallocated. Thus, at this stage, such inactive or deallocated flow tracking containers can be overwritten by new flow tracking containers.

[0074] ​If a flow trace container can be allocated, flow 200 proceeds to block 260, which includes allocating and storing a flow trace container within the storage space. For example, in one embodiment, each storage space can include a number of slots, each of which can be occupied by a flow trace container. If one of these slots is empty, a flow trace container can be created within that slot. Flow 200 can then proceed to block 240 for writing tracking information to the flow trace container, including the flow identifier of the newly identified traffic flow.

[0075] If a flow trace container cannot be allocated, flow 200 proceeds to block 270. Block 270 includes identifying one or more existing flow trace containers within the storage space that meet the conditions for being ejected (or deallocated) from the storage space in order to make room for a flow trace container for the flow identified in block 210. Alternatively, viewed from another perspective, block 270 includes identifying one or more existing flow trace containers that can be reallocated or repurposed for the flow identified in block 210.

[0076] In one embodiment, the flow trace containers that meet the conditions are any flow trace containers whose rate counters are below an ejection threshold. In one embodiment, the set can also be filtered by other criteria. For example, if a re-prioritization feature of an over-rate policy is currently enabled for a certain traffic flow, its container can not meet the conditions for reallocation unless a timeout value stored therein indicates that the particular traffic flow has been idle for a certain amount of time to avoid out-of-sequence transmission of data units belonging to the particular traffic flow.

[0077] Flow 200 then proceeds to block 290. Block 290 includes selecting a particular one of the flow trace containers that meet the conditions to be repurposed for the newly identified traffic flow. Different selection policies can be used in different embodiments. For example, in one embodiment, a flow trace container to be replaced is selected at random.

[0078] In one embodiment, if no flow trace container that meets the conditions is identified in block 270, then the container with the smallest byte count is selected instead. In another embodiment, if no flow trace container that meets the conditions is identified, then flow 200 can terminate without storing or updating any tracking information for the newly identified flow.

[0079] Block 295 includes deallocating the selected flow trace container and reallocating the space it occupies for storing a new flow trace container for the newly identified traffic flow. This can be viewed as repurposing an existing flow trace container for the newly identified traffic flow. Conceptually, the traffic flow associated with the flow trace container that is deallocated is no longer tracked in order to make room for tracking the newly identified traffic flow.

[0080] Flow 200 then proceeds to block 240 for writing the trace information to the flow trace container. Any counters or timeout values are reset for the newly identified flow, and the identifier for the newly identified flow is stored therein.

[0081] Flow 200 is merely one example flow for managing flow trace containers. Other flows can include fewer or more elements in different arrangements. For example, in one embodiment, the eligible container and pop threshold concepts can be omitted, and containers with the smallest byte count can always be released.

[0082] Multi-part storage space

[0083] Storage space mapping mechanisms such as described with reference to block 220 can map different flows to the same portion of memory. In many embodiments, this portion of memory need not be large enough to store flow trace containers for all of the flows mapped to this portion. Thus, in essence, a method such as flow 200 can be utilized to determine which flows can actually store flow trace containers within this portion of memory.

[0084] In one embodiment, if a set of flows is mapped to other portions of flow trace memory that are primarily inactive while the set of flows includes many active flows, inefficient resource usage can result due to the set of flows being able to store flow trace containers only in the same region. That is, even though there are other portions of memory available to trace active flows, only a limited number of active flows can be traced.

[0085] In one embodiment, to reduce the likelihood of this situation, the storage space to which flow trace containers are mapped can include a combination of memory portions, each located by a different mapping mechanism. For example, there can be a first hash function and a second hash function, the first hash function resolving to a first memory portion to which a flow is mapped, and the second hash function resolving to a second memory portion to which the same flow is mapped. The storage space allocated to a first flow can include memory portion A resolved by the first hash function and memory portion K resolved by the second hash function. The first hash function can also resolve to a second flow to memory portion A, and the second hash function can resolve to the second flow to memory portion L. Thus, the first flow and the second flow will have partially overlapping but distinct storage spaces in which they can store flow trace containers. Thus, if memory portion A is occupied by flow trace containers for a highly active flow, flow trace containers can still be allocated from memory portions K (for the first flow) and L (for the second flow).

[0086] According to an embodiment, the storage space can include any number of memory portions, and the memory portions need not be contiguous. In fact, in one embodiment, each memory portion is located in a different memory bank, so that they can be read simultaneously.

[0087] 2.3. Example Excessive Rate Policy Features

[0088] According to an embodiment, the excessive rate policy can include various features that affect the device's handling of data units belonging to an excessive rate flow. Each feature can indicate a particular action that the device should perform when handling data units from the affected traffic flow, which would not normally be performed.

[0089] The exact features of the policy can be set globally for all excessive rate flows, or can be customized for particular flows or groups of flows. In one embodiment, there can be different rate level classifications, and different features can be enabled for different classifications. For example, there can be an excessive rate threshold and an extreme excessive rate threshold, and the set of features enabled at each threshold can be different.

[0090] The following are example features that can be enabled for the excessive rate policy. Features other than those listed below can also be equally supported.

[0091] Excessive Rate Flow Notification

[0092] According to one embodiment, the excessive rate policy can include an excessive rate notification feature. Generally, the excessive rate notification feature causes the device to send a notification to a collection entity about an excessive rate flow. In one embodiment, the notification takes the form of a cloned data unit from the traffic flow that is forwarded to the collector instead of the destination specified by the data unit. Such a cloned data unit can include an excessive rate notification indicator in its header, as well as potentially other status information. In other embodiments, the notification can be a separate message generated by the device.

[0093] In another embodiment, the notification can include modifying the first detected packet or the first detected packet and all subsequent packets of the excessive rate flow to indicate that the flow exceeds the rate. This can be done by adding an attribute to the packet (similar to in-band telemetry) or modifying a select bit of the packet.

[0094] A collection entity is a processing element, such as a separate server, off-chip central processing unit, graphics processor unit, etc., that is configured to collect information related to the operation of network devices and / or networks in which the network devices are deployed. While the exact capabilities of such a collector can vary, the collector can include reporting logic, analysis components, interface(s) for presenting the collected information to a management user, etc. In one embodiment, the collector can include logic for taking corrective action in response to certain network conditions indicated by the collected information, such as sending reconfiguration instructions to affected network devices. For example, the collection entity can be a dedicated network management appliance, off-chip central processing unit, etc.

[0095] According to one embodiment, there can be different kinds of over-rate notification features. One such variation can cause the device to clone and send only the first data unit in a traffic flow to the collector after detecting that the traffic flow has become an over-rate flow. Another such variation can cause the device to clone and send every data unit in a traffic flow to the collector upon detecting that the traffic flow has become an over-rate flow. Yet another variation can clone and send only a sample of the data units in an over-rate traffic flow. The sample can be selected according to some pattern (e.g., every tenth data unit), randomly based on a probability threshold, or based on some characteristic of the sampled data units.

[0096] Differential congestion notification

[0097] Network devices can be configured to send congestion notifications to the sender and / or receiver of data units being transmitted through the network device at various times. The communicating parties can implement a communication protocol that uses these notifications as an indicator of when and how to take action to reduce congestion, e.g., to slow the transmission rate or take other corrective measures to mitigate the consequences of the congestion. A common protocol for congestion notification is Explicit Congestion Notification, which typically involves a device modifying or inserting a particular marker (e.g., two ECN bits in an IPv4 header) into certain data units when the device is experiencing congestion or detects conditions that are likely to lead to congestion.

[0098] According to one embodiment, the over-rate policy can include a differential congestion notification feature. This feature changes the conditions under which congestion notifications are sent, resulting in notifications being sent earlier (or later) for over-rate traffic flows. When a certain component of the packet switching logic, e.g., a traffic manager, processes a data unit from an over-rate flow, the component essentially changes the test it uses to determine whether to issue a congestion notification associated with the data unit (e.g., to insert an ECN marker in the data unit).

[0099] WRED and ECN curve selection

[0100] In one embodiment, the device can be configured to selectively mark or drop data units according to a curve. The WRED curve is used to select data units to drop, while the ECN curve is used to select data units to mark. Different curves can be used when dealing with over-speed flows. The device can implement both WRED and ECN curves, or only one of the curves. In addition, the device can use the curves to decide when to take other types of actions, and these curves can also be different for over-speed traffic flows.

[0101] In one embodiment, traffic management logic within the device applies the curves when queuing data units in a queue to determine whether a packet should be admitted or dropped. Each curve specifies a mapping of a selected metric (e.g., average queue size) to a probability value. For example, the x-axis of the curve can be the metric, while the y-axis can be the probability.

[0102] The weighted average queue size (or queue delay or other suitable metric) is continuously computed. The current value of the measurement is compared to the curve to determine a probability of taking an action for a given packet. Once the probability is resolved for a given packet, the decision to take the action indicated by the curve (e.g., drop for WRED or mark for ECN) is determined by generating a random number and comparing it to a threshold value corresponding to the resolved probability.

[0103] If the flow is determined to be an over-speed flow, a more aggressive curve can be selected. Such a curve will more aggressively drop or mark, thus not severely impacting compliant flows, nor allowing less aggressive curves to be used for compliant flows.

[0104] Instead of queue size, a different metric such as average queue delay can be used.

[0105] In one embodiment, the network device can include multiple different ECN profiles, each profile describing a different ECN curve. A group of traffic flows that have some common attributes can generally be assigned to a first ECN profile. However, if a traffic flow from the group becomes an over-speed flow, the device can apply a second ECN profile to the traffic flow. Similarly, the device can include multiple different WRED profiles.

[0106] Differential dropping

[0107] According to one embodiment, the over-speed policy can include a differential dropping feature. Generally, when the device identifies a traffic flow as over-speed in the presence of the differential dropping feature, it is more likely that data units belonging to the over-speed will be dropped than other traffic flows.

[0108] In one embodiment, the differentiated discard feature can cause the device to adjust the threshold at which it begins discarding data units associated with a traffic flow. For example, as data units await processing and egress from the network device, the traffic manager can assign the data units to a queue. The traffic manager can be configured to discard data units assigned to a particular queue once that particular queue reaches a particular size. In the case where differentiated discard is enabled, when deciding whether to discard a data unit, the traffic manager can compare the queue size or other metric to a different threshold if the data unit is from an over-speed flow rather than a regular traffic flow. Thus, over-speed flows can be configured to be discarded earlier than normal flows in the presence of congestion. If there is no congestion, then in some configurations no action can be needed as the device is not stressed by the over-speed flows. Of course, other metrics such as queue delay can be used instead of queue size.

[0109] In one embodiment, rather than discarding all data units once the queue size reaches a threshold, the device increases the frequency with which data units from a traffic flow are discarded. Depending on the embodiment, this can be implemented in various ways. For example, the traffic manager can simply be instructed to discard a larger sample of data units than it otherwise would (e.g., every ten data units rather than every hundred).

[0110] Re-determining priority

[0111] According to one embodiment, the over-speed policy can include a re-determine priority feature. Generally, this function causes the device to prioritize data units from over-speed flows differently than it otherwise would prioritize them. This can result in, for example, a change in the amount of time a data unit waits in the network device before being forwarded to its next hop and / or a change in the likelihood that the device will take certain actions such as discarding the data unit or requesting that the sender pause transmission of the data unit. For example, in one embodiment, over-speed data flows can be assigned a lower priority than normal traffic flows, such that data units from over-speed data flows are more likely to be delayed or discarded.

[0112] In one embodiment, as data units await processing and egress from the network device, the data units can again be assigned to a queue. The network device can rely on a scheduler to determine which queue to select the next data unit to process from at any given time. Data units from over-speed data flows can be placed in a designated queue that is different from the queue they would be placed in if their traffic flow were not experiencing a high rate of traffic.

[0113] In one embodiment, re-prioritization is performed when the forwarding logic initially assigns a data unit to a queue. In another embodiment, re-prioritization can be performed downstream by a traffic manager. For example, a queue associated with a port can be identified by a combination of a port identifier and a queue offset. When a data unit arrives at the traffic manager, the queue can already be assigned (e.g., by control information generated for the data unit). However, the traffic manager can re-assign a data unit from an over- rate data stream to a queue associated with the same port identifier that has already been assigned to the data unit, but with a different queue offset, such as a designated queue offset that is specifically associated with over-rate data streams.

[0114] According to an embodiment, the scheduler can be configured to process the queue more frequently or less frequently than other queues, such that data units can end up being processed and transmitted sooner or later than they otherwise would. At the same time, various traffic management decisions (e.g., dropping data units) conditioned on queue size or other metrics can occur more frequently or less frequently due to the more frequent or less frequent processing of the designated queue. In addition, the designated queue can have different thresholds associated with it that also affect the likelihood of such decisions (e.g., a lower drop threshold).

[0115] In one embodiment, the queue can be allocated less buffer space than other queues. In one embodiment, the queue can be limited to a particular transmission rate.

[0116] In other embodiments, re-prioritization can be achieved through mechanisms other than the use of a designated queue.

[0117] 2.4. Other

[0118] According to one embodiment, over-rate flow tracking and / or management is enabled only for unicast traffic. In one embodiment, a network device can include a global configuration setting that selectively enables or disables over-rate flow tracking and management, allowing over-rate flow tracking and management to be programmatically enabled under certain system conditions (e.g., at certain times, when the system experiences a certain level of congestion, etc.).

[0119] In one embodiment, a device can be configured to count the number of times a device policy takes a particular action specified by an over-rate flow policy, such as the number of times an over-rate flow policy causes a data unit from a flow to be dropped, the number of times an over-rate flow policy causes a notification to be triggered, or the number of data units that have been re-prioritized to a designated queue. In one embodiment, these counts can be used to determine when to take additional actions, in addition to reporting and analysis purposes.

[0120] 3.0. System Overview

[0121] Figure 3 is a schematic diagram of various components of an example system 300 according to one embodiment, configured for flow tracking and management according to the techniques described herein. The system 300 can be a subsystem within a switch or other network device, as described elsewhere herein. For example, in one embodiment, the system 300 forms part of the forwarding logic of an implementation device, such that traffic flows are inspected as data units come in for the purpose of high-rate management. In one embodiment, the various components of the system 300 described below are hardware-based logic units within an application-specific integrated circuit (ASIC), field programmable gate array (FGA), or other integrated circuit(s). In other embodiments, some or all of the components can be software-based logic implemented by one or more processors executing instructions stored in one or more computer-readable media.

[0122] 3.1. Data unit receiver

[0123] The system 300 includes a data unit receiver 310. The data unit receiver 310 receives a data unit 302 from a sending entity such as a communication interface, ingress arbiter, packet processor, or other appropriate component of a network device.

[0124] Based on information in the data unit 302 or sideband information accompanying the data unit 302, the data unit receiver 310 identifies a flow tracking configuration profile 312 associated with the data unit 302. For example, there can be a different configuration profile 312 associated with each egress port of a network device. The data unit receiver 310 can receive control information from an upstream packet processor, for example, indicating the egress port(s) to which the data unit is to be forwarded. The data unit receiver 310 will thus identify the configuration profile 312 associated with the data unit. Alternatively, the data unit receiver 310 can be configured to map different configuration profiles 312 to different data unit attributes or combinations of data unit attributes. In any case, the data unit receiver 310 forwards the identified profile 312 to a flow management configuration resolver 320.

[0125] The data unit receiver 310 also extracts certain information from the data unit 302, referred to herein as flow hash information 314, based on which the associated flow tracking information is identified. For example, the data unit receiver 310 can extract one or more header fields used to derive a flow identifier, as described elsewhere herein. The data unit receiver 310 forwards the extracted flow hash information 314 to a flow tracking context resolver 330.

[0126] 3.2. Flow management configuration resolver

[0127] The flow management configuration resolver 320 uses the profile 312 to resolve various configuration settings for flow tracking and / or management of any traffic flows associated with the profile 312. Each profile 312 can be associated with one or more traffic flows that share the same characteristics. For example, where the profile 312 corresponds to an egress port, all traffic flows that target that egress port will be associated with the profile 312. The configuration settings are specific to the flows associated with the respective profile, so flows from different profiles can have different tracking or management settings (e.g., different policy characteristics, different threshold values, etc.).

[0128] The set of available configuration options varies from embodiment to embodiment, but can include other options: an option to enable or disable flow tracking, an option to enable or disable over-rate flow management, an option to set a threshold rate to compare against a flow's rate to determine when to apply an over-rate policy, an option to set a target rate, an option to set a threshold to compare against a rate counter to determine when the over-rate policy should be applied, an option to set a timeout period after which to consider a flow idle, an option to set a maximum rate counter size, an option to set a decrement amount to decrement from a rate counter at each refresh period, an option to enable an over-rate notification feature to apply when a flow is detected to be an over-rate flow, an option to enable a reprioritize feature to apply when a flow is detected to be an over-rate flow, an option to enable a differentiated drop feature to apply when a flow is detected to be an over-rate flow, an option to enable a differentiated congestion notification feature to apply when a flow is detected to be an over-rate flow, and / or an identifier of a WRED or ECN curve to use when a flow is determined to be an over-rate flow.

[0129] In one embodiment, each profile can have a different set of memory portions in which flow tracking containers for the respective traffic flows can be held. Thus, the configuration options can further specify the location(s) of those memory portions, and in some embodiments, the number of those memory portions (e.g., the size of memory allocated to the profile for storing flow tracking containers).

[0130] The flow management configuration resolver 320 can include or be coupled to various memory in which settings for the above options can be stored on a per-profile basis. Any suitable structure can be used, such as a configuration table. In one embodiment, the structure can be manipulated through any suitable user or programmatic interface.

[0131] The flow management configuration resolver 320 uses the structure to resolve the appropriate settings for an input profile 312. The flow management configuration resolver 320 can be coupled to any components that rely on those settings (e.g., the flow tracking container resolver 330, the flow tracker 340, and / or the flow policy resolver 350) so as to output each resolved setting to the appropriate component(s) for use in processing the data units 302 for which the settings were resolved.

[0132] 3.3. Stream tracking container resolver

[0133] Based on the input stream hash information 314, the stream tracking container resolver 330 identifies a storage space in which a stream tracking container associated with the data unit 302 (or, more precisely, the traffic flow to which the data unit 302 belongs) can be stored. According to embodiments, the storage space can store only a stream tracking container specific to the stream of the data unit, or can store multiple stream tracking containers.

[0134] To identify the storage space, the stream tracking container resolver 330 determines storage space address identification information 332, which according to embodiments can include a number of different sub-components. For example, the storage space address identification information can include a flow identifier, or "flow key," of the traffic flow to which the data unit 302 belongs. The stream tracking container resolver 330 can input some or all of the stream hash information 314 into a flow identifier function that outputs a flow identifier. The flow identifier function can include any suitable computation, including one or more hash functions, a modulo operation, and the like. Alternatively, the stream hash information 314 can be used directly as the flow identifier.

[0135] In one embodiment, the storage space address identification information 332 can also include one or more index values. Each index value can be computed from, or otherwise mapped to, the stream hash information 314. For example, in one embodiment, each index value is a hash value produced by (or derived from) a different hash function of the stream hash information 314.

[0136] The storage space address identification information 332 can include other components, such as a base address or offset of the associated profile 312 (although this can be determined by the configuration resolver 320 and passed directly to the stream tracker 340). In some embodiments, the stream tracking container resolver 330 resolves one or more memory portion addresses of the storage space based on the components of the storage space address identification information 332, and sends the address to the stream tracker 340. In other embodiments, the address resolution is actually performed at the stream tracker 340, and the stream tracking container resolver 330 sends the individual components of the storage space address identification information 332 to the stream tracker 340 accordingly.

[0137] 3.4. Stream tracker

[0138] The system 300 also includes a flow tracker unit 340 that implements flow tracking logic and over-rate flow management logic for the network device. As described above, the flow tracker 340 receives storage space address identification information 332 from the flow tracking container resolver 330. Based on this information 332, the flow tracker 340 resolves one or more addresses that constitute one or more particular memory portions of the storage space if the flow tracking container resolver 330 has not already resolved these addresses. For example, the flow tracker 340 can look up flow identifiers in the storage space address identification information 332 from the memory map that indicate which portion(s) have been allocated to which flow(s). Alternatively, if necessary, the flow tracker 340 can convert one or more index values in the storage space address identification information 332 to addresses within one or more respective storage units, and further take into account the size of each memory portion and / or an offset specified for the associated profile.

[0139] The flow tracker 340 reads the contents of each memory portion address in the region. In embodiments where there is only a single flow tracking container per storage space, no further steps are needed to obtain the flow tracking container. In other embodiments, the flow tracker 340 compares the flow identifier to the value of the flow identifier for each flow tracking container to locate the flow tracking container for the data unit 302.

[0140] If no flow tracking container is found, the flow tracker 340 can attempt to allocate a new flow tracking container for the traffic flow of the data unit within the identified storage space. If there is no space within the region, the flow tracker 340 can optionally search for an existing container that meets the conditions for deallocation using techniques described elsewhere herein. A new flow tracking container can then be created in the space previously allocated for the existing container.

[0141] Assuming a flow tracking container is found or allocated for the traffic flow of the data unit, the flow tracker 340 then updates information within the flow tracking container. This typically includes updating a rate counter and / or a log to indicate that the data unit 302 was received.

[0142] In some embodiments, the update can also include implementing over-rate flow monitoring logic that determines whether the current traffic exceeds a threshold rate using techniques such as described in other sections. This can also include updating one or more status indicators to reflect that the over-rate policy is enabled (or not enabled) depending on the result of the comparison.

[0143] In other embodiments, the comparison and updating of the super- rate status indicator is performed asynchronously by a background super- rate flow monitoring process that periodically (or at other intervals) processes each flow tracking container. Such a background process can also perform or instead perform other steps, such as decrementing the timeout value, determining whether to update the active or idle status indicator, deallocating flow tracking containers for inactive traffic flows, etc.

[0144] The flow tracker 340 then sends an indicator 342 to the super- rate flow policy resolver 350 as to whether the traffic flow of the data unit is a super-rate flow (determined by comparing the tracking information to threshold information or by reading the super-rate status indicator within the container).

[0145] Example flow tracker

[0146] Figure 4 An example flow tracker component 440 according to one embodiment is shown. The flow tracker 440 is an example of the flow tracker 340, although the flow tracker 340 can be implemented in other ways without the specific details. Figure 4 The flow tracker 440 includes a flow tracking memory 450, which in turn includes a plurality of memory units (or "banks") 460. Each memory bank 460 is divided into addressable rows 470, and each row 470 includes a plurality of slots 472. Each slot 472 can store at most a single flow tracking container 475. Although Figure 4 Only two memory banks 460 are depicted with ten rows 470 each having four slots 472, but other embodiments can feature additional or fewer memory banks 460, rows 470 per memory bank 460, and / or slots 472 per row 470.

[0147] The flow tracker 440 includes a reader 430 that receives an index value 442 for each row. These index values 442 can be part of the storage space address information 332 received from the flow tracking container resolver 330, for example. The reader 430 uses the index values 442 to locate the rows 470 in each memory bank 460 to read. For example, the reader 430 can read row 470g from memory bank 460a and row 470q from memory bank 460b. The rows 470 that are read constitute the storage space to search for the relevant flow tracking containers.

[0148] The flow trace updater 420 inputs the row 470 read by the reader 430. The flow trace container updater 420 includes a slot selector 410 that selects a slot in which new or updated trace information should be written for the flow corresponding to the input flow key 441. The flow key 441 can be provided by the flow trace container resolver 330. The slot selector 410 includes a flow trace container search component 422 that searches each slot 472 of the row 470 until it finds a slot 472 that stores a flow trace container 475 whose flow identifier matches the flow key 441.

[0149] The slot selector 410 also includes a flow trace container allocation component 428. If the flow trace container search component 422 does not find a matching slot 472, the flow trace container allocation component 428 selects an existing slot 472 in which to create a new flow trace container 475 for the traffic flow corresponding to the flow key 441. The selected slot 472 can be an empty slot 472, or a slot whose flow trace container is to be deallocated and overwritten by the flow trace container for the traffic flow corresponding to the flow key 441. In either case, the flow trace updater 420 includes a writer 435 that writes and / or updates the flow trace container 475 in the selected slot 472, as described in other sections. The writer 435 writes the affected row back to the appropriate location in the flow trace memory 450. The flow trace updater 420 also outputs any necessary state information 478 from the updated or newly written flow trace container 475 to downstream components, such as the over-rate flow policy resolver 350.

[0150] In some embodiments, neither the flow trace container allocation component 428 nor the flow trace container search component 422 locates the slot 472 to be written. In these cases, the writer 435 does not need to write anything back to the flow trace memory 450, and the flow trace updater 420 can simply output a state indicating that the flow corresponding to the flow key is not considered to be an over-rate flow.

[0151] 3.5. Over-Rate Flow Policy Resolver

[0152] The system 300 also includes an over-rate flow policy resolver 350 that is responsible for determining whether to apply an over-rate policy to the data unit 302 and what the characteristics of that policy will be. If the flow tracer 340 indicates that the traffic flow of the data unit 302 is not an over-rate flow, the over-rate flow policy resolver 350 does not perform any operations. Likewise, if the configuration settings from the flow management configuration resolver 310 indicate that over-rate flow management is disabled for the profile 314 of the data unit 302, the over-rate flow policy resolver 350 does not perform any operations. In either case, the data unit 302 is passed to downstream logic 360, such as a traffic manager or packet processor, for normal processing.

[0153] However, if the flow tracker 340 indicates that the traffic flow of the data unit 302 is an overspeed flow, and if overspeed flow management is enabled, then the overspeed flow policy resolver 350 resolves an overspeed policy for the data unit 302. The overspeed flow policy resolver 350 does this by first determining what features the overspeed policy should be. For example, the overspeed flow policy resolver 350 can determine whether a higher drop rate should be applied as part of the overspeed policy, whether an overspeed notification feature should be implemented, and so on. These features can be determined, for example, based on configuration settings received from the flow management configuration resolver 310.

[0154] Once the features of the overspeed policy are resolved, the overspeed flow policy resolver 350 then adds overspeed policy information 352 to the data unit 302, either within its header or as control information that is otherwise transported by the device along with the data unit 302. The overspeed policy information 352 indicates the particular action(s) to be taken as part of the processing of the data unit 302. This information can or can not be removed by downstream logic 360 before the data unit 302 leaves the implementing device.

[0155] The data unit 302 is then forwarded to downstream logic 360 of the implementing device. The downstream logic 360 will see any overspeed policy information 352 associated with the data unit 302, and take the appropriate action(s) indicated by this information 352, if any. If the action(s) do not preclude normal forwarding of the data unit 302, then the downstream logic 360 further processes the data unit 302 in the normal manner (e.g., forwards the data unit 302 to the destination address specified or indicated therein).

[0156] 3.6. Other

[0157] The system 300 is merely an example of a system in which the described techniques can be practiced. Other systems can include fewer and / or more components in different arrangements, and the allocation of work between components can likewise vary. For example, some or all of the functionality of the data unit receiver 310 can actually be performed by one or more upstream components such as an ingress packet processor and / or an arbitrator that are configured to output the flow hash information 314 directly to the flow tracking container resolver 330 and / or the profile 312 directly to the flow management configuration resolver 320.

[0158] In one embodiment, the configuration settings can be hard-coded on a global basis. Thus, the flow management configuration resolver 310 can be omitted. In other embodiments, the configuration settings can be applied on a per-flow basis rather than on a per-profile basis, or can be configured at any other level of granularity.

[0159] 4.0. Example packet switching logic

[0160] As already mentioned, the technology described herein relates to managing network traffic flow through network switches and / or other network devices having packet switching logic. This section introduces example packet switching logic components in network devices in more detail. However, the technology described herein is also useful in other switches and contexts different from those described in this section.

[0161] 4.1. Network

[0162] Figure 5 is a schematic diagram of various aspects of an example network system 500 (also referred to as a network) that can practice the technology described herein, in accordance with an embodiment. The network system 500 includes a plurality of interconnected nodes 510a-510n (collectively, nodes 510), each implemented by a different computing device. For example, a node 510 can be a single network computing device, such as a router or switch, in which some or all of the processing components described herein are implemented in an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other integrated circuit. As another example, a node 510 can include one or more memories storing instructions for implementing the various components described herein, one or more hardware processors configured to execute the instructions stored in the one or more memories, and various data stores in the one or more memories for storing data structures utilized and manipulated by the various components.

[0163] Each node 510 is connected to one or more other nodes 510 in the network 500 by one or more communication links, which are depicted as lines between the nodes 510. The communication links can be any suitable wired or wireless links. Note that the system 500 illustrates only one of many possible arrangements of nodes within a network. Other networks can include fewer or more nodes 510 with any number of links between them.

[0164] 4.2. Data units

[0165] While each node 510 can or can not have various other functionality, in one embodiment each node 510 is configured to send, receive, and / or relay data to one or more other nodes 510 via the links. Generally, data is communicated as a series of discrete data units or structures represented by signals transmitted over the communication links.

[0166] Different nodes 510 within network 500 can send, receive, and / or relay data units at different levels or layers of communication. For example, a first node 510 can send a data unit at the network layer (e.g., a TCP segment) to a second node 510 through a path that includes intermediate nodes 510. Before sending the data unit from the first node 510, the data unit will be broken down into smaller data units at various sub-levels. These smaller data units can be referred to as "subunits" or "portions" of the larger data unit.

[0167] For example, a TCP segment can be broken into packets, then into cells, and finally into a set of signal-encoded bits sent to an intermediate device. Depending on the network type and / or device type of the intermediate nodes 510, the intermediate nodes 510 can reconstruct the entire original data unit before routing the information to the second node 510, or the intermediate nodes 510 can simply reconstruct certain data subunits (e.g., frames and / or cells) and route those subunits to the second node 510 without assembling the entire original data unit.

[0168] When a node 510 receives a data unit, it typically examines addressing information within the data unit (and / or other information within the data unit) to determine how to handle the unit. The addressing information can be, for example, an Internet Protocol (IP) address, an MPLS label, or any other suitable information. If the addressing information indicates that the receiving node 510 is not the destination of the data unit, the receiving node 510 can look up the destination node 510 within the receiving node's routing information and route the data unit to another node 510 connected to the receiving node 510 based on forwarding instructions associated with the destination node 510 (or the address group to which the destination node belongs). The forwarding instructions can indicate, for example, an egress port through which to send the data unit, a label to append to the data unit, etc. In cases where there can be multiple paths to the destination node 510, the forwarding instructions can include information indicating an appropriate method for selecting one of those paths, or a path can have been defined that is considered the best path.

[0169] The addressing information, flags, labels, and other metadata used to determine how to handle a data unit are typically embedded within a portion of the data unit referred to as a header. The header is typically located at the beginning of the data unit, followed by the payload of the data unit, which is the information actually sent in the data unit. The header is typically composed of different types of fields, such as a destination address field, a source address field, a destination port field, a source port field, etc. In some protocols, the number and arrangement of fields can be fixed. Other protocols allow for an arbitrary number of fields, some or all of which are preceded by type information that explains to the node what the field means.

[0170] Node 510 can operate network data at several different layers, and thus treat the same data as belonging to several different types of data units. At a higher level, node 510 can treat data as belonging to a particular type of protocol data unit ("PDU"), such as a packet or data unit at any other suitable network level. Node 510 does not necessarily need to assemble data in a PDU together, but in one embodiment can act on constituent parts of a PDU, which can be linked together by an identifier, linked list, or other suitable construct. These parts are referred to herein as transport data units ("TDUs"). For example, if the PDU is a packet, then a TDU can be one or more cells or frames. The first TDU in a PDU is referred to as the start of packet ("SOP"), while the last TDU in a PDU is referred to as the end of packet ("EOP").

[0171] Generally speaking, a TDU is the largest contiguous unit of data that certain internal components of node 510 are configured to communicate with each other within a given time period. For example, node 510 can have a traffic manager that is capable of receiving no more than a single TDU from each interface during a single clock cycle. Further, in one embodiment, the contiguous portion of data that each port of node 510 transmits within a given time period can be no larger than a TDU. In one embodiment, each TDU has a fixed size, except for the last TDU in a PDU, which can be smaller than the fixed size.

[0172] In some embodiments, a TDU can also be divided into chunks referred to as storage data units ("SDUs") for the purposes of physical storage. In one embodiment, an SDU is the largest contiguous portion of data that can be stored in a physical buffer entry. In other words, the maximum size of an SDU is the same as the maximum size of a physical buffer entry. In one embodiment, the maximum number of SDUs in a TDU is fixed. However, the EOP TDU can have fewer than this number of SDUs. Further, the last SDU in a TDU (e.g., the EOP TDU) can be smaller than the maximum SDU size.

[0173] In one embodiment, TDU and SDU boundaries can be relative to components acting on the data. That is, for example, a node 510 whose traffic manager is configured to use TDU of a first size and SDUs of a second size can also include other components configured to transfer or buffer data units of sizes other than the first and second sizes.

[0174] For convenience, many of the techniques described in this disclosure are described with respect to embodiments in which the PDUs are IP packets in an L3 (level 3) network and the TDUs are constituent cells and frames in an L2 (level 2) network, in which context the described techniques have particular advantages. However, it will be recognized that the techniques can also be applied to achieve advantages in other types of data units that are routed in accordance with other protocols and / or at other communication layers within a network. Thus, unless otherwise noted or apparent, the techniques described herein should also be understood to apply in the context of any other type of data structure in which PDUs, TDUs, and SDUs are communicated through a network, such as segments, InfiniBand messages, or datagrams. That is, in these contexts, other types of data structures can be used in place of packets, cells, frames, and so on.

[0175] 4.3. Network Path

[0176] Any node in the depicted network 500 can communicate with any other node in the network 500 by sending data units through a series of nodes 510 and links, referred to as a path. For example, node B (510b) can send a data unit to node H (510h) via a path from node B to node D to node E to node H. There can be a large number of valid paths between two nodes. For example, another path from node B to node H is from node B to node D to node G to node H.

[0177] In one embodiment, the nodes 510 do not actually need to specify a complete path for data units that they send. Instead, the nodes 510 can simply be configured to compute the best path for a data unit to exit the device (e.g., on which egress port it should be sent). When a node 510 receives a data unit that is not directly addressed to the node 510, based on header information associated with the data unit, such as path and / or destination information, the node 510 relays the data unit either to the destination node 510 or to a "next hop" node 510 that the node 510 computes is in a better position to relay the data unit to the destination node 510. In this way, the actual path of a data unit is the product of each node 510 along the path making routing decisions about how best to move the data unit to the destination node 510 identified by the data unit.

[0178] 4.4. Network Device

[0179] Figure 6is a schematic diagram of various aspects of an example network device 600 that can practice the techniques described herein, according to an embodiment. The network device 600 is a computing device that includes any combination of hardware and software configured to implement the various logical components described herein, including components 610-690. For example, the apparatus can be a single network computing device, such as a router or switch, in which application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) are used to implement some or all of the components 610-690 described herein. As another example, the apparatus can include one or more memories that store instructions for implementing the various components described herein, one or more hardware processors configured to execute the instructions stored in the one or more memories, and various data stores in the one or more memories for storing data structures used and manipulated by the various components 610-690.

[0180] The device 600 is generally configured to receive and forward data units 605 through a series of operations performed at various components within the device 600 to other devices in a network, such as the network 500. Note that certain nodes 510 in a system such as the network 500 can each be or include a separate network device 600. In one embodiment, a node 510 can include more than one device 600. In one embodiment, a device 600 can itself be one of multiple components within a node 510. For example, the network device 600 can be an integrated circuit or chip that is dedicated to performing switching and / or routing functions within a network switch or router. The network switch or router can also include one or more central processor units, storage units, memories, physical interfaces, LED displays, or other components outside of the chip, some or all of which can communicate with the chip.

[0181] A non-limiting example flow of a data unit 605 through the various subcomponents of the switch logic of the device 600 is as follows. After receiving a data unit 605 via a port 610, the data unit 605 can be buffered by an arbiter until the data unit 605 can be processed by an ingress packet processor 650, which then passes to an interconnect connection. The data unit 605 can be forwarded from the interconnect connection to a traffic manager 640. The traffic manager 640 can store the data unit 605 in a buffer 644 and assign the data unit 605 to a queue 645. The traffic manager 640 manages the flow of data units 605 through the queue 645 until the data unit 605 is released to an egress packet processor 650. Depending on the processing, the traffic manager 640 can then assign the data unit 605 to another queue so that the data unit 605 can be processed by another egress processor 650, or the egress packet processor 650 can send the data unit 605 to an egress arbiter from which the data unit 605 is ultimately forwarded out another port 690. Of course, depending on the embodiment, the switch logic can omit some of these subcomponents and / or include other subcomponents in a different arrangement.

[0182] Example components of the device 600 are now described in more detail.

[0183] 4.5. Ports

[0184] The network device 600 includes ports 610 / 690. The ports 610, including ports 610a through 610n, are ingress ("ingress") ports through which data units, referred to herein as data units 605, are received on a network such as the network 500. The ports 690, including ports 690a through 690n, are egress ("egress") ports through which at least some of the data units 605 are sent to other destinations within the network after being processed by the network device 600.

[0185] The data units 605 can be any suitable PDU type, such as packets, cells, frames, etc. In one embodiment, the data units 605 are packets. However, the various atomic data units on which the described components operate can actually be subunits of the data units 605, such as the previously described TDUs. For example, the data units 605 can be received, acted upon, and sent at the cell or frame level. To determine how to process these cells or frames, the cells or frames can be logically linked together as the data units 605 (e.g., packets) to which they respectively belong. However, the subunits can not actually be assembled into the data units 605 within the device 600, particularly if the subunits are being forwarded through the device 600 to another destination.

[0186] For illustration purposes, ports 610 / 690 are depicted as separate ports, but can in fact correspond to the same physical hardware port (e.g., network jack or interface) on network device 610. That is, network device 600 can receive data units 605 and transmit data units 605 on a single physical port, and that single physical port can thus function as both ingress port 610 and egress port 690. However, for various functional purposes, certain logic of network device 600 can treat the single physical port as separate ingress port 610 and egress port 690. Further, for various functional purposes, certain logic of network device 600 can subdivide a single physical ingress or egress port into multiple ingress ports 610 or egress ports 690, or aggregate multiple physical ingress or egress ports into a single ingress port 610 or egress port 690. Thus, in various embodiments, ports 610 and 690 should be understood as different logical constructs mapping to a physical port, rather than simply as different physical constructs.

[0187] In some embodiments, each port 610 / 690 of device 600 can be coupled to one or more transceivers in a serializer / deserializer ("SerDes") block or other suitable component through which device 600 receives and transmits data.

[0188] 4.6. Packet Processors

[0189] Device 600 includes one or more packet processing components 650. These packet processors 650 can be any suitable combination of fixed circuitry and / or software-based logic, such as specific logic components implemented by one or more field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), or general-purpose processors executing software instructions.

[0190] Different packet processors 650 can be configured to perform different packet processing tasks. These tasks can include, for example, identifying a path for forwarding data units 605, forwarding data units 605 to egress ports 690, implementing flow control and / or other policies, manipulating packets, performing statistical or debugging operations, etc. Device 600 can include any number of packet processors 650 configured to perform any number of processing tasks.

[0191] In one embodiment, the packet processors 650 within the device 600 can be arranged such that the output of one packet processor 650 can ultimately be input into another packet processor 650 in a manner that passes the data unit 605 from some packet processor(s) 650 to other packet processor(s) 650 in a series of stages until the data unit 605 is finally disposed of (e.g., by sending the data unit 605 out an egress port 690, "dropping" the data unit 605, etc.). In some embodiments, the exact set and / or sequence of packet processors 650 that process a given data unit 605 can vary depending on the attributes of the data unit 605 and / or the state of the device 600. Any number of packet processors 650 can be linked together in this manner.

[0192] Based on decisions made in processing the data unit 605, in some embodiments, and / or for certain processing tasks, the packet processor 650 can directly manipulate the data unit 605. For example, the packet processor 650 can add, delete, or modify information in the data unit header or payload. In other embodiments, and / or for other processing tasks, the packet processor 650 can generate control information that accompanies or is merged with the data unit 605 as the data unit 605 continues through the device 600. This control information can then be utilized by other components of the device 600 to implement decisions made by the packet processor 650.

[0193] In one embodiment, the packet processor 650 need not process the entire data unit 605, but can instead receive and process only a sub-unit of the data unit 605, such as a TDU that includes header information of the data unit. For example, if the data unit 605 is a packet that includes a plurality of cells, the first cell or a subset of the first cells can be forwarded to the packet processor 650, while the remaining cells of the packet (and potentially the first cell(s) as well) are forwarded in parallel to a merge component, where they await the results of processing.

[0194] Ingress and egress processors

[0195] In one embodiment, packet processors can generally be classified as ingress packet processors 650 or egress packet processors 650. Generally, ingress processors 650 resolve the destination for the traffic manager 640 to determine from which ports 690 and / or queues 645 the data unit 605 should exit. There can be any number of ingress processors 650, including only a single ingress processor 650.

[0196] In one embodiment, the ingress processor 650 performs certain ingress tasks on data units 605 as they arrive. These ingress tasks can include, for example and without limitation, parsing the data units 605, performing routing-related lookup operations, classifying to block data units 605 with certain attributes and / or replicating certain types of data units 605 when the device 600 is in a particular state, initial classification of data units 605, and so on. Once the appropriate ingress task(s) are performed, the data units 605 are forwarded to the appropriate traffic manager 640, to which the ingress processor 650 can be directly coupled or coupled via various other components such as interconnect components.

[0197] In contrast, the egress packet processor(s) 650 of the device 600 can be configured to perform non-ingress tasks necessary to implement the switching logic of the device 600. These tasks can include, for example, tasks such as identifying paths to forward data units 605, implementing flow control and / or other policies, manipulating data units, performing statistical or debugging operations, and so on. In one embodiment, there can be different egress packet processor(s) 650 assigned to different flows or other categories of traffic, such that not all data units 605 will be processed by the same egress packet processor 650.

[0198] In one embodiment, each egress processor 650 is coupled to a different set of egress ports 690 to which it can send data units 605 processed by the egress processor 650. In one embodiment, access to a set of ports 690 can be regulated by an egress arbiter coupled to the egress packet processor 650. In some embodiments, the egress processor 650 can also or alternatively be coupled to other potential destinations such as an internal central processing unit, a storage subsystem, or a traffic manager 640.

[0199] 4.7. Buffers

[0200] Because not all data units 605 received by the device 600 can be processed by the packet processors 650 at the same time, various components of the device 600 can temporarily store data units 605 in one or more buffers 644 as the data units 605 await processing. For example, a particular packet processor 650 can only be able to process a particular number of data units 605 or portions of data units 605 in a given clock cycle, meaning that other data units 605 or portions of data units 605 destined for the packet processor 650 must either be ignored (i.e., dropped) or stored. At any given time, depending on network traffic conditions, a large number of data units 605 can be stored in the buffers 644 of the device 600.

[0201] Device 600 can include various buffers 644, each of which is used for a different purpose and / or component. Generally, data units 605 that are waiting for a component to process are saved in a buffer 644 associated with that component until the data unit 605 is "released" to the component for processing. For example, traffic manager 640 will generally have a relatively large buffer 644, referred to as an egress buffer, in which it buffers data units 605 before releasing them to egress packet processor 650.

[0202] According to embodiments, buffers 644 can be implemented using a single physical memory unit (e.g., SRAM, DRAM, etc.), a designated portion of a memory unit, or a collection of memory units. Buffers 844 are divided into addressable units or entries that store SDUs, one or more of which form a TDU. Each TDU stored in a buffer 644 belongs to a PDU. However, the data belonging to a TDU of a PDU does not have to be stored adjacent to each other. If it is desired to reconstruct a PDU based on the buffered SDUs, it can not be possible to do so using only TDU buffer storage. Thus, in one embodiment, buffers 644 can also store or be associated with link data that indicates which SDUs belong to a given TDU and / or which TDUs belong to a given PDU, which is also referred to as intra-packet link data.

[0203] For each PDU, buffers 644 can also store or be associated with various PDU metadata. PDU metadata can include any suitable information about a PDU, such as a PDU identifier, a location(s) for link data for the PDU (e.g., an address(es) for an intra-packet entry(s) where a link list(s) for the PDU begins), a count of TDUs in the PDU, source information, destination information, control information, a timestamp, statistics, an assigned queue, flow control information, and so on.

[0204] 4.8. Queues

[0205] In one embodiment, to manage the order in which data units 605 from buffers 644 are processed, various components of device 600 can implement queuing logic. For example, the flow of data units 605 through egress buffer 644 of traffic manager 640 can be managed using an egress queue, while the flow of data units 605 through a buffer of an ingress arbiter can be managed using an ingress queue.

[0206] A queue 645 is a collection of nodes arranged in an order by metadata describing the queue 645. A queue 645 includes a head node or head, which is typically the next node to be processed, and a tail node or tail, which is typically the most recently added node to the queue. As time passes, other nodes are processed and removed from the queue, and nodes typically progress from the tail to the head.

[0207] In the case of a queue 645, a node is a data unit 605, or a buffer location(s) where a data unit 605 begins. A data unit 605 that has been added to a queue 645 is said to be "linked" to that queue 645. A data unit 605 can belong to one or more queues 645.

[0208] In many embodiments, the order in which a queue 645 arranges its constituent data units 605 typically corresponds to the order in which data units 605 or portions of data units in the queue 645 are to be released and processed. Such a queue 645 is referred to as a first-in-first-out ("FIFO") queue, although other types of queues can be used in other embodiments. In some embodiments, the number of data units 605 or portions of data units that are allocated to a given queue 645 at a given time can be global or per-queue based, and this limit can change over time.

[0209] In one embodiment, a queue 645 is implemented using a linked list memory separate from the associated buffer memory 644, referred to as "inter-packet" linked list memory. Each entry in the linked list memory is considered to be a node in the queue. Each linked list entry point includes a data pointer that points to a memory location in the buffer memory 844 where the corresponding data unit (or at least the beginning of the data unit) is found (e.g., a buffer entry, the first entry in the data unit in the inter-packet linked list memory, etc.) when the linked list entry is occupied. Each entry in the linked list memory can also include a link pointer to another linked list entry that corresponds to the next node in the queue. Of course, other types of linked list memory and / or other structures can be used to represent a queue in other embodiments.

[0210] 4.9. Traffic Management

[0211] According to one embodiment, the device 600 also includes one or more traffic managers 640 configured to control the flow of data units 605 to one or more packet processors 650. The traffic managers 640 can receive data units 605 directly from the ports 610, from the ingress processors 650, and / or from other appropriate components of the device 600. In one embodiment, the traffic managers 640 are configured to receive up to one TDU from each possible source (e.g., each port 610) per clock cycle of the traffic manager 640.

[0212] The traffic manager 640 can include or be coupled to a buffer 644 for buffering data units 605 prior to sending the data units 605 to their respective processor(s) 650. While a data unit 605 is waiting to be processed by the processor(s) 650, a buffer manager within the traffic manager 640 can temporarily store the data unit 605 in the buffer 644. A data unit 605 or portion of a data unit in the buffer 644 can eventually be“released” to one or more processors 650 for processing by reading the data unit 605 from the buffer 644 and sending the data unit 605 to the processor(s) 650. In one embodiment, the traffic manager 640 can release up to a particular number of data units 605 from the buffer 644 to the processors 650 per clock cycle.

[0213] In addition to managing the use of the buffer 644 to store data units 605 (or copies thereof), the traffic manager 640 can also include queue management logic configured to assign data units 605 to queues 645 and manage the flow of data units 605 through the queues 645. For example, the traffic manager 640 can“en-queue” a PDU that has been fully buffered by identifying a particular queue 645 to which to assign the PDU, and then linking the PDU identifier or other PDU metadata to the assigned queue. The traffic manager 640 can further determine when to release - also referred to as de-queue - a data unit 605 from a queue 645 by sending instructions to the buffer manager 644 to read / release the data unit 605, and then providing the data read from the buffer 644 to a particular packet processor(s) 650.

[0214] In one embodiment, there can be different queues 645 for different sources or destinations. For example, each port 610 and / or port 690 can have its own set of queues 645. The queue 645 to which an incoming data unit 605 is assigned and linked can be selected, for example, based on forwarding information indicating from which port 690 the data unit 605 should exit. In one embodiment, different egress processors 650 can be associated with each different set of one or more queues 645. In one embodiment, the current processing context of a data unit 605 can be used to select which queue 645 the data unit 605 should be assigned to.

[0215] In one embodiment, there can be additional or alternative different queues 645 for different flows or sets of flows. That is, each identifiable traffic flow or group of traffic flows is assigned its own set of queues 645 to which its data units 605 are respectively assigned. In one embodiment, the different queues 645 can correspond to different traffic classes or quality of service (QoS) levels. There can be additional or alternative different queues 645 for any other suitable distinguishing attribute of data units 605, such as source address, destination address, packet type, etc.

[0216] Although only one packet processor 650 and traffic manager 640 are depicted, the device 600 can include any number of packet processors 650 and traffic managers 640. For example, different ports 610 and / or sets of ports 690 can have their own traffic managers 640 and packet processors 650. As another example, in one embodiment, the traffic manager 640 can be replicated for some or all of the stages of processing data units. For example, the system 600 can include a traffic manager 640 and egress packet processor 650 for an egress stage performed as data units 605 exit the system 600, and / or traffic managers 640 and packet processors 650 for any number of intermediate stages. Thus, data units 605 can pass through any number of traffic managers 640 and / or packet processors 650 before exiting the system 600. In other embodiments, only a single traffic manager 640 is needed. If intermediate processing is desired, the flow of data units 605 can loop back to the traffic manager 640 for buffering and / or queuing after each stage of intermediate processing.

[0217] In one embodiment, the traffic manager 640 is coupled to the output of the ingress packet processor(s) 650 such that data units 605 (or portions thereof) are assigned to the buffers 644 only when initially processed by the ingress packet processor 650. Once in the egress buffers 644, the data units 605 (or portions thereof) can be "released" to one or more egress packet processors 650 for processing.

[0218] In processing data units 605, device 600 can replicate data units 605 one or more times for purposes such as, but not limited to, multicasting, mirroring, debugging, and the like. For example, a single data unit 605 can be replicated to multiple egress queues 645. For example, data unit 605 can be linked to separate queues 645 for each of ports 1, 3, and 6. As another example, data unit 605 can be replicated multiple times (e.g., for different egress processors 650) after it reaches the head of a queue 645. Thus, although certain techniques described herein can refer to original data units 605 received by device 600, it can be appreciated that these techniques will apply equally to copies of data units 605 generated for various purposes. Copies of data units 605 can be partial or complete. Further, there can be actual physical copies of data units 605 in buffers 644, or a single copy of data unit 605 can be linked to multiple queues 645 from a single buffer location at the same time.

[0219] 4.10. Forwarding Logic

[0220] The logic used by device 600 to determine how to process data units 605 - e.g., where to send data units 605 and whether to send data units 605, whether to perform additional processing on data units 605, and the like - is referred to as the forwarding logic of device 600. This forwarding logic is implemented collectively by various components of device 600, as described elsewhere herein. For example, ingress packet processors 650 can be responsible for resolving the destination of data units 605 and determining the set of actions / edits to perform on data units 605, while egress packet processors 650 can perform the edits. Further, there can also be embodiments where ingress packet processors 650 also perform edits.

[0221] According to embodiments, the forwarding logic can be hard-coded and / or configurable. For example, in some cases, the forwarding logic of device 600 or portions thereof can be at least partially hard-coded into one or more ingress processors 650 and / or egress processors 650. As another example, the forwarding logic or elements thereof can also be configurable, in that the logic changes over time in response to analysis of state information collected from various components of device 600 and / or other nodes in the network in which device 600 is located or instructions received from other nodes of device 600.

[0222] In one embodiment, the device 600 generally stores one or more forwarding tables (or equivalent structures) in its memory that map certain data unit attributes or characteristics to actions to be taken for data units 605 having those attributes or characteristics, such as sending the data unit 605 to a selected path, or processing the data unit 605 using a specified internal component. For example, such attributes or characteristics can include a quality of service level specified by or associated with another characteristic of the data unit 605, a flow control group, an ingress port 610 through which the data unit 605 was received, a label or tag in the packet header, a source address, a destination address, a packet type, or any other suitable distinguishing attribute. The forwarding logic can, for example, read such a table, determine one or more ports 690 to which to send the data unit 605 based on the table, and add or associate the data unit 605 with information indicating that downstream logic is to send the data unit along a path that includes a particular traffic manager 640 and / or egress processor 650 coupled to the one or more ports 690.

[0223] According to one embodiment, the forwarding tables describe one or more groups of addresses, such as subnets of IPv4 or IPv6 addresses. Each address is an address of a network device on the network, although a network device can have more than one address. Each group is associated with a potentially different set of one or more actions to perform with respect to data units that resolve to (e.g., are directed to) addresses within the group. Any suitable set of one or more actions can be associated with a group of addresses, including but not limited to forwarding a message to a specified "next hop," duplicating a message, changing a message's destination, discarding a message, performing a debugging or statistical operation, applying a quality of service policy or flow control policy, and so on.

[0224] For purposes of illustration, these tables are described as "forwarding tables," although it should be recognized that the range of actions described by these tables can be much greater than simply forwarding a message somewhere. For example, in one embodiment, the tables can be simple forwarding tables that simply specify a next hop for each group. In other embodiments, the tables can describe one or more complex policies for each group. Furthermore, there can be different types of tables for different purposes. For example, one table can be a simple forwarding table that compares each packet's destination address, while another table can specify a policy to apply to a packet at ingress based on the packet's destination (or source) group, and so on.

[0225] In one embodiment, the forwarding logic can read port state data of the port 610 / 690. The port state data can include, for example, flow control state information describing various traffic flows and associated traffic flow control rules or policies, link state information indicating whether a link is up or down, port utilization information indicating port utilization (e.g., a percentage of utilization, a utilization state, etc.). The forwarding logic can be configured to implement the associated rules or policies associated with the flow(s) to which a given packet belongs.

[0226] As data units 605 are routed through different nodes in a network, the nodes can at times drop, fail to transmit, or fail to receive certain data units 605, resulting in the data units 605 not reaching their intended destinations. The act of dropping a data unit 605 or failing to deliver a data unit 605 is often referred to as "dropping" the data unit. The dropping of data units 605, referred to herein as "dropped" or "packet loss," can occur for a variety of reasons, such as resource limitations, errors, or intentional policies. Different components of the device 600 can make decisions to drop data units 605 for a variety of reasons. For example, the traffic manager 640 can determine to drop data units 605 for reasons such as a buffer 644 being over-utilized, a queue 645 exceeding a particular size, and / or the data unit 605 having a particular characteristic, etc.

[0227] 4.11. Multi-Pipeline Architecture

[0228] In one embodiment, a network device can include multiple pipelines of data unit processing components such as described above. Figure 7 An example of one such network device having multiple packet processing pipelines is shown, in accordance with an embodiment. The network device 700 includes multiple ingress ports 710 and egress ports 790, similar to the ingress ports 610 and egress ports 690 of the device 600. The ingress ports 710 are divided into groups of ports 710a-710n, and each group of ports 710 feeds data units to a different pipeline 702 of processing components. There can be any number of groups of ports 710, and thus any number of corresponding pipelines 702.

[0229] Each pipeline includes an ingress arbiter 720. Each ingress arbiter 720 is coupled to a corresponding group of ingress ports 710 and receives data units from those ports 710. In some aspects, each ingress arbiter 720 can be viewed as an ingress version of the traffic manager 640. The ingress arbiter 720 is responsible for determining when data units are sent to downstream components, particularly to ingress packet processors 730 coupled to the ingress arbiter 720 within the corresponding pipeline 702. According to an embodiment, the ingress arbiter 720 can or can not include its own buffer memory in which to buffer data units waiting for processing.

[0230] In one embodiment, the data units sent by ingress arbitrator 720 are actually subunits of larger parent data units, such as cells, frames, segments, or other TDUs, such as packets or other PDUs. In one embodiment, ingress arbitrator 720 can divide (or ensure are divided) the larger data units into these subunits before sending the data units to corresponding ingress packet processors 730. In one embodiment, packets or other PDUs can arrive at ingress arbitrator 720 as sets of TDUs. For convenience, examples are given herein in which the TDUs are cells and the PDUs are packets, but it will be understood that the cells can actually be any type of subunit and the packets can actually be any larger data unit that includes those subunits.

[0231] Each pipeline 702 also includes an ingress packet processor 730 to which its ingress arbitrator 720 ultimately sends data units. Meanwhile, each ingress packet processor 730 operates in a manner similar to ingress packet processor 650 described above. In particular, ingress packet processor 730 performs forwarding tasks, such as resolving data unit destinations, adding or removing headers, and so on. For example, ingress packet processor can be responsible for generating control information that indicates how a data unit is to be handled by downstream components of pipeline 702, and that control information can be inserted into the data unit or conveyed with the data unit as sideband information.

[0232] Each pipeline 702 also includes an egress traffic manager 740 that operates in a manner similar to traffic manager 640. A common interconnect 738 is coupled at one end to each ingress packet processor 730 and at the other end to each egress traffic manager 740. Interconnect 738 conveys data units to the traffic manager(s) 740 indicated by ingress packet processor 730 (e.g., based on control information), thereby allowing data units to "switch" pipelines 702 in cases where their destination(s) include port 790 that is not on the same pipeline 702 as the ingress port 710 that received the data unit. Ingress packet processor 730 can be coupled to interconnect 738 directly or indirectly via other components such as a merge unit that merges control portions of data units processed by ingress packet processor 730 with data portions of data units that bypass ingress packet processor 730.

[0233] The egress traffic manager 740 of the pipeline then regulates the data units flowing to the egress packet processor 750 of the pipeline in a similar manner as described for the reference traffic manager 640. The egress packet processor 750 processes the data units in a similar manner as described for the egress packet processor 650. The egress packet processor then forwards the processed data units to the egress transmission unit 760 of the pipeline, which is responsible for transmitting the data units out of one or more egress port sets 790 belonging to the pipeline 702. The egress port sets 790 of the pipeline correspond to the ingress port 710 of the pipeline.

[0234] In other embodiments, the egress traffic manager 740 and / or other components can be shared among such pipelines.

[0235] 4.12. Integration with traffic tracing and management

[0236] In one embodiment, the system 300 can be integrated into the system 600 or the system 700. That is, Figure 3 and Figure 6 and / or 7 can be complementary views of the same system. The components 310-350 of the system 300 can be implemented by or directly coupled to the traffic manager 740 or the ingress packet processing block 730, for example. For example, the components 310-350 can process the data units (or at least the control portions of the data units) just before, after, or concurrently with the ingress packet processor 650 or 730, and then generate the overrate policy instructions to be accompanied by the downstream data units, if necessary.

[0237] Meanwhile, different functions of the downstream packet switching logic 360 can be implemented by appropriate downstream components. Different downstream components can be responsible for different action(s) prescribed by the overrate policy. For example, the ingress packet processor 730 can be responsible for taking the reprioritization action, the traffic manager 740 can be responsible for taking the actions required for the differentiated dropping or differentiated congestion notification features, and the egress packet processor 750 can be responsible for taking the overrate flow notification action. The responsible downstream component can observe the flag or other marking information associated with the data unit that indicates the overrate policy manager 350’s decision on which overrate policy feature(s) to enable. The downstream component can then take the appropriate action(s) corresponding to that feature(s) on the data unit.

[0238] In other embodiments, the flow tracing techniques described herein can be practiced in the system 600 without the particular features and details of the system 300. Similarly, the system 300 can be implemented without the particular details of the system 600. The components 310-350 can be integrated into a network device in any other suitable manner.

[0239] 4.13. Other

[0240] The devices 600 and 700 illustrate only a few of many possible arrangements of components in a network device configured to provide the functionality described herein. Other arrangements can include fewer, more, or different components, and the division of labor between the components can vary from arrangement to arrangement. Furthermore, the techniques described herein can be used in various computing contexts other than the network 500, in one embodiment.

[0241] For simplicity, the traffic managers, packet processors, and other components are sometimes described herein as acting on or in relation to data units, when in fact only a portion of the data unit can be transmitted to or otherwise available to the component. For example, a packet processor can be said to process a data unit, when in fact only a data unit control portion is available to the packet processor. In such a context, it should be understood that the information about the data unit that is actually available to the component is a representation of the data unit to the component. Any action described herein as being taken by or on a data unit by a component can occur with respect to the representation of the data unit and not necessarily the complete data unit.

[0242] It should be understood that the actual physical representation of a data unit can change as a result of the processes described herein. For example, as a data unit moves from one component to another within or even between network devices, the data unit can be converted from a physical representation at a particular location in one memory to a signal-based representation and back to a physical representation at a different location in potentially a different memory. Such movement can involve deleting, converting, and / or copying some or all of the data unit any number of times, technically. However, for simplicity, even though the physical representation of the data unit changes, the data unit is logically considered to remain the same data unit as it moves through the device. Similarly, the contents and / or structure of a data unit can change as it is processed, for example by adding or deleting header information, adjusting cell boundaries, or even modifying payload data. However, even after changing its contents and / or structure, the modified data unit is still considered to be the same data unit.

[0243] 5.0. Example Embodiments

[0244] Example embodiments of some embodiments are represented as follows, but are not limited to:

[0245] According to one embodiment, a network device comprises: a plurality of communication interfaces configured to receive and transmit data units; flow tracking logic configured to identify traffic flows to which the data units belong; over-rate flow monitoring logic configured to track a rate at which one or more of the communication interfaces is receiving data belonging to a particular traffic flow, and to mark the particular traffic flow as an over-rate traffic flow when the rate exceeds a threshold rate; over-rate flow policy logic configured to enable an over-rate policy at least with respect to the particular traffic flow; packet switching logic configured to process the data units, including forwarding at least some of the data units to other network devices via the plurality of communication interfaces, the packet switching logic being configured to implement the over-rate policy with respect to data units belonging to the particular traffic flow by processing the data units belonging to the particular traffic flow differently when the particular traffic flow is marked as an over-rate traffic flow than when the particular traffic flow is not marked as an over-rate traffic flow.

[0246] In one embodiment, marking the particular traffic flow as an over-rate traffic flow comprises marking data units belonging to the particular traffic flow as belonging to an over-rate traffic flow; wherein the packet switching logic comprises: one or more ingress packet processors coupled to the over-rate flow monitoring logic and the over-rate flow policy logic and configured to process data units on an ingress of the network device; one or more egress packet processors configured to process data units on an egress of the network device; and one or more traffic managers configured to buffer data units while the data units await processing by the one or more packet processors; wherein the one or more ingress packet processors, the one or more traffic managers, and the one or more egress packet processors are configured to implement different features of the over-rate policy in response to receiving data units marked as belonging to an over-rate traffic flow.

[0247] In one embodiment, the packet switching logic further comprises: forwarding logic configured to determine where to forward data units, and to send the data units to the one or more traffic managers, the forwarding logic including the over-rate flow monitoring logic.

[0248] In one embodiment, the flow tracking logic is further configured to: store counters for a plurality of traffic flows; increment a particular one of the counters in response to a communication interface receiving a data unit belonging to a respective one of the plurality of traffic flows; wherein determining when the rate at which the communication interface is receiving data belonging to the particular traffic flow exceeds the threshold comprises determining that the particular one of the counters corresponding to the particular traffic flow exceeds a threshold count; wherein the over-rate flow monitoring logic is further configured to periodically decrement the counters by a decrement amount based on one or more over-rate thresholds assigned to the corresponding traffic flows.

[0249] In one embodiment, the threshold is a threshold rate at which a particular traffic flow is deemed to be an over-rate traffic flow, wherein the threshold rate is a function of an expected target rate for the particular traffic flow, wherein the threshold count is selected based on the threshold rate, wherein the over-rate traffic monitoring logic is further configured to periodically decrement the particular counter by a particular decrement amount, wherein the particular decrement amount is selected based on the target rate.

[0250] In one embodiment, the flow tracking logic is further configured to store the counters in flow tracking containers, each flow tracking container storing at least a flow identifier of a traffic flow associated with the flow tracking container and a counter for the traffic flow; wherein for each given data unit of the plurality of data units, incrementing the particular counter comprises: deriving a flow tracking identifier from the given data unit; locating the flow tracking container associated with the flow tracking identifier; incrementing the counter in the flow tracking container.

[0251] In one embodiment, the flow tracking logic is further configured to store different sets of flow tracking containers in different storage spaces; wherein locating the flow tracking container associated with the flow tracking identifier comprises performing one or more hash functions on the flow tracking identifier to produce one or more tracking index values, the one or more tracking index values indicating which of the different storage spaces to search for the flow tracking container.

[0252] In one embodiment, the plurality of traffic flows for which counters are stored does not include all traffic flows, the flow tracking logic is further configured to, in response to determining that a first counter is below a particular value, reassign a flow tracking container storing a first counter for a first traffic flow to store a second counter for a second traffic flow that was not previously in the plurality of traffic flows.

[0253] In one embodiment, the plurality of traffic flows for which counters are stored does not include all traffic flows, and wherein the flow tracking logic is further configured to, each time a data unit belonging to a first traffic flow is received, reset a timeout value associated with the first traffic flow; in response to determining that the first traffic flow is idle based on the timeout value, reassign a flow tracking container storing a first counter for the first traffic flow to store a second counter for a second traffic flow that was not previously in the plurality of traffic flows.

[0254] In one embodiment, the over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes the forwarding logic of the packet switch logic to clone one or more of the data units belonging to the particular traffic flow and to forward the cloned one or more of the data units to a collector.

[0255] In one embodiment, the over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes a traffic manager in the packet switching logic to use a different weighted random early detection ("WRED") - explicit congestion notification ("ECN") curve for data units belonging to the particular traffic flow than for data units belonging to a second traffic flow for which the over-rate policy is not enabled and / or not marked as an over-rate traffic flow.

[0256] In one embodiment, the over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes a forwarding logic of the packet switching logic to send data units belonging to the particular traffic flow to a different queue than when the particular traffic flow is not marked as an over-rate traffic flow.

[0257] In one embodiment, the over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes a traffic manager of the packet switching logic to enqueue certain data units belonging to the particular traffic flow in a queue with a different queue offset than a queue to which the data units were originally assigned.

[0258] In one embodiment, the over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes a traffic manager of the packet switching logic to drop data units belonging to the particular traffic flow at a higher rate than when the particular traffic flow is not marked as an over-rate traffic flow.

[0259] According to one embodiment, a method comprises: receiving data units at a network device; processing the data units with packet switching logic of the network device, including forwarding at least some of the data units to other network devices; identifying traffic flows to which the data units belong; determining when a rate at which the network device is receiving data belonging to a particular traffic flow exceeds a threshold rate; enabling an over-rate policy for the particular traffic flow; implementing the over-rate policy for data units belonging to the particular traffic flow by processing the data units differently when the particular traffic flow is marked as an over-rate traffic flow than when the particular traffic flow is not marked as an over-rate traffic flow.

[0260] In one embodiment, the method further comprises identifying the traffic flow to which a given data unit belongs by deriving a flow identifier for the given data unit based on one or more header fields of the given data unit.

[0261] In one embodiment, the one or more header fields include a source IP address, a destination IP address, and a protocol.

[0262] In one embodiment, deriving the flow identifier includes inputting one or more header fields into a hash function, and outputting a hash value from the hash function, the flow identifier being based on the hash value.

[0263] In one embodiment, the data units are TCP / IP packets.

[0264] In one embodiment, data units belonging to traffic flows for which the over- rate policy is not enabled are processed using a default policy different from the over-rate policy.

[0265] In one embodiment, the method further includes storing counters for a plurality of traffic flows, incrementing a particular one of the counters in response to receiving a data unit belonging to a corresponding one of the plurality of traffic flows, wherein determining when the rate at which the network device is receiving data belonging to a particular traffic flow exceeds a threshold includes determining that the particular one of the counters corresponding to the particular traffic flow exceeds a threshold count, decrementing the counters periodically based on one or more over-rate thresholds assigned to the corresponding traffic flows.

[0266] In one embodiment, the method further includes storing the counters in flow tracking containers, each flow tracking container storing at least a flow identifier for a traffic flow associated with the flow tracking container and a counter for the traffic flow, wherein incrementing the particular counter at the time a given data unit is received for each given data unit of the plurality of data units includes deriving a flow tracking identifier from the given data unit, locating a flow tracking container associated with the flow tracking identifier, and incrementing the counter in the flow tracking container.

[0267] In one embodiment, incrementing the counter includes adding to the counter an amount corresponding to a size of the given data unit.

[0268] In one embodiment, each flow tracking container further stores an over-rate policy status value indicating whether the over-rate policy is enabled for the traffic flow associated with the flow tracking container.

[0269] In one embodiment, the method further includes storing different sets of flow tracking containers in different storage spaces, wherein locating a flow tracking container associated with the flow tracking identifier includes performing one or more hash functions on the flow tracking identifier to produce one or more tracking index values, the one or more tracking index values indicating which of the different storage spaces to search for the flow tracking container.

[0270] In one embodiment, the plurality of traffic flows for which counters are stored does not include all of the traffic flows, the method further including:

[0271] In response to determining that the first counter is below a particular value, reallocating the flow tracking container storing the first counter for the first traffic flow to store a second counter for a second traffic flow not previously in the plurality of traffic flows.

[0272] In one embodiment, reallocating the flow tracking container is further in response to determining that the first counter is below the particular value when a second data unit belonging to the second traffic flow is received and no counter for the second traffic flow exists.

[0273] In one embodiment, the particular value is a value of a next lowest counter stored in a set of flow tracking containers within a storage space that meets a condition for storing the second counter, the set of flow tracking containers including the flow tracking container.

[0274] In one embodiment, the first counter is a member of a set of counters whose values are lowest in a set of flow tracking containers within a storage space that meets a condition for storing the second counter, the particular value is a value of a lowest counter stored in a storage space that is not in the set of counters, wherein the first counter is randomly selected from the set of counters for reallocation.

[0275] In one embodiment, the plurality of traffic flows for which counters are stored does not include all traffic flows, and the method further comprises resetting a timeout value associated with the first traffic flow each time a data unit belonging to the first traffic flow is received, reallocating the flow tracking container storing the first counter for the first traffic flow to store a second counter for a second traffic flow not previously in the plurality of traffic flows in response to determining that the first traffic flow is idle based on the timeout value.

[0276] In one embodiment, the method further comprises periodically decrementing the timeout value in a background process, wherein reallocating the flow tracking container comprises deallocating the flow tracking container in response to the timeout value reaching zero, allocating the flow tracking container for the second counter in response to receiving a second data unit belonging to the second traffic flow and determining that no counter is stored for the second traffic flow.

[0277] In one embodiment, determining that the particular flow is idle comprises comparing the timeout value to a current time value.

[0278] In one embodiment, an over-rate policy, when enabled for a particular traffic flow while the particular traffic flow is marked as an over-rate traffic flow, causes the packet switching logic to perform one or more of the following: clone one or more of the data units belonging to the particular traffic flow and forward the one or more cloned data units to a collector; use a different Weighted Random Early Detection ("WRED") - Explicit Congestion Notification ("ECN") curve for data units belonging to the particular traffic flow than for data units of a second traffic flow that is not marked as an over-rate flow; send data units belonging to the particular traffic flow to a different queue than when the particular traffic flow is not marked as an over-rate traffic flow; or drop data units belonging to the particular traffic flow at a higher drop rate than when the particular traffic flow is not marked as an over-rate traffic flow.

[0279] Other example embodiments are described in other sections of this document.

[0280] 6.0. Implementation Mechanism - Hardware Overview

[0281] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices can be desktop computer systems, portable computer systems, handheld devices, network devices, or any other devices that incorporate hardwired and / or program logic to implement these techniques. The special-purpose computing devices can be hardwired to perform these techniques, or can include digital electronic devices, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform these techniques. Such special-purpose computing devices can also combine custom hardwired logic, ASICs, FPGAs, or other

[0282] While certain of the foregoing implementations have been described above in the context of providing a number of advantages in hardware implementations, it will be appreciated that not all of these advantages are required in other implementations. Accordingly, some embodiments described herein are not limited to hardware implementations, but rather can encompass a broader scope of implementations across both hardware and software. Accordingly, whereas an example described herein can include an embodiment that implements functions by way of a hardware assembly configured to perform certain operations, another example can be realized in terms of an arrangement of functions provided for on any general purpose computing device under the control of appropriate software. Similarly, where the specification describes a component of a device as including a hardwired means to perform certain functionality, it will be appreciated that such component can alternatively be embodied in terms of an arrangement of components that are specifically programmed or otherwise configured to perform the same functionality.

[0283] Figure 8 FIG. 8 is a block diagram illustrating an example computer system 800 that can be used to implement the techniques described above, according to one embodiment. The computer system 800 can be, for example, a desktop computing device, a laptop computing device, a tablet computer, a smartphone, a server device, a computing mainframe, a multimedia device, a handheld device, a network device, or any other suitable device. In one embodiment, the computer system 800 includes a processor 802, a memory 804, a storage device 806, an input / output interface 808, and a communication interface 810, which each couple to a bus 812. The bus 812 can be any suitable bus scheme known in the art, including a memory bus or memory chip, a peripheral bus, an external bus, a Figure 8 The foregoing sections constitute different views of the devices and systems described in the previous sections.

[0284] The computer system 800 can include one or more ASICs, FPGAs, or other specialized circuits 803 for implementing the program logic described herein. For example, the circuits 803 can include fixed and / or configurable hardware logic blocks, input / output (I / O) blocks, hardware registers, or other embedded memory resources such as random access memory (RAM) for storing various data, and the like, for implementing some or all of the described techniques. The logic blocks can include, for example, arrangements of logic gates, flip-flops, multiplexers, and the like, configured to generate output signals based on logic operations performed on input signals.

[0285] In addition and / or as an alternative, the computer system 800 can include one or more hardware processors 804 configured to execute software-based instructions. The computer system 800 can also include one or more buses 802 or other communication mechanisms for communicating information. The bus 802 can include various internal and / or external components, including but not limited to internal processor or memory buses, Serial ATA buses, PCI Express buses, Universal Serial Bus, HyperTransport buses, InfiniBand buses, and / or any other suitable wired or wireless communication channels.

[0286] The computer system 800 also includes one or more memories 806, such as RAM, hardware registers, or other dynamic or volatile storage devices, for storing data units to be processed by the one or more ASICs, FPGAs, or other specialized circuits 803. The memory 806 can also be used, or instead, for storing information and instructions to be executed by the processor 804. The memory 806 can be directly connected or embedded within the circuits 803 or the processor 804. Alternatively, the memory 806 can be coupled to the bus 802 and accessed via the bus 802. The memory 806 can also be used to store temporary variables, data units describing rules or policies, or other intermediate information during execution of program logic or instructions.

[0287] The computer system 800 also includes one or more read-only memories (ROMs) 808 or other static storage devices coupled to the bus 802 for storing static information and instructions for the processor 804. One or more storage devices 810, such as solid state drives (SSDs), magnetic disks, optical disks, or other suitable non-volatile storage devices, can optionally be provided and coupled to the bus 802 for storing information and instructions.

[0288] In one embodiment, computer system 800 can also include one or more communication interfaces 818 coupled to bus 802. Communication interfaces 818 provide a one-way or two-way communication coupling to one or more network links that are connected to local network 822. For example, communication interfaces 818 can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, one or more communication interfaces 818 can include a local area network (LAN) card to provide a data communication connection to a compatible LAN. As yet another example, one or more communication interfaces 818 can include a wireless network interface controller, such as an 802.11-based controller, a Bluetooth controller, a long term evolution (LTE) modem, and / or other types of wireless interfaces. In any such implementation, communication interfaces 818 send and receive electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0289] Network link 820 typically provides data communication through one or more networks to other data devices. For example, network link 820 can provide a connection through local network 822 to a host computer 824 or to data equipment operated by an Internet Service Provider (ISP) 826. ISP 826 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 828. Local network 822 and Internet 828 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 820 and through communication interface 818, are example forms of transmission media for these digital data streams.

[0290] In one embodiment, computer system 800 can send and receive data units through the network(s) 820 and communication interface 818. In some embodiments, the data can be data units that computer system 800 is required to process and, if necessary, redirect to other computer systems via appropriate network links 820. In other embodiments, the data can be instructions for implementing various processes related to the described technology. For example, in the example of the Internet, a server 830 can send a request code for an application through the Internet 828, ISP 826, local network 822, and communication interface 818. The received code can be executed by processor 804 as it is received, and / or stored in storage 810 or other non-volatile memory for later execution. As another example, information received via network link 820 can be interpreted and / or processed by software components of computer system 800, such as a web browser, application, or server, which in turn can issue instructions based thereon to processor 804 via an operating system and / or other intermediary layers of software components.

[0291] Computer system 800 can optionally be coupled via bus 802 to one or more displays 812, for presenting information to computer users. For example, computer system 800 can connect to a liquid crystal display (LCD) monitor via a high-definition multimedia interface (HDMI) cable or other suitable cable, and / or to a light-emitting diode (LED) television via a wireless connection such as a peer-to-peer Wi-Fi direct connection. Other examples of suitable types of displays 812 can include, but are not limited to, plasma display devices, projectors, cathode ray tube (CRT) monitors, electronic paper, virtual reality headsets, Braille terminals, and / or any other suitable devices for outputting information to computer users. In one embodiment, any suitable type of output device can be used in place of display 812, such as an audio speaker or a printer.

[0292] One or more input devices 814 are optionally coupled to bus 802 for communicating information and command selections to processor 804. One example of an input device is a keyboard, which includes alphanumeric and other keys. Another type of user input device is cursor control 816, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to processor 804 and for controlling cursor movement on display 812. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify positions in a plane. Other examples of suitable input devices 814 include a touch screen panel that is fixed to display 812, a camera, a microphone, an accelerometer, a motion detector, and / or other sensors. In one embodiment, a network-based input device 814 can be used. In such an embodiment, user input and / or other information or commands can be relayed from the input device 814 to network link 820 on computer system 800 via a router and / or switch on a local area network (LAN) or other suitable shared network, or via a peer-to-peer network.

[0293] As discussed, computer system 800 can implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs 803, firmware and / or program logic which in combination with the computer system causes or programs computer system 800 to be a special-purpose machine. However, according to one embodiment, the techniques herein are performed by computer system 800 in response to processor 804 executing one or more sequences of one or more instructions contained in main memory 806. Such instructions can be read into main memory 806 from another storage medium, such as storage device 810. Execution of the sequences of instructions contained in main memory 806 causes processor 804 to perform the process steps described herein.

[0294] The term "storage media" as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media can comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical disks or magnetic disks, such as storage device 810. Volatile media includes dynamic memory, such as main memory 806. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0295] Storage media is distinct from, but can be used in combination with, transmission media. Transmission media participate in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 802. Transmission media also can take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications.

[0296] Various forms of media can be involved in carrying one or more sequences of instructions to processor 804 for execution. For example, the instructions can initially be borne on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a network, such as the Internet or a cellular network, using a modem. A modem local to computer system 800 can receive the data on the network and demodulate the signal to decode the instructions. An appropriate circuitry can then place the data on bus 802. Bus 802 carries the data to main memory 806, from which processor 804 retrieves and executes the instructions. The instructions received by main memory 806 can optionally be stored on storage device 810 either before or after execution by processor 804.

[0297] 7.0. Extensions and Alternatives

[0298] As used herein, the terms "first," "second," "certain," and "particular" are used as naming conventions to distinguish queries, plans, representations, steps, objects, devices, or other items from one another so that the items can be referred to after they are introduced. The use of these terms does not imply an ordering, timing, or any other characteristic of the referenced items unless otherwise noted herein.

[0299] In the drawings, various components are depicted as communicatively coupled to various other components through arrows. These arrows illustrate only certain examples of information flow between components. The direction of the arrows or the absence of arrow lines between certain components should not be interpreted as indicating whether or not there is communication between certain components themselves. Rather, each component can feature appropriate communication interfaces by which the component can become communicatively coupled to other components as necessary to implement any of the functionality described herein.

[0300] In the foregoing specification, embodiments of the inventive subject matter have been described with reference to a number of specific details that may vary depending on implementation. Thus, to what is the inventive subject matter and intended to be the only and exclusive indicator of what is the inventive subject matter is the set of claims issued from this application, in the form in which such claims are issued, including any subsequent corrections, is the sole and exclusive indicator of what is the inventive subject matter. In this regard, although specific claim from dependencies are recited in the claims of this application, it should be noted that the features of the dependent claims of this application can be combined with the features of other dependent claims as well as the features of the independent claims of this application, as appropriate, and not just according to the specific dependencies recited in the claim set. Moreover, although individual embodiments are discussed herein, any combination of the embodiments discussed herein and / or portions thereof can be combined to form further embodiments.

[0301] Any definitions of terms contained in these claims shall be governed by the express definitions contained in this document, rather than the meanings of such terms that leave open the possibility of delineating the meaning of such terms for the purposes of a determination of equivalency. Accordingly, no limitation, element, property, characteristic, advantage, or the like that is not expressly recited in a claim should in any way limit the scope of such claims. The specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A network device, comprising: Multiple communication interfaces are configured as data receiving units; The flow tracing logic is configured as follows: Identify the service flow to which the data unit belongs; Counters for multiple service flows are stored in a flow tracing container, and each flow tracing container stores at least the flow identifier of the service flow associated with the flow tracing container and the counter for that service flow; In response to a communication interface among the plurality of communication interfaces receiving a data unit belonging to a corresponding service flow among the plurality of service flows, a specific counter among the counters is incremented; Increasing the specific counter upon receiving the given data unit, for each given data unit among a plurality of data units, further includes: Obtain the stream tracing identifier from the given data unit; Locate the stream tracing container associated with the stream tracing identifier, and Increment the counter in the stream tracing container; The high-rate stream monitoring logic is configured as follows: Track the rate at which one or more of the communication interfaces are receiving data belonging to a specific service flow; By determining that a specific counter among the counters corresponding to the specific service flow exceeds a threshold count, the specific service flow is marked as an excessively high-rate service flow when the rate exceeds the threshold rate; and Based on one or more excessive rate thresholds assigned to the corresponding service flow, the counter is periodically decremented by the amount of decrement. The excessively high-rate flow policy logic is configured to enable the excessively high-rate policy at least for the specific service flow. A packet switching logic is configured to process the data units, including forwarding at least some of the data units to other network devices via multiple output communication interfaces. The packet switching logic is configured to implement the high-rate policy for data units belonging to a specific service flow when the specific service flow is marked as an excessively high-rate service flow, by processing the data units belonging to the specific service flow differently than when the specific service flow is not marked as an excessively high-rate service flow.

2. The apparatus according to claim 1, The specific service flow marked as an excessively high-rate service flow includes: Data units belonging to the specific service flow are marked as belonging to an excessively high-rate service flow; The packet switching logic mentioned above includes: One or more ingress packet processors are coupled to the over-rate flow monitoring logic and the over-rate flow policy logic, and are configured to process the data units on the ingress of the network device; One or more egress packet processors are configured to process the data units from the egress of the network device; and One or more service managers are configured to buffer the data unit while it awaits processing by the one or more egress packet processors; The one or more ingress packet processors, one or more service managers, and one or more egress packet processors are configured to implement different features of the over-rate policy in response to receiving data units marked as belonging to an over-rate service flow.

3. The apparatus according to claim 1, wherein the packet switching logic further comprises: Forwarding logic is configured to determine where to forward the data unit and to send the data unit to the one or more service managers, the forwarding logic including the high-rate flow monitoring logic.

4. The apparatus of claim 1, wherein the flow tracing logic is further configured to: Different sets of the stream tracing containers are stored in different storage spaces; The location of the stream tracing container associated with the stream tracing identifier includes: One or more hash functions are performed on the stream tracing identifier to generate one or more tracing index values, which indicate which storage spaces among the different storage spaces are searched for the stream tracing container.

5. The apparatus of claim 1, wherein the plurality of service flows does not include all service flows in the service flows, the counter is stored for the plurality of service flows, and the flow tracing logic is further configured to: In response to determining that the first counter is below a certain value, the flow tracing container storing the first counter for the first service flow is reallocated to store the second counter for the second service flow that was not previously in the plurality of service flows.

6. The apparatus of claim 1, wherein the plurality of service flows does not include all service flows in the service flows, the counter is stored for the plurality of service flows, and wherein the flow tracing logic is further configured to: Whenever a data unit belonging to the first service flow is received, the timeout value associated with the first service flow is reset; In response to determining that the first service flow is idle based on the timeout value, a flow tracing container storing the first counter for the first service flow is reallocated to store the second counter for the second service flow that was not previously in the plurality of service flows.

7. The apparatus of claim 1, wherein when the high-rate policy is enabled for a particular service flow at the same time the particular service flow is marked as a high-rate service flow, the forwarding logic of the packet switching logic clones one or more data units belonging to the particular service flow, and forwards the one or more cloned data units to the collector.

8. The apparatus of claim 1, wherein when the over-rate policy is enabled for a particular service flow at the same time the particular service flow is marked as an over-rate service flow, the service manager in the packet switching logic causes the service manager to use a different weighted random early detection ("WRED")-explicit congestion notification ("ECN") curve for the data units belonging to the particular service flow than for data units belonging to a second service flow where the over-rate policy is not enabled and / or not marked as an over-rate service flow.

9. The apparatus of claim 1, wherein when the high-rate policy is enabled for a specific service flow at the same time the specific service flow is marked as a high-rate service flow, the forwarding logic of the packet switching logic causes the data unit belonging to the specific service flow to be sent to a different queue than when the specific service flow is not marked as a high-rate service flow.

10. The apparatus of claim 1, wherein the excessive rate policy, when enabled for a specific service flow at the same time the specific service flow is marked as an excessive rate service flow, causes the service manager of the packet switching logic to discard data units belonging to the specific service flow at a higher discard rate than when the specific service flow is not marked as an excessive rate service flow.

11. A computer-implemented method, comprising: Data units are received at network devices via multiple communication interfaces; Identify the service flow to which the data unit belongs; Counters for multiple service flows are stored in a flow tracing container, and each flow tracing container stores at least the flow identifier of the service flow associated with the flow tracing container and the counter for that service flow; In response to a communication interface among the plurality of communication interfaces receiving a data unit belonging to a corresponding service flow among the plurality of service flows, a specific counter among the counters is incremented; Specifically, for each given data unit among a plurality of data units, incrementing the specific counter upon receiving the given data unit includes: Obtain the stream tracing identifier from the given data unit; Locate the stream tracing container associated with the stream tracing identifier, and Increment the counter in the stream tracing container; Track the rate at which one or more of the communication interfaces are receiving data belonging to a specific service flow; The specific service flow is marked as an excessively high-rate service flow when the rate exceeds the threshold rate, by determining that a specific counter in the counters corresponding to the specific service flow exceeds the threshold count; The counter is periodically decremented based on one or more excessive rate thresholds assigned to the corresponding service flow. Enable the excessively high rate policy for the specific service flow; At least some of the data units in the data unit are forwarded to other network devices via multiple output communication interfaces; When a specific service flow is marked as an excessively high-rate service flow, the excessively high-rate strategy for the data units belonging to the specific service flow is implemented by processing the data units belonging to the specific service flow differently than when the specific service flow is not marked as an excessively high-rate service flow.

12. The method of claim 11, further comprising: Marking the specific service flow as an excessively high-rate service flow includes: marking data units belonging to the specific service flow as belonging to the excessively high-rate service flow; One or more ingress packet processors are used to process the data units at the ingress points of the network device; One or more egress packet processors are used to process the data units from the egress of the network device; While the data unit is waiting to be processed by the one or more egress packet processors using one or more service managers, the data unit is buffered. The one or more ingress packet processors, one or more service managers, and one or more egress packet processors are configured to implement different features of the over-rate policy in response to receiving data units marked as belonging to an over-rate service flow.

13. The method of claim 11, further comprising: Determine where to forward the data unit, and send the data unit to the one or more business managers.

14. The method of claim 11, further comprising: Different sets of the stream tracing containers are stored in different storage spaces; Locating the flow tracing container associated with the flow tracing identifier includes: performing one or more hash functions on the flow tracing identifier to generate one or more tracing index values, the one or more tracing index values ​​indicating which storage spaces among the different storage spaces are searched for the flow tracing container.

15. The method of claim 11, wherein the plurality of service flows does not include all service flows in the service flows, the counter is stored for the plurality of service flows, and the method further comprises: In response to determining that the first counter is below a certain value, the flow tracing container storing the first counter for the first service flow is reallocated to store the second counter for the second service flow that was not previously in the plurality of service flows.

16. The method of claim 11, wherein the plurality of service flows does not include all service flows in the service flows, the counter is stored for the plurality of service flows, and the method further comprises: Whenever a data unit belonging to the first service flow is received, the timeout value associated with the first service flow is reset; In response to determining that the first service flow is idle based on the timeout value, a flow tracing container storing the first counter for the first service flow is reallocated to store the second counter for the second service flow that was not previously in the plurality of service flows.

17. A non-transient computer-readable medium storing one or more instructions, which, when executed by one or more computing devices, cause the method according to any one of claims 11-16 to be performed.

Images