Load-aware packet size distribution measurement in network devices

Abstract

The network device determines a load metric corresponding to the processing load of the network device. In response to determining that the load metric meets a threshold, the network device begins measuring distribution information regarding the distribution of packet sizes processed by the network device, and then terminates the measurement of distribution information. The network device then uses the distribution information to control the network device.

Description

Technical Field

[0001] The present disclosure generally relates to communication networks, and more particularly to power saving techniques for use within network devices.

Background Art

[0002] The techniques described in this section are techniques that could be pursued, but are not necessarily techniques that have been previously devised or pursued. Thus, unless otherwise indicated, none of the techniques described in this section should be assumed to be eligible as prior art merely by virtue of their inclusion in this section.

[0003] A computer network is a set of computing components interconnected by communication links. Each computing component can be a separate computing device such as, but not limited to, a hub, network switch, bridge, router, server, gateway, or personal computer, or components thereof. Each computing component or “network device” is considered a node within the network. A communication link is a mechanism that connects at least two nodes such that each node can transmit data to and receive data from other nodes. Such data can be transmitted in the form of signals via a transmission medium such as, but not limited to, an electrical cable, an optical cable, or a wireless medium.

[0004] The structure and transmission of data between nodes are governed by numerous different protocols. Multiple protocol layers may exist, typically starting from the lowest layer, such as the “physical” layer, which regulates the transmission and reception of raw bitstreams as signals over the transmission medium. Each layer defines data units (protocol data units, or “PDUs”), and multiple data units in one layer combine to form a single data unit in another layer. Further examples of layers may include, for example, the data link layer, where bits defined by the physical layer combine to form frames or cells; the network layer, where frames or cells defined by the data link layer combine to form packets; and the transport layer, where packets defined by the network layer combine to form Transmission Control Protocol (TCP) segments or User Datagram Protocol (UDP) datagrams. The Open Systems Interconnection (OSI) model of communication describes these and other layers of communication. However, other models may also be used that define other ways of layering information. The Internet Protocol (IP) suite, or "TCP / IP stack," is an example of a common group of protocols that can be used together across multiple layers to communicate information. However, the techniques described herein may have applications to other protocols outside of the TCP / IP stack.

[0005] A given node in a network may not necessarily have links to every other node in the network, especially in more complex networks. For example, in a wired network, each node may have only a limited number of physical ports to which cables can be connected to create links. A particular “terminal” node, often a server or end-user device, may have only one or a few ports. Other nodes, such as switches, hubs, or routers, may have many more ports and are typically used to relay information between terminal nodes. The arrangement of nodes and links in a network is called the network topology and is typically visualized as a network graph or tree.

[0006] A given node in a network may communicate with another node in the network by transmitting data units along one or more different “paths” through the network to other nodes, each path including any number of intermediate nodes. Transmission of data across a computing network typically involves transmitting units of data, such as packets, cells, or frames, along paths through intermediate networking devices, such as switches or routers, which either directly or redirect each data unit to its corresponding destination.

[0007] The period while a data unit is passing through an intermediate networking device is conceptualized as a "visit" or "hop," during which the device may perform a variety of actions or processing steps on the data unit. The exact set of actions taken depends on various characteristics of the data unit, such as metadata found in the data unit's header, and often on the context or state of the network device. For example, address information specified by or associated with the data unit, such as source address, destination address, virtual local area network (VLAN) identifier, and path information, is typically used to determine how to process the data unit (i.e., what actions to take on the data unit). For example, an IP data packet may include a destination IP address field in its header, which a network router may use to determine one or more other networking devices from a number of possible other networking devices to which the IP data packet should be forwarded.

[0008] Different types of data units tend to have different packet sizes. For example, control packets (e.g., packets for setting up a connection, packets for disconnecting a connection, acknowledgment packets, etc.) tend to be relatively small, less than 100 bytes, while data packets tend to be significantly larger than control packets, often exceeding 1000 bytes. As another example, data packets containing video data are typically larger than 1000 bytes, while data packets containing audio data are typically smaller, for example, several hundred bytes. Typically, when a connection is being set up or disconnected, there tends to be a higher proportion of packets with relatively small packet sizes, while when a connection is up and running, there tends to be a lower proportion of packets with relatively small packet sizes.

[0009] As briefly discussed above, network devices analyze the header of a data unit (e.g., a packet) to determine how to process it. For example, network devices with multiple ports connected to multiple network links, such as network switches, bridges, routers, and gateways, analyze the header of a received data unit to determine which one or more ports the data unit should be transmitted through. At a given data rate, the processing load on a network device is higher for smaller data units than for larger data units because the rate at which the network device receives packet headers is higher for smaller data units than for larger data units (for a given data rate). In other words, assuming the same data rate, the processing load on a network device varies depending on the relative amount of small and large packets. [Overview of the project]

[0010] In one embodiment, a method for controlling the operation of a network device comprises the steps of: determining a load metric corresponding to the processing load of the network device; in response to determining that the load metric satisfies a first threshold, starting to measure distribution information relating to the distribution of packet sizes processed by the network device; ending the measurement of distribution information relating to the distribution of packet sizes processed by the network device; and controlling the network device using the distribution information.

[0011] In another embodiment, the network device comprises: a plurality of network interfaces; a packet processor configured to process data units received through the plurality of network interfaces and to determine which of the plurality of network interfaces transmits the data units; a first circuit configured to determine a load metric corresponding to the processing load of the network device; a second circuit configured to start measuring distribution information relating to the distribution of packet sizes processed by the network device and to end measuring distribution information relating to the distribution of packet sizes processed by the network device in response to determining that the load metric satisfies a first threshold; and a controller configured to control the network device using the distribution information. [Brief explanation of the drawing]

[0012] [Figure 1] This is a simplified diagram of an exemplary networking system in which the load-aware packet size distribution (PSD) information generation technology described herein is implemented according to one embodiment.

[0013] [Figure 2A] This is a simplified diagram of an exemplary network device that utilizes PSD information generation technology according to one embodiment.

[0014] [Figure 2B] This is another simplified diagram of the exemplary network device shown in Figure 2A, according to one embodiment.

[0015] [Figure 3] This is a simplified block diagram of an exemplary PSD information generation circuit according to one embodiment.

[0016] [Figure 4]This graph shows a descriptive example of PSD information measured at different processing load levels of a network device according to one embodiment.

[0017] [Figure 5] This is a simplified illustrative state diagram of a circuit that controls the generation of PSD information in a network device according to one embodiment.

[0018] [Figure 6] This is another simplified illustrative state diagram of a circuit that controls the generation of PSD information in a network device, according to another embodiment.

[0019] [Figure 7] This is a simplified flowchart of an exemplary method for controlling a network device based on PSD measurement, according to one embodiment. [Modes for carrying out the invention]

[0020] In the following description, numerous specific details are given for illustrative purposes to provide a full understanding of the subject matter of the present invention. However, it is clear that the subject matter of the present invention can be implemented without these specific details. In other examples, known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the subject matter of the present invention.

[0021] The power consumption of a network device is typically most important when the network device (or a portion thereof) is fully loaded. During periods when the network device is under high load, the packet processor and data path of the network device are under great stress. Often, the proportion of small-sized packets is large during periods of low load and small during periods of high load. However, current network devices do not distinguish between collecting packet size distribution information during periods of high load and low load. As a result, the packet size distribution information generated by current network devices is typically measured over both high-load and low-load periods and, thus, often does not accurately represent the packet size distribution during high load. For this reason, the packet size distribution information generated by current network devices has low utility for reducing the power consumption of the network device during high-load periods.

[0022] Techniques, technologies, and mechanisms are disclosed for measuring distribution information (which may be referred to herein as "packet size distribution information") regarding the size of packets processed by a network device in response to the processing load of the network device meeting a condition. Such packet size distribution information is useful in controlling the network device to more optimally adjust the operation of the network device in at least some embodiments. As an illustrative example, if the packet size distribution information indicates that the processing load is high and the proportion of packets having a small packet size is relatively high, the network device may well identify packet flows that contribute to a large number of packets having a small packet size and adjust the processing of those flows to redistribute processing and / or memory resources within the packet flows and / or reduce the power consumption of the network device.

[0023] FIG. 1 is a simplified diagram of an exemplary networking system 100, also referred to as a network, in which the power-saving technology described herein is implemented according to one embodiment. The networking system 100 includes a plurality of interconnected nodes 110a - 110n (collectively nodes 110), and each node is implemented by a different computing device. For example, a node 110 may be a single network computing device such as a router or a switch, where some or all of the processing components described herein are implemented in an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other integrated circuits. As another example, a node 110 may include one or more memories storing machine-readable instructions for implementing the various components described herein, one or more hardware processors configured to execute the instructions stored in the one or more memories, and various data repositories within the one or more memories for storing data structures utilized and manipulated by the various components.

[0024] Each node 110 is connected to one or more other nodes 110 within the network 100 by one or more communication links illustrated as lines between the nodes 110. The communication links may be any suitable wired cable or wireless link. It should be noted that system 100 shows only one of many possible arrangements of nodes within the network. Other networks may include fewer or additional nodes 110 having any number of links therebetween.

[0025] Each node 110 may or may not have a variety of other functions, but in one embodiment, each node 110 is configured to transmit, receive, and / or relay data to one or more other nodes 110 via a communication link. Generally, data is communicated as a series of individual units or structures of data represented by signals transmitted over the communication link.

[0026] Different nodes 110 within the network 100 may transmit, receive, and / or relay data units at different communication levels or layers. For example, a first node 110 may transmit a data unit at the network layer (e.g., a TCP segment) to a second node 110 over a path that includes an intermediate node 110. The data unit may be divided into smaller data units ("subunits") at various lower levels before being transmitted from the first node 110. For example, the data unit may be divided into packets, then into cells, and finally sent to the intermediate device as a set of signal-encoded bits. Depending on the network type and / or the device type of the intermediate node 110, the intermediate node 110 may reconstruct the entire original data unit before routing the information to the second node 110, or the intermediate node 110 may simply reconstruct subunits (e.g., packets or frames) without ever constructing the entire original data unit and route those subunits to the second node 110.

[0027] When node 110 receives a data unit, it typically examines the address information (and / or other information) within the data unit to determine how to process it. The address information may include, for example, a media access control (MAC) address, an IP address, a VLAN identifier, information within a multi-protocol label switching (MPLS) label, or any other suitable information. If the address information indicates that the receiving node 110 is not the destination of the data unit, the node may look up forwarding information in its forwarding database and, based on that forwarding information, forward the data unit to one or more other nodes 110 connected to the receiving node 110. The forwarding information may indicate, for example, the outgoing port through which the data unit should be transmitted, a header to be added to the data unit, a new destination address to overwrite the data unit, and so on. If multiple paths are possible to the destination node 110, the forwarding information may include information indicating a preferred method for selecting one of those paths, or a path considered to be the best path may already be defined.

[0028] Address information, flags, labels, and other metadata used to determine how to process a data unit are typically embedded within a portion of the data unit known as a header. One or more headers are typically at the beginning of a data unit, followed by the data unit's payload. For example, a first data unit having a first header corresponding to a first communication protocol may be encapsulated within a second data unit by adding at least a second header corresponding to a second communication protocol to the first data unit. For example, in some embodiments, the second communication protocol is subordinate to the first communication protocol in the protocol stack.

[0029] According to some embodiments, a header has a structure defined by the communication protocol and includes different types of fields, such as a destination address field, a source address field, a destination port field, and a source port field. In some protocols, the number and arrangement of fields are fixed. Other protocols allow for a variable number of fields and / or fields of variable length, with some or all of these fields preceded by type information indicating the meaning of the field to the node, and / or length information indicating the length of the field. In some embodiments, the communication protocol defines a header having multiple different formats, where one or more values ​​in each of the one or more fields within the header indicate the format of the header to the node. For example, the header may include a type field, a version field, etc., indicating which of the multiple formats the header conforms to.

[0030] Different communication protocols typically define their own headers, each with its own format.

[0031] For convenience, data units may be referred to as “packets” in this specification, a term commonly used to refer to data units defined by IP. However, the methods, techniques, and mechanisms described herein are applicable to data units defined by preferred communication protocols other than IP. Therefore, unless otherwise explicitly stated or evident, the term “packet” as used herein should be understood to refer to any type of data structure communicated over a network, including, in addition to packets, segments, cells, data frames, datagrams, and the like.

[0032] Any node in the illustrated network 100 can communicate with any other node in network 100 by sending packets through a series of nodes 110 and links called a path. For example, node B (110b) may send a packet to node H (110h) via a path from node B to node D, node E, and then to node H. There may be many valid paths between the two nodes. For example, another path from node B to node H may go from node B to node D, node G, and then to node H.

[0033] In one embodiment, node 110 does not actually need to specify the complete path for the packets it transmits. Rather, node 110 may be configured to simply calculate the best path for sending packets out of the device (e.g., which of one or more egress ports the packets to be transmitted should be sent through). According to some embodiments, if node 110 receives a packet that is not directly addressed to node 110, node 110 will forward the packet along either the destination node 110 or a "next-hop" node 110 that node 110 has calculated to be in a more suitable location for relaying the packet to the destination node 110, based on header information associated with the packet, such as path and / or destination information. Thus, according to some embodiments, the actual path of the packet is the product of each node 110 along the path through which routing decisions are made about how best to move the packet to the destination node 110 identified by the packet.

[0034] As data units are routed through different nodes in a network, nodes may, in some cases, discard the data unit, fail to send it, or fail to receive it, resulting in the data unit failing to reach its intended destination. The act of discarding a data unit, or the failure to deliver a data unit, is typically referred to as a data unit “drop.” Examples of data unit drops, referred to herein as “drops” or “packet losses,” can occur for a variety of reasons, including resource limitations, errors, or intentional policies.

[0035] One or more of the nodes 110 utilize load-aware packet size distribution (PSD) measurement techniques, examples of which are described below. For example, Figure 1 shows nodes 110d and 110g having load-aware PSD measurement modules, which utilize PSD measurement techniques such as those described below, which include initiating PSD measurement in response to the processing load of node 110 meeting certain conditions.

[0036] Figure 2A is a simplified diagram of an exemplary network device 200 in which load-aware PSD measurement technology is utilized according to one embodiment. The network device 200 is a computing device comprising any combination of i) hardware and / or ii) one or more processors that execute machine-readable instructions, and is configured to implement various logical components described herein.

[0037] In some embodiments, nodes 110d and 110g in Figure 1 have the same or similar structure as network device 200. In another embodiment, network device 200 may be one of several components within node 110. For example, network device 200 may be implemented on one or more integrated circuits or "chips" configured to perform switching and / or routing functions within node 110, such as a network switch or router. Node 110 may further include one or more other components, such as one or more central processing units, storage units, memory, physical interfaces, LED displays, or other components outside of one or more chips, some or all of which may communicate with one or more chips. In some such embodiments, node 110 includes multiple network devices 200.

[0038] In other embodiments, the network device 200 is used in a preferred networking system different from the exemplary networking system 100 shown in Figure 1.

[0039] The network device 200 includes a plurality of packet processing modules 204, each packet processing module being associated with a plurality of ingress network interfaces 208 (which may be referred to herein as “ingress ports” for simplicity) and a plurality of egress network interfaces 212 (which may be referred to herein as “egress ports” for simplicity). The ingress ports 208 are ports through which packets are received via a communication link in the communication network, and the egress ports 212 are ports through which at least a number of packets are transmitted via the communication link after being processed by the network device 200.

[0040] The term "packet" is used to describe data units processed by the network device 200, but data units may be packets, cells, frames, or other preferred structures. For example, in some embodiments, the individual atomic data units on which the illustrated components operate are cells or frames. That is, in some such embodiments, data units are received, processed, and transmitted at the cell or frame level. In some embodiments, these cells or frames are logically linked to each other as packets to which each belongs, for the purpose of determining how to process the cell or frame. However, in some embodiments, particularly when the cell or frame is forwarded through the device 200 to another destination, the cell or frame is not actually assembled into a packet within the device 200.

[0041] Although the ingress port 208 and egress port 212 are illustrated as separate ports for illustrative purposes, they typically correspond to the same physical network interface of the network device 200. That is, in some embodiments, a single network interface serves as both an ingress port 208 and an egress port 212. Nevertheless, for various functional purposes, the specific logic of the network device 200 may treat a single physical network interface as logically separate ingress ports 208 and egress ports 212. Furthermore, for various functional purposes, the specific logic of the network device 200 may subdivide a single physical network interface into multiple ingress ports 208 or egress ports 212 (e.g., "virtual ports"), or aggregate multiple physical network interfaces into a single ingress port 208 or egress port 212 (e.g., a trunk, a link aggregate group (LAG), an equal cost multipath (ECMP) group, etc.). Therefore, in various embodiments, the ingress port 208 and the egress port 212 are considered not merely as separate physical structures, but as separate logical structures mapped to a physical network interface.

[0042] In some embodiments, at least some ports 208 / 212 are coupled to one or more transceivers (not shown in Figure 2A), such as serializer / deserializer ("SerDes") blocks. For example, an ingress port 208 provides a serial input of a received data unit to a SerDes block, which then outputs the data unit in parallel to a packet processing module 204. On the other side, the packet processing module 204 provides a data unit in parallel to another SerDes block, which then outputs the data unit serially to an egress port 212. Depending on the specific embodiment, there may be any number of input and output SerDes blocks of any preferred size (e.g., four groups of 4 x 25 gigabit blocks, eight groups of 4 x 100 gigabit blocks, etc.).

[0043] Each packet processing module 204 includes an ingress portion 204-xa and an egress portion 204-xb. The ingress portion 204-xa generally performs ingress processing operations on packets, such as packet classification, tunnel termination, Layer 2 (L2) forwarding lookups, Layer 3 (L3) forwarding lookups, and any preferred combination of two or more of these.

[0044] The egress portion 204-xb generally performs egress processing operations on packets, such as one, two, or any preferred combination of the following: packet duplication (e.g., for multicast packets), header modification, rate limiting, traffic shaping, egress policing, flow control, and maintaining statistics about packets.

[0045] Each ingress portion 204-xa is communicably coupled to a plurality of egress portions 204-xb via an interconnect 216. Similarly, each egress portion 204-xb is communicably coupled to a plurality of ingress portions 204-xa via an interconnect 216. The interconnect 216 includes, according to various embodiments, one or more switching fabrics, one or more crossbars, and the like.

[0046] During operation, the ingress portion 204-xa receives a packet via its associated ingress port 208 and performs an ingress processing operation on the packet, which includes determining one or more egress ports 212 (which may be referred to herein as “target ports”) through which the packet should be transmitted. The ingress portion 204-xa then forwards the packet via the interconnect 216 to one or more egress portions 204-xb corresponding to the determined one or more target ports 212. Each egress portion 204-xb that receives a packet performs an egress processing operation on the packet and then forwards the packet to one or more determined target ports 212 associated with that egress portion 204-xb so that it may be transmitted from the network device 200.

[0047] In some embodiments, an ingress portion 204-xa determines a virtual target port, and one or more egress portions 204-xb corresponding to the virtual target port map the virtual target portion to one or more physical egress ports 212. In some embodiments, an ingress portion 204-xa determines a group of target ports 212 (e.g., a trunk, LAG, ECMP group), and one or more egress portions 204-xb corresponding to the group of target ports select one or more specific target egress ports 212 within the group of target ports. In this disclosure, the term “target port” means a physical port, a virtual port, a group of target ports, etc., unless otherwise stated or evident.

[0048] According to various embodiments, each packet processing module 204 is implemented using any preferred combination of fixed circuits and / or processors that execute machine-readable instructions, such as one or more FPGAs, specific logic components implemented by ASICs, or one or more processors that execute machine-readable instructions.

[0049] In some embodiments, at least each portion of the multiple packet processing modules 204 is implemented on a single IC (or "chip"). In some embodiments, each portion of the multiple packet processing modules 204 is implemented on a different chip.

[0050] In one embodiment, the components of each ingress portion 204-xa are arranged in a pipeline such that the output of one or more components is provided as input to one or more other components. In some embodiments where the components are arranged in a pipeline, one or more components of the ingress portion 204-xa are skipped or bypassed for a particular packet. In other embodiments, the components are arranged in a preferred manner that is not pipelined. In some embodiments, the exact set and / or sequence of components that process a given packet may vary depending on the packet attributes and / or the state of the network device 200.

[0051] Similarly, in one embodiment, the components of each egress portion 204-xb are arranged in a pipeline such that the output of one or more components is provided as input to one or more other components. In some embodiments where the components are arranged in a pipeline, one or more components of the egress portion 204-xb are skipped or bypassed for a particular packet. In other embodiments, the components are arranged in a preferred manner that is not pipelined. In some embodiments, the exact set and / or sequence of components that process a given packet may vary depending on the packet attributes and / or the state of the network device 200.

[0052] Each ingress section 204-xa includes a circuit 220 (which may be referred to herein as the “Ingress Arbitration Circuit”) configured to reduce traffic loss during bursty traffic and / or other periods of congestion. In some embodiments, the Ingress Arbitration Circuit 220 is configured to function in such a manner as to facilitate the economics of the size, number, and / or quality of these components within the packet processing module 204 by more intelligently controlling the release of data units to downstream components. In some embodiments, the Ingress Arbitration Circuit 220 is further configured to support features such as lossless protocols and cut-through switching, while still enabling high-rate bursts from port 208.

[0053] The ingress arbitration circuit 220 is coupled to an ingress buffer memory 224, which is configured to temporarily store packets received via port 208 while the components of the packet processing module 204 process the packets.

[0054] Each data unit received by the ingress portion 204-xa is stored in one or more entries in one or more buffers, and these entries are marked as available to prevent data units already buffered in buffer memory 224 from being overwritten by newly received data units. In some embodiments, after a data unit is released to the egress portion 204-xb, the one or more entries in the ingress buffer memory 224 in which the data unit was buffered are then marked as available to store newly received data units.

[0055] Each buffer may be a portion of any preferred type of memory, including volatile memory and / or non-volatile memory. In one embodiment, the ingress buffer memory 224 includes a single-port memory that supports only one input / output (I / O) operation per clock cycle (i.e., either a single read operation or a single write operation). Single-port memory is used for higher operating frequencies, but in other embodiments, multi-port memory is used instead. In one embodiment, the ingress buffer memory 224 includes multiple physical memories that can be accessed simultaneously in the same clock cycle, although full realization of this capability is not required. In one embodiment, each buffer is a separate memory bank or set of memory banks. In yet another embodiment, different buffers are different regions within a single memory bank. In one embodiment, each buffer includes a number of addressable “slots” or “entries” (e.g., rows, columns, etc.) that may store data units or portions thereof.

[0056] Generally, the buffers within the ingress buffer memory 224 include a variety of buffers or sets of buffers, each used for various purposes and / or for various components within the ingress portion 204-xa.

[0057] The ingress portion 204-xa includes a buffer manager (not shown) configured to manage the use of the ingress buffer 224. In various embodiments, the buffer manager performs, for example, one or any preferred combination of the following: allocating and deallocating specific segments of memory for buffers; creating and deleting buffers in said memory; identifying available buffer entries for storing data units; maintaining a mapping of those buffer entries to the data units stored in them (e.g., by the packet sequence number assigned to each packet when the first data unit of each packet is received); marking a buffer entry as available when a data unit stored in the buffer is dropped, sent, or released from the buffer; determining when a data unit should be dropped due to its inability to be stored in the buffer; and performing garbage collection of buffer entries for data units (or portions thereof) that are no longer needed.

[0058] According to one embodiment, the buffer manager includes buffer allocation logic (not shown) configured to identify a buffer from among a plurality of buffers in the ingress buffer memory 224 that should be used to store a given data unit or portion thereof. In some embodiments, each packet is stored in a single entry in its allocated buffer. In yet another embodiment, the packet is received as a constituent data unit such as a fixed-size cell or frame, or is divided into these, and the constituent data units are stored separately (e.g., not in the same location or even in the same buffer).

[0059] The ingress arbitration circuit 220 is also configured, according to some embodiments, to maintain an ingress queue 228 used to manage the order in which data units are processed from buffers in the ingress buffer memory 224. Each data unit or the buffer location in which that data unit is stored is said to belong to one or more structures called a queue. Typically, a queue is a set of memory locations (e.g., in the ingress buffer memory 224) arranged in some order by metadata describing the queue. Memory locations may (and often are) be discontinuous relative to their address scheme and / or physical or logical arrangement.

[0060] In some embodiments, the sequence of configuration data units placed in the queue generally corresponds to the order in which data units or portions of data units in the queue are released and processed. Such queues are known as first-in-first-out (FIFO) queues, but other types of queues may be used in other embodiments.

[0061] According to various embodiments, the ingress portion 204-xa also includes an ingress packet processor 232, which is configured to perform one, two, or more preferred combinations of ingress processing operations on packets, such as packet classification, tunnel termination, L2 forwarding lookup, L3 forwarding lookup, etc. For example, the ingress packet processor 232 includes an L2 forwarding database and / or an L3 forwarding database, and the ingress packet processor 232 performs an L2 forwarding lookup and / or an L3 forwarding lookup to determine the target port of the packet. In some embodiments, the ingress packet processor 232 uses header information in the packet to perform an L2 forwarding lookup and / or an L3 forwarding lookup.

[0062] The ingress arbitration circuit 220 is configured to release a certain number of data units (or portions of data units) from the ingress rescue queue 228 for processing (e.g., by the ingress packet processor 232) or transfer (e.g., via the interconnect 216) during each clock cycle or other defined period. The next data unit (or portion of data unit) to be released may be identified using one or more ingress rescue queues 228. In some embodiments, for example, each ingress port 208 (or each group of ingress ports 208) is assigned to its respective ingress rescue queue 228, and the ingress arbitration circuit 220 selects the queue 228 to which one or more data units (or portions of data units) should be released according to a selection scheme such as a round-robin scheme or another preferred selection scheme. Furthermore, if the ingress rescue queue 228 is a FIFO queue, in some embodiments, the ingress arbitration circuit 220 selects from the beginning of the FIFO ingress rescue queue 228 the data unit (or part of a data unit) that has been in the FIFO ingress rescue queue 228 for the longest time.

[0063] Generally, when an ingress portion 204-xa finishes processing a packet, the packet is forwarded to one or more egress portions 204-xb via the interconnect 216. According to one embodiment, the forwarding of a data unit from an ingress portion 204-xa to an egress portion 204-xb includes releasing (or dequeuing) the data unit and forwarding the data unit to the egress portion 204-xb via the interconnect 216.

[0064] The egress section 204-xb includes a circuit 248 (which may be referred to herein as the “traffic manager circuit 248”), which is configured to control the flow of data units from the ingress section 204-xa to one or more other components of the egress section 204-xb. The egress section 204-xb is coupled to an egress buffer memory 252 configured to store egress buffers. A buffer manager (not shown) within the traffic manager circuit 248 temporarily stores data units received from one or more ingress sections 204-xa in the egress buffer while they await processing by one or more other components of the egress section 204-xb. The buffer manager of the traffic manager circuit 248 is configured to operate in a similar manner to the buffer manager of the ingress arbitration circuit 220 discussed above.

[0065] The egress buffer memory 252 (and the buffer of the egress buffer memory 252) is structured in the same or similar manner as the ingress buffer memory 224 (and the buffer of the ingress buffer memory 224) discussed above. For example, each data unit received by the egress portion 204-xb is stored in one or more entries in one or more buffers, and these entries are marked as available to prevent data units already buffered in the egress buffer memory 252 from being overwritten by newly received data units. In some embodiments, after a data unit is released from the egress buffer memory 252, the one or more entries in the egress buffer memory 252 that contained the data unit are then marked as available to store newly received data units.

[0066] Generally, the buffers within the egress buffer memory 252 include a variety of buffers or sets of buffers, each used for various purposes and / or for various components within the egress portion 204-xb.

[0067] A buffer manager (not shown) is configured to manage the use of the egress buffer 252. In various embodiments, the buffer manager performs, for example, one or any preferred combination of the following: allocating and deallocating specific segments of memory for buffers; creating and deleting buffers in said memory; identifying available buffer entries for storing data units; maintaining a mapping of those buffer entries to the data units stored in them (e.g., by the packet sequence number assigned to each packet when the first data unit of each packet is received); marking a buffer entry as available when a data unit stored in the buffer is dropped, sent, or released from the buffer; determining when a data unit should be dropped due to its inability to be stored in the buffer; and performing garbage collection of buffer entries for data units (or parts thereof) that are no longer needed.

[0068] The traffic manager circuit 248 is also configured, according to some embodiments, to maintain an egress queue 256 used to manage the order in which data units are processed from the egress buffer 252. The egress queue 256 is structured the same as or similar to the ingress queue 228 discussed above.

[0069] In one embodiment, different egress queues 256 may exist for different destinations. For example, each port 212 may be associated with one or more sets of egress queues 256. The egress queue 256 to which a data unit is assigned may be selected, for example, based on forwarding information indicating the target port determined for the packet.

[0070] In some embodiments, different egress queues 256 correspond to their respective flows or sets of flows. That is, packets for each identifiable traffic flow or group of traffic flows are assigned to their respective sets of egress queues 256. In some embodiments, different egress queues 256 correspond to different traffic classes, QoS levels, etc.

[0071] In some embodiments, the egress queue 256 corresponds to each egress port 212 and / or each priority set. For example, each set of multiple queues 256 corresponds to each of at least some of the egress ports 212, and each queue 256 in the set of multiple queues 256 corresponds to its respective priority set.

[0072] Generally, when the egress portion 204-xb receives a packet from the ingress portion 204-xa via the interconnect 116, the traffic manager circuit 248 stores (or "enqueues") the packet in the egress queue 256.

[0073] In various embodiments, the ingress buffer memory 224 corresponds to the same or different physical memory as the egress buffer memory 252. In some embodiments, where the ingress buffer memory 224 and the egress buffer memory 252 correspond to the same physical memory, the ingress buffer 224 and the egress buffer 252 are stored in different portions of the same physical memory, allocated to the ingress and egress operations, respectively.

[0074] In some embodiments, where the ingress buffer memory 224 and the egress buffer memory 252 correspond to the same physical memory, the ingress buffer 224 and the egress buffer 252 include at least some of the same physical buffers and are separated only in a logical sense. In one such embodiment, metadata or internal marking may indicate whether a given individual buffer entry belongs to the ingress buffer 224 or the egress buffer 252. To avoid conflicts when they are distinguished only in a logical sense, the ingress buffer 224 and the egress buffer 252 may be allocated a specific number of entries in each of the physical buffers they share, and the number of entries allocated to a given logical buffer is said to be the size of that logical buffer. In some such embodiments, when a packet is forwarded from an ingress portion 204-xa to an egress portion 204-xb within the same packet processing module 204, instead of copying the packet from the ingress buffer entry to the egress buffer, the data unit remains within the same buffer entry, and the designation of the buffer entry (e.g., whether it belongs to an ingress queue or an egress queue) changes with the stage of processing.

[0075] According to various embodiments, the egress portion 204-xb also includes an egress packet processor 268, which is configured to perform egress processing operations on packets, such as one, two, or more preferred combinations of packet duplication (e.g., for multicast packets), header modification, rate limiting, traffic shaping, egress policing, flow control, and maintaining statistics about packets. In some embodiments, for example, when modifying the header of a packet (e.g., changing the destination address, adding a tunneling header, removing a tunneling header), the egress packet processor 268 modifies the header information in the egress buffer 252.

[0076] In one embodiment, the egress packet processor 268 is coupled to a group of egress ports 212 via an egress arbitration circuit 272 configured to coordinate the egress packet processor 268's access to the group of egress ports 212.

[0077] In some embodiments, the egress packet processor 268 is additionally or alternatively coupled to a preferred destination for packets other than the egress port 212, such as one or more internal central processing units (not shown), one or more storage subsystems, etc.

[0078] During the processing of a data unit, the egress packet processor 268 may duplicate the data unit once or more times. For example, a data unit may be duplicated for purposes such as multicast, mirroring, or debugging. For this reason, a single data unit may be duplicated and stored in multiple egress queues 256. Thus, while certain techniques described herein may refer to the original data unit received by the network device 200, it is understood that these techniques also apply equally to copies of data units generated by the network device for various purposes. Copies of data units may be partial or complete. Furthermore, an actual physical copy of the data unit may reside in the egress buffer 252, or a single copy of the data unit 252 may be linked from a single buffer location (or a single set of locations) within the egress buffer 252 to multiple egress queues 256.

[0079] Figure 2B is another simplified block diagram of a network device 200 according to one embodiment. As shown in Figure 2B, the network device 200 also includes one or more central processing units (CPUs) 276. The one or more CPUs 276 are configured to perform management functions of the network device 200, such as configuring a packet processing module 204, optimizing the network device 200, collecting data, and collecting statistics. The CPUs 276 are coupled to one or more memories 278 that store machine-readable instructions, and the CPUs 276 are configured to execute machine-readable instructions.

[0080] Referring again to Figure 2A, the ingress arbitration circuit 220 includes one or more load-aware PSD modules 280. Each load-aware PSD module 280 is configured to initiate measurement of PSD information regarding the distribution of packet sizes processed by the network device in response to the determination that the processing load of the network device meets a condition. The processing load is represented by a preferred load metric. In some embodiments, the load metric corresponds to individual entities corresponding to ingress portions 204-xa, such as i) the rate at which data is received at port 208, ii) the fill level of the ingress rescue 228, iii) the length of the ingress rescue 228, iv) the time delay between when a packet is added to the ingress rescue 228 and when the packet is dequeued from the ingress rescue 228, and v) the occupancy level of the ingress buffer 224. In some embodiments, the load-aware PSD module 280 is configured to determine when the processing load of the network device meets a condition by comparing at least the individual load metrics with thresholds. In some embodiments, the load-aware PSD module 280 is configured to determine when the processing load of a network device meets a condition by comparing at least a plurality of individual load metrics to a threshold (or a plurality of different thresholds) and determining whether any of the plurality of individual load metrics meet the corresponding threshold. In other embodiments, the load-aware PSD module 280 is configured to determine when the processing load of a network device meets a condition by comparing at least a plurality of individual load metrics to a threshold (or a plurality of different thresholds) and determining whether all of the plurality of individual load metrics meet the corresponding threshold.

[0081] In some embodiments, the load metric is a preferred mathematical combination of two or more preferred individual load metrics, such as those described above, and the load-aware PSD module 280 is configured to determine when the processing load of a network device meets a condition by comparing a mathematical combination of at least two or more preferred individual load metrics with a threshold.

[0082] In one embodiment, the load-aware PSD module 280 is configured to measure PSD information corresponding to individual entities corresponding to ingress portions 204-xa, such as packets received by port 208, packets stored in ingress rescue 228, and packets stored in ingress buffer 224. In one embodiment, for example, if the load metric corresponds to individual entities, the load-aware PSD module 280 measures PSD information for those individual entities.

[0083] In other embodiments, the load-aware PSD module 280 is configured to additionally or alternatively measure PSD information corresponding to a group of entities corresponding to ingress portions 204-xa, such as packets received by a set of multiple ports 208, packets stored in a set of multiple ingress rescue queues 228, packets stored in a set of multiple ingress buffers 224, etc. In one embodiment, for example, if the load-aware PSD module 280 determines whether any of several individual load metrics corresponding to multiple entities meet a corresponding threshold, the load-aware PSD module 280 measures PSD information for all of the multiple entities.

[0084] In some embodiments, Figure 2A shows that each ingress arbitration circuit 220 includes one load-aware PSD module 280, while in other embodiments, each of at least one ingress arbitration circuit 220 includes multiple load-aware PSD modules 280. In some embodiments, two or more ingress arbitration circuits 220 include a different number of multiple load-aware PSD modules 280. In some embodiments, at least one ingress arbitration circuit 220 does not include any load-aware PSD modules 280.

[0085] The traffic manager circuit 248 includes one or more load-aware PSD modules 284. The load-aware PSD module 284 is similar to the load-aware PSD module 280, but measures load metrics and PSD information for entities in the egress section 204-xb. For example, the load-aware PSD module uses load metrics such as i) the rate at which data is being transmitted through port 212, ii) the fill level of the egress queue 256, iii) the length of the egress queue 256, iv) the time delay between when a packet is added to the egress queue 256 and when the packet is dequeued from the egress queue 256, and v) the occupancy level of the egress buffer 252.

[0086] Furthermore, the load-aware PSD module 284 is configured to measure PSD information corresponding to individual entities corresponding to egress portions 204-xab, such as PSD information relating to packets transmitted by port 212, packets stored in egress queue 256, packets stored in egress buffer 252, etc. In one embodiment, for example, if the load metric corresponds to individual entities, the load-aware PSD module 284 measures PSD information relating to those individual entities.

[0087] In other embodiments, the load-aware PSD module 284 is configured to additionally or alternatively measure PSD information corresponding to a group of entities corresponding to the egress portion 204-xb, such as packets transmitted by a set of multiple ports 212, packets stored in a set of multiple egress queues 256, packets stored in a set of multiple egress buffers 252, etc. In one embodiment, for example, if the load-aware PSD module 284 determines whether any of a plurality of individual load metrics corresponding to a plurality of entities meets a corresponding threshold, the load-aware PSD module 284 measures PSD information for all of the plurality of entities.

[0088] The egress arbitration circuit 272 also includes one or more load-aware PSD modules 288. The load-aware PSD module 288 is similar to the load-aware PSD module 284, but measures PSD information after the packet has been processed by the egress packet processor 268, thereby allowing the packet size to be changed by adding or removing a tunnel header, modifying the header, etc. The load-aware PSD module 288 uses load metrics such as i) the rate at which data is being transmitted through port 212, ii) the fill level of the egress queue 256, iii) the length of the egress queue 256, iv) the time delay between when a packet is added to the egress queue 256 and when the packet is dequeued from the egress queue 256, and v) the occupancy level of the egress buffer 252.

[0089] Furthermore, the load-aware PSD module 288 is configured to measure PSD information for packets corresponding to individual entities corresponding to egress portions 204-xab that have been processed by the egress packet processor 268, such as packets transmitted by port 212, packets stored in egress queue 256, and packets stored in egress buffer 252. In one embodiment, for example, if the load metric corresponds to individual entities, the load-aware PSD module 288 measures PSD information for those individual entities.

[0090] In other embodiments, the load-aware PSD module 288 is configured to measure PSD information for packets corresponding to groups of entities corresponding to egress portions 204-xb that have been processed by the egress packet processor 268, such as packets transmitted by a set of multiple ports 212, packets stored in a set of multiple egress queues 256, packets stored in a set of multiple egress buffers 252, etc. In one embodiment, for example, if the load-aware PSD module 288 determines whether any of several individual load metrics corresponding to multiple entities meet the corresponding threshold, the load-aware PSD module 288 measures PSD information for all of the multiple entities.

[0091] In various embodiments, the PSD modules 280, 284, and 288 are implemented using hardware circuitry and / or one or more processors that execute machine-readable instructions stored in one or more memories coupled to one or more processors.

[0092] In some embodiments, the PSD information generated by the PSD modules 280, 284, and 288 is used to control the network device 200. As an illustrative example, if ports 208, 212 or queues 228, 256 are heavily loaded, and the PSD information indicates a relatively high proportion of short packets corresponding to ports 208, 212, or queues 228, 256, the network device 200 redistributes within itself the processing of one or more packet types, one or more flows of packets, etc., that contribute to the high proportion of short packets corresponding to ports 208, 212, or queues 228, 256. For example, the network device 200 redirects one or more packet types, one or more flows of packets, etc., that contribute to a high proportion of short packets to dedicated queues 228, 256, and processes the packets in the dedicated queues 228, 256 at a reduced rate (e.g., by an ingress packet processor 232, an egress packet processor 268, etc.) to reduce the power consumption associated with processing packets in the dedicated queues 228, 256.

[0093] According to some embodiments, more broadly, the network device 200 adjusts the buffer allocation algorithm, queue allocation, buffer admission policy, buffer storage algorithm, etc., based on PSD information determined when the load metric indicates a high processing load.

[0094] As another example, the network device 200 reduces power consumption associated with processing packets in dedicated queues 228, 256 by redirecting one or more packet types, one or more packet flows, etc., that contribute to a high proportion of short packets, to dedicated queues 228, 256, and the clock rate of the processor that processes packets in dedicated queues 228, 256 at a reduced rate.

[0095] As another example, network device 200 redirects one or more packet types, one or more flows of packets, etc., that contribute to a high proportion of short packets to another egress port 212 with a lower load, and the other egress port 212 corresponds to an alternative path through the network.

[0096] In some embodiments, the network device 200 transmits the collected PSD information and optionally other telemetry information, such as buffer length, queue length, and latency measurements, to an analyzer and / or controller located outside the network device 200. In one embodiment, the analyzer and / or controller determines initial operating parameters (e.g., the processing rate of the network device 200, the selection of network paths routed through the network device 200, etc.) to reduce power consumption by the network device 200. In one embodiment, the analyzer and / or controller further determines, based on the collected PSD information and optionally other telemetry information, whether the operating parameters of the network device and / or other network devices in the network should be adjusted, and adjusts the operating parameters of the network device 200 and / or other network devices accordingly. According to one embodiment, the analyzer and / or controller uses reinforcement learning to determine the optimal operating parameters of the network device 200 and / or other network devices, for example, by using the collected PSD information and optionally other telemetry information as feedback.

[0097] Referring here to Figures 2A and 2B, in some embodiments, when one or more ports 208, 212 or one or more queues 228, 256 are heavily loaded, and the PSD information indicates a relatively high proportion of short packets corresponding to ports 208, 212 or queues 228, 256, one or more CPUs (276) adjust the operations being performed by the CPUs (276). For example, CPU 276 reduces the rate at which statistics about network devices 200 are collected by CPU 276. Additionally or alternatively, in embodiments where CPU 276 performs functions related to artificial intelligence and / or machine learning (AI / ML) network analysis and / or control, CPU 276 reduces the rate at which such functions are performed.

[0098] In some embodiments, PSD information is provided to another device in the communication network to which network device 200 belongs for network control and / or monitoring operations, such as network optimization, congestion management, and troubleshooting. For example, CPU 276 generates one or more packets containing PSD information, and CPU 276 controls network device 200 to transmit the one or more packets to another network device in the communication network. In one embodiment, the other communication uses the PSD information in one or more packets to perform one or more functions related to network optimization, congestion management, troubleshooting, etc.

[0099] In another embodiment, in response to PSD information indicating a high proportion of small packets during a period of high processing load, the network device 200 identifies one or more other network devices in the communication network that are transmitting a large number of small packets to the network device 200, and then transmits flow control packets to those one or more other network devices.

[0100] In another embodiment, in response to PSD information indicating a high proportion of small packets during a period of high processing load, network device 200 identifies one or more packet flows contributing to a large number of small packets, and then begins to notify one or more other network devices that packets in one or more packet flows are causing congestion at network device 200, either by explicit congestion notification (ECN) marked packets in one or more packet flows, or by using another preferred congestion notification mechanism.

[0101] PSD information is statistical information about the distribution of packet sizes in a set of multiple packets. The count of packets with packet sizes that fall within each packet size range is a descriptive example of PSD information and may be called a packet size histogram or packet size histogram. Other examples of PSD information include combinations of statistical measures, such as i) one or more statistics measuring the central trend of each (e.g., mean, median, mode, etc.) and ii) one or more statistics measuring the scatter or variability (e.g., range, standard deviation, variance, mean absolute difference, median absolute deviation, mean deviation, etc.).

[0102] Figure 3 is a simplified block diagram of an exemplary PSD module 300 according to one embodiment. In some embodiments, the PSD module 300 corresponds to one or more of the PSD modules 280, 284, and 288 in Figure 2A. In some embodiments, one or more (or all) of the PSD modules 280, 284, and 288 have a preferred structure different from that of the PSD module 300. Furthermore, the PSD module 300 is used in a preferred network device different from the network device 200 in Figures 2A and 2B.

[0103] The PSD module 300 includes a bank of counters 304 (which may also be referred to herein as “counter bank 304”) used to maintain counts of packets that fall within different packet size ranges. Counts of packets that fall within different packet size ranges are an example of packet size distribution (PSD) information.

[0104] Counter bank 304 contains H sets of counters, where H is a preferred positive integer. Each set of counters may be referred to herein as a “histogram set.” Each histogram set corresponds to an entity (e.g., a port, queue, buffer, etc.) or group of entities of a network device, and each histogram set is used to maintain a count of packets corresponding to an entity or group of entities contained within different packet size ranges. Each histogram set contains a plurality of counters 312, each counter 312 corresponding to its respective packet size range. Different packet size ranges counted by counters 312 in a histogram set may be referred to herein as “bins,” and counters 312 in a histogram set may be referred to herein as “bin counters.”

[0105] In one embodiment, the granularity of the packet size range counted in the histogram set is configurable. Therefore, in some embodiments, some counters 312 in the histogram set are not used for larger size ranges.

[0106] In some embodiments, at least some of the counters 312 can be selectively used for different histogram sets. For example, if the granularity of a first histogram set does not require the maximum number of counters 312, the counters 312 not required for the first histogram set can be used for another histogram set.

[0107] The PSD module 300 also includes a histogram index generator 320 configured to generate an indicator (which may be referred to herein as a “histogram index”) of a corresponding set of histograms in the counter bank 304 based on an indicator of an entity (or group of entities) associated with a packet (e.g., a port that received or transmitted the packet, a queue in which the packet was stored, a buffer in which the packet was stored, etc.). In one embodiment, the entity indicator includes a port identifier (e.g., an identifier for ingress port 208 and / or egress port 212). In another embodiment, the entity indicator includes an identifier for a group of ports. In another embodiment, the entity indicator additionally or alternatively includes a queue identifier (e.g., an identifier for ingress queue 228 and / or egress queue 256) and / or an identifier for a group of queues. In some embodiments, the entity indicator additionally or alternatively includes a buffer identifier (e.g., an identifier for ingress buffer 224 and / or egress buffer 252) and / or an identifier for a group of buffers.

[0108] In some embodiments, the histogram index generator 320 is configured to generate a histogram index based on one or more characteristics of a packet, such as the packet type, protocol type, or the type of packet flow to which the packet belongs. Thus, in one embodiment, a histogram set can be used to generate PSD information for packets associated with a specific entity (or group of entities) and having one or more specific packet characteristics. For example, a first histogram set is used to generate PSD information for packets associated with a specific entity and having a first set of one or more specific characteristics, and a second histogram set is used to generate PSD information for packets associated with a specific entity and having a second set of one or more specific characteristics. In another example, a histogram set is used to generate PSD information for packets associated with a specific entity and having one or more specific characteristics, but packets associated with a specific entity but not having one or more specific characteristics are not used to generate PSD information.

[0109] In some embodiments, the histogram index generator 320 includes (or is coupled to) a configuration memory 322 that stores configuration information including associations between histogram sets and entities. In such embodiments, the histogram index generator 320 uses the associations between histogram sets and entities to determine one or more histogram sets associated with an entity (or group of entities). In some embodiments, the configuration information in the configuration memory 322 includes associations between histogram sets and entity / packet characteristic tuples. In such embodiments, the histogram index generator 320 uses the associations between histogram sets and entity / packet characteristics to determine the histogram sets associated with an entity / packet characteristic tuple.

[0110] The PSD module 300 also includes a granularity table 324 configured to store each granularity index for the histogram sets of the counter bank 304. In one embodiment, for example, the granularity table 324 stores each granularity index for each histogram set. In one embodiment, the granularity index indicates the size range of each counter 312 in the histogram set. In another embodiment, the granularity index additionally or alternatively indicates the number of counters 312 in the histogram set.

[0111] The granularity table 324 is configured to receive histogram indices from the histogram set index generator 320. The granularity table 324 uses the histogram indices to look up the granularity indicators corresponding to the histogram indices and outputs the granularity indicators.

[0112] The PSD module 300 also includes a bin counter index generator 328 configured to generate a relative index of the bin counters 312 in a histogram set based on i) the packet size of the packets to be counted, and ii) a granularity index received from the granularity table 324.

[0113] In some embodiments, such as those where the granularity of packet size measured by counter bank 304 is not configurable, the granularity table 324 is omitted, and the bin counter index generator 328 generates a relative index without a granularity index received from the granularity table 324.

[0114] The integrated index generator 332 is configured to generate an integrated index for the counter bank 304 using i) a histogram index generated by the histogram index generator 320 and ii) a bin counter index generated by the bin counter index generator 328. In one embodiment, the integrated index generated by the integrated index generator 332 selects a counter 312 from among multiple histogram sets in the counter bank 304. In another embodiment, the integrated index generated by the integrated index generator 332 selects a counter 312 from among all counters 312 in the counter bank 304.

[0115] The PSD module 300 also includes an update signal generator 348 configured to initiate the measurement of PSD information for packets processed by a network device in response to a load metric being determined to meet certain conditions. According to one embodiment, for example, when a load metric corresponding to an entity (e.g., a port, queue, buffer, etc.) meets certain conditions, the update signal generator 348 initiates the measurement of PSD information for packets associated with the entity (e.g., packets received through a port, packets transmitted by a port, packets stored in a queue, packets stored in a buffer, etc.). In some embodiments, when a load metric corresponding to a group of entities meets certain conditions, the update signal generator 348 initiates the measurement of PSD information for packets associated with that group of entities.

[0116] In some embodiments, when the update signal generator 348 begins measuring PSD information for packets associated with an entity, the update signal generator 348 determines when the counter bank 304 should update the PSD information for the entity. For example, the update signal generator 348 generates an update signal indicating when the counter bank 304 should update the PSD information.

[0117] In one embodiment, the update signal generator 348 generates an update signal based on a packet event corresponding to an entity. According to various embodiments, examples of a packet event corresponding to an entity include a packet being received through a port, a packet being transmitted through a port, a packet being scheduled for transmission through a port, a packet being placed in a queue, a packet being dequeued from the queue, a packet being placed in a buffer, a packet being retrieved from the buffer, and so on.

[0118] In one embodiment, the update signal generator 348 generates an update signal based on one or more characteristics of the packet corresponding to the packet event, such as packet type, protocol type, type of packet flow to which the packet belongs, packet classification, etc. In one embodiment, for example, the update signal generator 348 generates an update signal only for packets having one or more specific packet characteristics, thereby measuring PSD information only for packets having those one or more specific packet characteristics.

[0119] In some embodiments, the update signal generator 348 includes (or is coupled to) a configuration memory 352 that stores configuration information including associations between entities and packet characteristics, which indicate the packet characteristics of packets corresponding to entities (or groups of entities) for which packet distribution information should be measured, for each entity (or group of entities). In one embodiment, in such an embodiment, the update signal generator 348 uses the associations between entities and packet characteristics to determine when to generate an update signal so that PSD information is measured only for packets having certain packet characteristics.

[0120] In some embodiments, the update signal generator 348 is configured to start measuring PSD information for packets corresponding to an entity (or group of entities) in response to the determination that the load metric corresponding to that entity exceeds a first threshold. In one embodiment, the update signal generator 348 is configured to stop measuring PSD information for packets corresponding to an entity in response to the determination that the load metric corresponding to that entity falls below a second threshold. In one embodiment, the second threshold is the same as the first threshold. In another embodiment, the second threshold is below the first threshold to provide hysteresis.

[0121] In various embodiments, the histogram index generator 320, the bin counter index generator 328, the integrated index generator 332, and / or the update signal generator 348 are implemented using hardware circuitry and / or one or more processors that execute machine-readable instructions stored in one or more memories coupled to one or more processors.

[0122] Figure 4 is a graph illustrating a descriptive example of PSD information 400 measured at peak loads less than 50% and greater than 50% according to one embodiment. The PSD information 400 includes a count of packets contained in six packet size ranges (or "bins"), namely i) less than or equal to 64 bytes, ii) 65 to 127 bytes, iii) 128 to 511 bytes, iv) 512 to 2047 bytes, v) 2048 to 4095 bytes, and vi) 4096 to 9216 bytes. In various other embodiments, the PSD information measured by load-aware PSD modules 280, 284, 288, and 300 includes more packet size ranges and / or different packet size ranges compared to the exemplary PSD information 400 in Figure 4.

[0123] Figure 5 is a simplified exemplary state diagram 500 of a circuit controlling the generation of PSD information in a network device according to one embodiment. The update signal generator 348 in Figure 3 implements the state diagram 500 in one embodiment, and the state diagram 500 is described with reference to Figure 3 for ease of explanation. In other embodiments, the update signal generator 348 implements a different preferred set of state transitions different from the state diagram 500. In some embodiments, the state diagram 500 is further implemented by a different PSD measuring device different from the PSD module 300 in Figure 3.

[0124] In state 504, PSD measurement is turned off for an entity or group of entities. The update signal generator 348 remains in state 504 while the load metric corresponding to the entity / group of entities remains below the first threshold.

[0125] In response to the load metric exceeding (and / or being equal to the first threshold in some embodiments), the update signal generator 348 transitions to state 508. Upon transitioning to state 508, the update signal generator 348 turns on PSD measurement for entities / groups of entities. Furthermore, while the update signal generator 348 remains in state 508, PSD measurement remains on for entities / groups of entities.

[0126] The update signal generator 348 remains in state 508 while the load metric corresponding to an entity / group of entities remains above a second threshold. In response to the load metric falling below (and / or equaling in some embodiments to) the second threshold, the update signal generator 348 transitions to state 504. Upon transitioning to state 504, the update signal generator 348 turns off the PSD measurement for the entity / group of entities. In one embodiment, the second threshold falls below the first threshold to provide hysteresis. In another embodiment, the second threshold is equal to the first threshold.

[0127] In some embodiments, the PSD measurement corresponding to an entity / group of entities is reset upon transition to state 508. For example, the counter 312 of the histogram set corresponding to an entity or group of entities is reset upon transition to state 508. In other embodiments, the PSD measurement corresponding to an entity / group of entities is not reset upon transition to state 508, thereby allowing the ongoing PSD measurement to be performed across multiple separate, high-load instances associated with the entity / group of entities.

[0128] Figure 6 is another simplified exemplary state diagram 600 of a circuit controlling the generation of PSD information in a network device according to another embodiment. The update signal generator 348 in Figure 3 implements state diagram 600 in one embodiment, and state diagram 600 is described with reference to Figure 3 for ease of explanation. In other embodiments, the update signal generator 348 implements a different preferred set of state transitions different from state diagram 600. In some embodiments, state diagram 600 is further implemented by a different PSD measuring device different from the PSD module 300 in Figure 3.

[0129] In state 604, PSD measurement is turned off for an entity or group of entities. The update signal generator 348 remains in state 604 while the load metric corresponding to the entity / group of entities remains below the threshold.

[0130] In response to the load metric exceeding (and / or being equal to) a threshold in some embodiments, the update signal generator 348 transitions to state 608. Upon transitioning to state 608, the update signal generator 348 starts its timer. The timer is configured to measure a suitable period of time.

[0131] Furthermore, the update signal generator 348 turns on PSD measurement for an entity or group of entities when transitioning to state 608. PSD measurement remains on for an entity or group of entities while the update signal generator 348 remains in state 608.

[0132] The update signal generator 348 remains in state 608 while the timer has not expired. In response to the timer expiring, the update signal generator 348 transitions to state 604. Upon transitioning to state 604, the update signal generator 348 turns off the PSD measurement for the entity / group of entities.

[0133] With respect to Figures 4 and 5, the terms “above” and “below” are relative terms that depend on the load metric being compared. For example, a load metric “above” a threshold corresponds to a relatively high processing load on the network device, while a load metric “below” a threshold corresponds to a relatively low processing load on the network device. If a particular load metric is inversely proportional to the processing load (such as processing load indicating available processing capacity), then a load metric below a threshold indicates a relatively high processing load, while a load metric above a threshold indicates a relatively low processing load.

[0134] In various embodiments, the state transition diagrams 500 and / or 600 are implemented using hardware circuitry (e.g., a hardware state machine) and / or one or more processors that execute machine-readable instructions stored in one or more memories coupled to one or more processors.

[0135] Figure 7 is a simplified flowchart of an exemplary method 700 for controlling a network device based on PSD measurement, according to one embodiment. The network device 200 in Figures 2A-2B implements method 700 according to one embodiment, and method 700 is described with reference to Figures 2A-2B for ease of explanation. In other embodiments, the network device 200 implements another preferred method for controlling the network device 200 based on a different PSD measurement than method 700. Furthermore, in some embodiments, method 700 is implemented by another preferred network device different from the network device 200 in Figures 2A-2B.

[0136] In some embodiments, Method 700 is implemented using the PSD module 300 shown in Figure 3, and Method 700 will be described with reference to Figure 3 for ease of explanation. In other embodiments, the PSD module 300 is used to implement another preferred method for controlling network devices based on PSD measurements. Furthermore, in some embodiments, Method 700 is implemented using another preferred PSD measuring device different from the PSD module 300 shown in Figure 3.

[0137] In block 704, the network device determines a load metric corresponding to the processing load of the network device. For example, the packet processing module 204 determines the load metric (for example, the ingress arbitration circuit 220 determines it, the traffic manager circuit 248 determines it, etc.), and the load metric corresponds to an entity (or group of entities) of the packet processing module 204, such as ports 208, 212, queues 228, 256, buffers 224, 252, etc.

[0138] In block 708, the network device determines whether the load metric determined in block 704 meets a threshold. In response to determining that the load metric does not meet the threshold, the flow repeats block 708. For example, block 704 includes repeatedly determining the load metric over time, and block 708 includes repeatedly comparing the load metric with a threshold over time. In some embodiments, the ingress arbitration circuit 220 and / or the traffic manager circuit 248 determines whether the load metric meets a threshold. In one embodiment, the PSD module 300 (e.g., update signal generator 348) determines whether the load metric meets a threshold.

[0139] In response to the determination in block 708 that the load metric meets the threshold, the flow proceeds to block 712. In block 712, the network device begins generating PSD measurements. In one embodiment, the load metric determined in block 704 corresponds to an entity or group of entities of the network device, and the PSD measurement initiated in block 712 is for packets corresponding to that entity or group of entities. In some embodiments, the ingress arbitration circuit 220 and / or the traffic manager circuit 248 begin generating PSD measurements in block 712. In one embodiment, the PSD module 300 begins generating PSD measurements in block 712.

[0140] In block 716, the network device terminates the PSD measurement that was started in block 712. In some embodiments, the ingress arbitration circuit 220 and / or the traffic manager circuit 248 start generating the PSD measurement in block 712. In one embodiment, the PSD module 300 starts generating the PSD measurement in block 712.

[0141] In one embodiment, the threshold against which the load metric is compared in block 708 is a first threshold, and method 700 further includes comparing the load metric to a second threshold. In such an embodiment, in response to the determination that the load metric is below the second threshold, the PSD measurement terminates in block 716.

[0142] In another embodiment, method 700 further includes starting a timer in connection with the commencement of PSD measurement generation in block 712. In such an embodiment, the PSD measurement terminates in block 716 in response to the determination that the timer has expired.

[0143] In block 720, the network device is controlled using PSD measurements performed in relation to blocks 712 and 716.

[0144] In one embodiment, using PSD measurement in block 720 includes adjusting the buffer allocation algorithm implemented by the network device based on the PSD measurement. In another embodiment, using PSD measurement in block 720 includes adjusting the queue allocation of the network device based on the PSD measurement.

[0145] In another embodiment, using PSD measurement in block 720 includes the network device 200 redistributing within the network device 200 the processing of one or more packet types, one or more flows of packets, etc., that contribute to the high proportion of short packets corresponding to ports 208, 212, or queues 228, 256, when the load on ports 208, 212, or queues 228, 256 is high and the PSD measurement indicates a relatively high proportion of short packets corresponding to ports 208, 212, or queues 228, 256.

[0146] In another embodiment, using PSD measurement in block 720 involves network device 200 redirecting one or more packet types, one or more flows of packets, etc., that contribute to a high proportion of short packets to another egress port 212 with a lower load, where the other egress port 212 corresponds to an alternative path through the network.

[0147] In another embodiment, the use of PSD measurement in block 720 includes one or more CPUs (276) coordinating operations performed by the CPU (276) based on the PSD measurement.

[0148] In another embodiment, Method 700 includes, in addition to or instead of block 720, a network device providing another network device with PSD measurements taken in relation to blocks 712 and 716, so that these measurements may be used when other network devices control and / or monitor the communication network on which the network device is operating.

[0149] In another embodiment, method 700 includes providing that a network device transmits flow control messages and / or congestion notification messages to another network device based on PSD measurements performed in addition to or instead of block 720 in relation to blocks 712 and 716.

[0150] At least some of the various blocks, operations, and techniques described above are appropriately implemented using dedicated hardware such as discrete components, integrated circuits, ASICs, programmable logic devices (PLDs), processors that execute firmware instructions, processors that execute software instructions, or any combination thereof, one or more of these. When implemented using processors that execute software or firmware instructions, the software or firmware instructions may be stored in any suitable computer-readable memory, such as random access memory (RAM), read-only memory (ROM), or solid-state memory. The software or firmware instructions may include machine-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the various actions described herein.

[0151] Although the present invention has been described with reference to specific examples, these examples are merely illustrative and are not intended to limit the invention, and various modifications, additions, and / or deletions may be made to the disclosed embodiments without departing from the scope of the invention.

Claims

1. A method for controlling the operation of a network device, In the aforementioned network device, a step of determining a load metric corresponding to the processing load of the network device; In response to determining that the load metric satisfies a first threshold, the network device begins measuring distribution information relating to the distribution of packet sizes processed by the network device; The step of completing the measurement of distribution information relating to the distribution of the sizes of packets processed by the network device in the network device; and In the aforementioned network device, the step of controlling the network device using the distribution information. A method for providing this.

2. A step in which the timer of the network device is started in response to the determination that the load metric meets the first threshold; Furthermore, The method according to claim 1, wherein the step of terminating the measurement of the distribution information includes a step of terminating the measurement of the distribution information in response to the expiration of the timer.

3. The method according to claim 1, wherein the step of terminating the measurement of the distribution information is in response to determining that the load metric satisfies a second threshold.

4. Determining that the load metric satisfies the first threshold includes at least one of: i) determining that the load metric is equal to the first threshold, and ii) determining that the load metric exceeds the first threshold; The method according to claim 3, wherein determining that the load metric satisfies the second threshold includes at least one of: i) determining that the load metric is equal to the second threshold, and ii) determining that the load metric is below the second threshold.

5. The method according to claim 4, wherein the first threshold is equal to the second threshold.

6. The method according to claim 4, wherein the first threshold is greater than the second threshold.

7. The step of controlling the network device using the distribution information is: The method according to any one of claims 1 to 6, wherein the network device has the step of moving the transmission of a packet flow from a first port of the network device to a second port of the network device based on the distribution information.

8. The step of controlling the network device using the distribution information is: The method according to any one of claims 1 to 6, wherein the network device has a step of changing from storing packets in a first queue of the network device to storing packets in a second queue of the network device, based on the distribution information.

9. The step of controlling the network device using the distribution information is: The method according to any one of claims 1 to 6, further comprising the step of adjusting the processing rate at which the network device processes packets based on the distribution information.

10. The step of controlling the network device using the distribution information is: The method according to any one of claims 1 to 6, further comprising the step of adjusting an algorithm corresponding to buffer allocation and / or buffer usage based on the distribution information.

11. i) The step of the network device transmitting one or more flow control messages to another network device based on the distribution information; and ii) Based on the distribution information, the network device marks one or more packets to be transmitted to another network device, the marking of the one or more packets is for the purpose of notifying the other network device that the network device is in a congested state. The method according to any one of claims 1 to 6, further comprising at least one of the following.

12. A network device, Multiple network interfaces; A packet processor configured to process data units received via the plurality of network interfaces and to determine which of the plurality of network interfaces transmits the data units; A first circuit configured to determine a load metric corresponding to the processing load of the network device; The second circuit, In response to determining that the load metric satisfies a first threshold, the measurement of distribution information regarding the distribution of packet sizes processed by the network device is initiated. A second circuit configured to complete the measurement of distribution information relating to the distribution of the sizes of packets processed by the network device; and A controller configured to control the network device using the distribution information. A network device equipped with the following features.

13. The second circuit has a timer, and the second circuit, In response to determining that the load metric satisfies the first threshold, the timer is started; In response to the expiration of the timer, the measurement of the distribution information is terminated. The network device according to claim 12, configured as described above.

14. The second circuit described above is In response to determining that the load metric meets the second threshold, the measurement of the distribution information is terminated. The network device according to claim 12, configured as described above.

15. The second circuit described above is i) determine that the load metric is equal to the first threshold, and ii) determine that the load metric is greater than the first threshold; i) that the load metric is equal to the second threshold, and ii) that the load metric is below the second threshold. The network device according to claim 14, configured as described above.

16. The network device according to claim 15, wherein the first threshold is equal to the second threshold.

17. The network device according to claim 15, wherein the first threshold exceeds the second threshold.

18. The plurality of network interfaces include a first network interface and a second network interface, and the controller is Based on the distribution information, the transmission of the packet flow is moved from the first network interface to the second network interface. A network device according to any one of claims 12 to 17, configured as described above.

19. A plurality of queues for storing packets processed by the network device, the plurality of queues including a first queue and a second queue; Furthermore, The network device according to any one of claims 12 to 17, wherein the controller is configured to change the storage of packets from a first queue to a second queue based on the distribution information.

20. The aforementioned controller, Based on the distribution information, the packet processor adjusts the processing rate at which it processes packets. A network device according to any one of claims 12 to 17, configured as described above.

21. The aforementioned controller, Based on the distribution information, adjust the algorithm corresponding to buffer allocation and / or buffer usage. A network device according to any one of claims 12 to 17, configured as described above.

22. The aforementioned controller, i) Controlling the network device to send one or more flow control messages to another network device based on the distribution information; and ii) Controlling the network device to mark one or more packets to be transmitted to another network device based on the distribution information, the marking of the one or more packets to notify the other network device that the network device is in a congested state. A network device according to any one of claims 12 to 17, configured to perform at least one of the following:

Images