Method and system for cache management in network devices

Abstract

Embodiments include methods for managing caches in a network device. In one embodiment, a method is performed by a network device, the method including a first cache of the network device being overloaded, wherein the first cache is within a cache hierarchy that includes a plurality of caches, each cache including a set of forwarding table entries for use by the network device in forwarding packets of traffic flows, and wherein a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy. The method further includes, upon determining that the first cache is overloaded, disabling the first cache for flow lookups, wherein the flow lookups skip the first cache and are performed at a second cache higher in the cache hierarchy; and, upon performing a forwarding table entry insertion in one or more of the plurality of caches, disabling the first cache for the forwarding table entry insertion.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 903,329, filed September 20, 2019, which is incorporated herein by reference. Technical Field

[0003] Embodiments of the present invention relate to the field of networking; and more specifically, to cache management in network devices. Background Technology

[0004] Network devices forward packets of traffic flows to their appropriate destinations within the network. Network devices can use various databases to perform this forwarding. For example, a database for packet forwarding might contain forwarding tables, which include forwarding entries. Matching these entries causes packets to be forwarded based on the action indicated by the mapping entry. To support a large volume of traffic flows, the database needs to contain thousands or even millions of forwarding entries. Databases are typically hosted on storage devices, and it is time-consuming for network devices to traverse the database to find matching forwarding entries.

[0005] To facilitate database traversal, network devices can implement a cache hierarchy with multiple cache levels. Each cache level stores multiple flow table entries corresponding to flow table entries in the database (typically stored in a storage device such as memory). When determining how to forward packets of a traffic flow, the network device first searches for matching flow table entries in the caches. The network device first searches the lowest-level cache (the cache closest to the processor / processor core); and if no match is found, the network device moves to the next higher-level cache. When no match is found in the cache hierarchy, the network device traverses the database. When a matching entry is found in a higher-level cache (or database), this match causes the corresponding matching entry to be inserted in the lower-level cache, so that when packets of the same traffic flow arrive, the network device can find the matching entry in the lower-level cache without continuing to search the higher-level cache.

[0006] In a cache hierarchy, lower level caches typically contain a smaller number of forwarding table entries. Cache insertion triggered by a matching entry found in a higher level cache (or database) can cause the lower level cache to evict one or more existing forwarding table entries in the cache to make room for the newly inserted matching forwarding table entry. However, the evicted forwarding table entries were inserted into the cache earlier to facilitate packets from other flows; when packets from other flows arrive after the eviction, the network device will need to insert the recently evicted forwarding table entries again, which can cause another recently inserted forwarding table entry to be evicted. Frequent insert-evict-insert cycles degrade cache performance, and this reduces the packet forwarding capacity of the network device. SUMMARY

[0007] Embodiments include methods for managing caches in a network device. In one embodiment, a method is performed by a network device, the method including determining that a first cache of the network device is overloaded, where the first cache is within a cache hierarchy that includes a plurality of caches, each cache including a set of forwarding table entries for the network device to forward packets of traffic flows, and where a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy. The method further includes, upon determining that the first cache is overloaded, disabling the first cache for the flow lookup, where the flow lookup skips the first cache and is performed at a second cache that is higher in the cache hierarchy, and disabling the first cache for forwarding table entry insertion when performing the forwarding table entry insertion in one or more of the plurality of caches.

[0008] Embodiments include network devices that manage cache disable / enable. In one embodiment, a network device includes a processor and a non-transitory computer readable storage medium that provides instructions that, when executed by the processor, cause the network device to perform the following operations: determine that a first cache of the network device is overloaded, wherein the first cache is within a cache hierarchy that includes a plurality of caches, each cache includes a set of forwarding table entries for the network device to forward packets of a traffic flow, and wherein a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy; upon determining that the first cache is overloaded, disable the first cache for the flow lookup, wherein the flow lookup skips the first cache and is performed at a second cache that is higher in the cache hierarchy, and disable the first cache for forwarding table entry insertion when the forwarding table entry insertion is performed in one or more of the plurality of caches.

[0009] Embodiments include non-transitory computer readable storage media for cache management. In one embodiment, a non-transitory computer readable storage medium (618, 648) that provides instructions that, when executed by a processor of a network device, can cause the network device to perform the following operations: determine that a first cache of the network device is overloaded, wherein the first cache is within a cache hierarchy that includes a plurality of caches, each cache includes a set of forwarding table entries for the network device to forward packets of a traffic flow, and wherein a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy; upon determining that the first cache is overloaded, disable the first cache for the flow lookup, wherein the flow lookup skips the first cache and is performed at a second cache that is higher in the cache hierarchy, and disable the first cache for forwarding table entry insertion when the forwarding table entry insertion is performed in one or more of the plurality of caches.

[0010] These embodiments reduce cache thrashing, in which a cache constantly encounters full conditions, with a plurality of entries in the cache constantly being evicted and inserted at a high rate. This reduction enhances flow lookup performance and improves cache performance. BRIEF DESCRIPTION OF DRAWINGS

[0011] The application can best be understood by referring to the following description and accompanying drawings that are used to exemplify embodiments of the application. In the drawings:

[0012] Figure 1 A cache hierarchy of a network device according to one embodiment is shown;

[0013] Figure 2 A cache state machine according to one embodiment is shown;

[0014] Figure 3 This is a flowchart illustrating the operation of cache management in a network device according to one embodiment;

[0015] Figure 4 This is a flowchart illustrating the operation of disabling the cache of a network device according to one embodiment;

[0016] Figure 5A This is a flowchart illustrating the operation of enabling caching in a network device according to one embodiment;

[0017] Figure 5B This is a flowchart illustrating the operation of enabling the caching of a network device according to another embodiment;

[0018] Figure 6A The diagram illustrates connectivity between network devices (NDs) within an exemplary network according to some embodiments, and three exemplary implementations of the NDs;

[0019] Figure 6B Exemplary ways of implementing a dedicated network device according to some embodiments are shown;

[0020] Figure 6C Various exemplary methods in which virtual network elements (VNEs) can be coupled, according to some embodiments, are shown;

[0021] Figure 6D A network with a single network element (NE) on each ND according to some embodiments is shown, and within this direct approach, a comparison is made between a traditional distributed approach (typically used by conventional routers) and a centralized approach for maintaining reachability and forwarding information (also known as network control).

[0022] Figure 6E This illustrates a simplified case where each ND implements a single NE according to some embodiments, but the centralized control plane has abstracted (represented) multiple NEs in different NDs into a single NE in one of the virtual networks;

[0023] Figure 6F The illustration shows a case in which multiple VNEs are implemented on different NDs and coupled to each other according to some embodiments, and in which a centralized control plane has abstracted these multiple VNEs so that they appear as a single VNE in one of the virtual networks. Detailed Implementation

[0024] The following description describes methods and apparatus for cache management in network devices. In the following description, numerous specific details are set forth (e.g., logical implementations, resource partitioning / sharing / duplication implementations, types and interrelationships of system components, and logic partitioning / integration choices) in order to provide a thorough understanding of the present application. However, it will be appreciated by one skilled in the art that the present application can be practiced without such specific details. In other instances, control structures, gate level circuits, and full software instruction sequences have not been shown in detail in order not to obscure the present application. Those of ordinary skill in the art, with the

[0025] Terminology

[0026] Reference in the specification to "one embodiment", "an embodiment", "example embodiment" etc., indicates that a described embodiment can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of those skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0027] Text and blocks in square brackets with dashed borders (e.g., large dashed, small dashed, dash-dot, and dotted) can be used herein to show optional operations that add additional features to embodiments of the application. However, such notation should not be taken to mean that these are the only options or optional operations, and / or that blocks with solid borders are not optional in certain embodiments of the application.

[0028] In the following description, embodiments and claims, the terms "coupled" and "connected" along with their derivatives, can be used. It should be understood that these terms are not intended as synonyms for each other. "Coupled" is used to indicate that two or more elements cooperate or interact with each other in some manner. Although the two or more elements can not be in direct physical or electrical contact with each other, they can still be coupled. "Connected" is used to indicate the establishment of communication or

[0029] A network device (also referred to as a network node, or simply a node) is an electronic device (defined below) that is communicatively interconnected with other electronic devices on a communication network. A switch can be implemented in a network device to forward packets of a traffic flow to another network device. A traffic flow (or simply a flow) can be defined as a set of packets whose headers match a given bit pattern. A flow can be identified by a set of attributes embedded into one or more packets of the flow. A set of exemplary attributes includes a 5-tuple (source and destination IP addresses, protocol type, source and destination TCP / UDP ports).

[0030] A switch can be virtualized as a virtual switch (also referred to as a software switch or a virtual network element). For example, OpenvSwitch (commonly referred to as OVS) is an open source implementation of a distributed virtual switch. A virtual switch can be used both as a software-based network switch running inside a virtual machine (VM) system hypervisor and as a control stack for dedicated switching hardware. Thus, virtual switches have been implemented in multiple virtualization platforms, switch chipsets, and network hardware accelerators.

[0031] A switch can operate in a network that implements one or more of various management interfaces and protocols, such as software-defined networking (SDN), NetFlow, sFlow (short for sampled flow), SPAN (switched port analyzer), RSPAN (remote switched port analyzer), CLI (command line interface), LACP (link aggregation control protocol), and 802. lag.

[0032] Cache hierarchy

[0033] Figure 1 A cache hierarchy of a network device is shown in accordance with one embodiment. Network device 120 can implement one or more virtual or physical switches (also referred to as hardware switches). Details of network device implementation are discussed below with respect to network devices 602, 604, or 606 in Figure 6A Network device 120 includes one or more processors 122 to process and forward packets of various flows.

[0034] Processor 122 uses a flow database 132 to forward packets, which includes a plurality of forwarding tables, such as a flow table and a group table. These forwarding tables can be generated based on instructions from a network device and / or a network controller (e.g., an SDN controller). Each table includes one or more table entries. Figure 1 An exemplary flow table entry 101 and an exemplary group table entry 111 are shown in accordance with one embodiment.

[0035] Flow table entry 101 includes:

[0036] • Match fields 102: used to match against packets. These match fields match include ingress port and packet header, and optionally metadata specified by previous tables.

[0037] • Priority 103: match priority of flow table entry.

[0038] • Counter 104: updated when packet matches.

[0039] • Instructions 106: used to modify action set or packet pipeline processing.

[0040] • Timeout 107: maximum amount of time or idle time before network device expires flow.

[0041] • Cookie 108: opaque data value selected by network controller. Can be used by network controller for filtering flow statistics, flow modification, and flow deletion.

[0042] A flow table entry can be identified by its match fields and priority; together the match fields and priority identify a unique flow table entry in the flow table. A flow table entry typically allows a packet of a matching flow to be dropped or forwarded to a port. That is, a flow table entry typically allows a network device to place a packet to a port.

[0043] In contrast, a group table entry can allow a network device to place a packet to a port, one of multiple ports, or multiple ports. A group table entry 111 includes:

[0044] • Group identifier 112: a numeric number (e.g., 32-bit unsigned integer) that uniquely identifies a group.

[0045] • Group type 114: used to determine group semantics.

[0046] • Counter 116: updated when a packet is processed by a group.

[0047] • Action buckets 118: an ordered list of action buckets, where each action bucket includes a set of actions to perform and associated parameters.

[0048] A group table entry can be identified by the group identifier. A group table entry typically allows a packet of a matching flow to be forwarded to: a random port on a set of ports (for load balancing), a first active port on a set of ports (for failover), and all ports on a set of ports (for multicast). Thus, a group table entry allows a network device to place a packet to multiple ports.

[0049] The flow database 132 can store thousands or even millions of flow table entries and group table entries, such as the flow table entry 101 and the group table entry 111. Thus, the flow database 132 can occupy a large storage space of the memory 130 of the network device 120. In fact, although Figure 1 The flow database 132 within the memory 130 of the network device 120 is shown, but a portion or the entire flow database 132 can be stored in a separate physical storage space (e.g., another electronic device). Note that flow table entries and group table entries are referred to as forwarding table entries (or simply entries), and the forwarding table entries (or simply entries) to be inserted into the cache and / or evicted from the cache can be flow table entries or group table entries.

[0050] The operation of the processor 122 searching for a matching forwarding table entry (e.g., a matching flow / group table entry) is referred to as a flow lookup, and the flow lookup traverses thousands / millions of flow table entries and group table entries is time-consuming. Thus, the network device can implement a cache hierarchy including multiple levels of caches. Figure 1 An example of three levels of caches is shown, i.e., levels 1-3 at references 124-128, but more or fewer levels of caches can be implemented.

[0051] A lower level of cache (e.g., a cache closer to the processor 122) typically has a smaller storage space and contains fewer forwarding table entries than a higher level of cache (e.g., a cache farther away from the processor 122). Additionally, the forwarding table entries in a lower level of cache are typically more specific in their matching criteria. For example, the matching fields can include fewer wildcard fields. The matching fields can be wildcard fields (matching any value), and in some cases bit mask fields (matching a subset of bits) to make the matching fields more general. For the same flow, a matching forwarding table entry in a lower level of cache can be more specific than the corresponding forwarding table entry in a higher level of cache. Thus, two different flows can match one forwarding table entry in a higher level of cache (e.g., the L2 cache), but they match two forwarding table entries in a lower level of cache (e.g., the L1 cache). Because a lower level of cache includes fewer forwarding table entries and / or more specific matching criteria, the flow lookup at a lower level of cache is typically more efficient (occupies less time / bandwidth / hardware resources) than at a higher level of cache.

[0052] When a packet of a flow is received at the network device 120, the network device 120 first performs a flow lookup at the LI cache 124. When a matching forwarding table entry is found (cache hit), the entry is provided to the processor 122 for further processing. If a matching forwarding table entry is not found (cache miss), the network device 120 proceeds to the next level of cache (L2 cache 126) to perform a flow lookup to see if there is a cache hit, and if so, the entry is provided to the processor 122. Otherwise (cache miss), the network device 120 proceeds to the next level of cache (L3 cache 128). If a matching forwarding table entry is not found at this point, the network device 120 determines that the flow lookup has exhausted the cache hierarchy (as a cache miss is returned in the full cache hierarchy), and then the network device 120 performs a flow lookup at the flow database 132 in the memory 130. If a matching forwarding table entry is not found in the cache hierarchy and the memory 130 storing the flow database 132, in some embodiments, the network device 120 will work with the network controller to determine how to forward the packet and generate a forwarding table entry to forward the packet.

[0053] When a matching forwarding table entry is found at a higher level of cache (cache hit) or the flow database (or a forwarding table entry is generated by the network device / network controller), the forwarding table entry is inserted into all caches to ensure that subsequent packets belonging to the same flow will be found in the cache and no further flow lookup in the flow database is needed. When a cache is full, inserting a forwarding table entry into the cache can cause one or more existing forwarding table entries to be evicted. When a cache is full of all allocated space, and a request to insert another entry causes one or more existing entries to be evicted. A cache can be considered full based on different criteria at different cache levels. For example, a lower level cache can be a hash table based cache such that for a given hash, two entries are stored in the forwarding table of the cache. When an entry needs to be inserted into the cache and both entries are already filled, the cache is full for that entry. The network device will evict one of these entries to insert the new entry.

[0054] Because inserting a forwarding table entry into a cache can cause an existing forwarding table entry to be evicted when the cache is full, for some traffic patterns, the cache can remain in a constant full state and thus cause repeated evictions. When the number of different flows is much larger than the cache size, the likelihood of encountering a cache full scenario (causing cache entry eviction) in a lower level cache is much higher.

[0055] When a cache continually encounters a full scenario, multiple entries in the cache are continually evicted and inserted at a high rate. This is often referred to as cache thrashing. When a cache experiences this, the cache can not provide the expected performance boost for the flow lookups, but rather cause the flow lookup performance to degrade.

[0056] Embodiments aim to reduce cache thrashing by disabling the cache when a cache thrashing condition is detected, partially enabling the cache and monitoring the performance of the cache, and enabling the cache when the performance is satisfactory. The disabling and enabling of the various levels of cache can be coordinated by the cache manager 150, which causes the flow lookups and forwarding table entry insertions to skip the cache when the cache is disabled, causes the flow lookups to skip the cache but allow forwarding table entry insertions at the cache when the cache is in a partially enabled state, and causes the cache to be enabled once the cache performance meets criteria as discussed herein.

[0057] Disable / enable cache

[0058] Figure 2 A state machine of a cache is shown according to one embodiment. The cache can be any one of the caches LI to L3 in Figure 1 (or any other level of cache in a different embodiment). The transitions between the different states of the cache can be coordinated by a cache manager (e.g., cache manager 150).

[0059] At reference numeral 202, the cache starts in a cache enabled state. In the cache enabled state, the network device can perform flow lookups at the cache, and forwarding table entries can be inserted at the cache due to a match at another level of cache or flow database (or due to a new matching forwarding table entry being generated by the network device / network controller).

[0060] At reference numeral 212, the cache in the cache enabled state monitors for an indication of overload. The cache can select a sampling period T and monitor the cache for an indication of overload. In one embodiment, the indication of overload can be calculated using measurements during the sampling period T, including counts of cache evictions, entry insertions, and cache hits. For example, the indication of overload during the sampling period can be calculated as follows:

[0061]

[0062] The overload indication can be compared to a threshold, which is user configurable, and can be different for different levels of cache and during different runtime periods and / or network conditions. In one embodiment, the cache is considered overloaded when the overload indication exceeds the threshold for a sampling period T. Alternatively, the cache is considered overloaded only after the overload indication exceeds the threshold (or another threshold specifically for multiple sampling periods) for multiple periods (either consecutively or for a majority of a defined consecutive sampling periods). Once the cache is considered overloaded, the overload condition is indicated at reference numeral 222 and the state of the cache transitions to the cache disabled state 204. When the cache is considered to be working normally without overload, the counts of cache evictions, entry insertions, and cache hits can be reset (i.e., to 0 or other defined value) and at reference numeral 212, the cache manager continues to monitor the overload indication for the next sampling period.

[0063] In the cache disabled state 204, the network device no longer performs flow lookups at the cache and the forwarding table entries are not inserted into the cache due to matches at another level of cache or flow database. The network device will skip the cache and perform flow lookups and entry insertions at other caches and flow databases. Once the cache enters the cache disabled state 204, at reference numeral 214, a disable timer is triggered to determine when to attempt to enable the cache, which can be set to T dry , which is the period that the cache is to wait until the soak period begins. The timer is also configurable and can be different for different levels of cache and during different runtime periods and / or network conditions.

[0064] At reference numeral 224, once the disable timer expires, the state machine moves to the cache probation state 206. During the cache probation state 206, the cache is partially enabled such that when flow lookups are not performed at the cache, the forwarding table entries can be inserted into the cache due to matches at another level of cache or flow database. Thus, when the cache is in the cache probation state 206, the network device will skip the cache and perform flow lookups at other caches and flow databases, however, the network device will insert the forwarding table entries into the cache due to the forwarding table entry matches at another level of cache or flow database.

[0065] In the cache probation state 206, the cache manager monitors the cache enable indication to determine whether to fully enable the cache at reference numeral 216. The cache enable indication can be based on the number of flow lookups performed at the cache during the soak period T soakmeasurements of cache eviction counts and entry insertion counts within the cache. The cache enable indication during the immersion period can be computed as follows:

[0066]

[0067] The cache enable indication can be compared to another threshold, which is also user configurable, and can be different for different levels of cache and during different run-time periods and / or network conditions. In one embodiment, the cache enable indication is considered to be set (i.e., cache enable is indicated) when the cache enable indication is below the threshold for the immersion period. Alternatively, the cache enable indication is considered to be set only after the cache enable indication is below the threshold for multiple periods (either consecutively or for a majority of a defined consecutive period) or another threshold specifically for multiple immersion periods. At reference numeral 226, once the cache enable is indicated, the state machine returns to the cache enable state 202.

[0068] If the cache enable indication exceeds the threshold for one or more immersion periods, the enable indication is considered to be unset at reference numeral 228, and the state machine returns to the cache disable state 204. In some embodiments, the cache manager adds a backoff time (T backoff ) to cause the cache to stay in the cache disable state 204 for a longer time before it returns to the cache trial state 206 (new disable timer = T dry + T backoff ). The backoff time can be randomly selected in a range or determined according to a formula.

[0069] Note that although the formulas (1) and (2) give examples of computing the overload indication and the cache enable indication, they are for illustration only. Embodiments include other ways of using other measurements to determine the overload indication and the cache enable indication. For example, other measurements can use other arithmetic relationships (e.g., addition, subtraction, multiplication, and division) to compute (1) the overload indication using two or more of the cache eviction number, the entry insertion number, and the cache hit number, and / or (2) the enable indication using the cache eviction number, the entry insertion number, and other measurements. The threshold exceeding criteria can also be updated when another measurement is used. For example, when the cache enable indication is defined according to the reversed formula (2) such that the cache eviction number is the denominator and the entry insertion number is the numerator, the cache enable indication is considered to be set when the cache enable indication exceeds the threshold.

[0070] Some embodiments

[0071] Figure 3 is a flowchart illustrating operations of cache management of a network device according to one embodiment. The network device can be the network device 120 discussed above. These operations can be performed by a cache manager (e.g., the cache manager 150).

[0072] At reference numeral 302, a determination is made that a first cache of a network device is overloaded. The first cache is within a cache hierarchy that includes a plurality of caches, each cache including a set of forwarding table entries for the network device to forward packets of a traffic flow. A lower cache in the cache hierarchy (closer to one or more processors of the network device) is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy. Based on a first measurement, the first cache is determined to be overloaded, as described in detail below with respect to Figure 4 reference numeral 402.

[0073] At reference numeral 304, upon determining that the first cache is overloaded, the first cache is disabled for flow lookups, where the flow lookups skip the first cache and are performed at a second cache higher in the cache hierarchy; and the first cache is disabled for forwarding table entry insertion when the forwarding table entry insertion is performed in one or more of the plurality of caches.

[0074] Optionally, at reference numeral 306, after a period of time (e.g., T dry i.e., the cache is to wait for a period of time before the soak period begins), the first cache is allowed to perform the forwarding table entry insertion when the forwarding table entry insertion is performed in one or more other caches of the plurality of caches. However, the first cache is still disabled for flow lookups. That is, the cache is in the cache probation state 206.

[0075] At reference numeral 308, after the forwarding table entry insertion is allowed, when the first cache is deemed full and a forwarding table entry needs to be inserted in the first cache, one or more forwarding table entries within the first cache can be evicted so that the forwarding table entry will be inserted in the first cache.

[0076] At reference numeral 310, the first cache is enabled to perform flow lookups when a second measurement of a first count of cache evictions and a second count of forwarding table entry insertions is below a second threshold for a second period of time. In one embodiment, the second measurement of the first count of cache evictions and the second count of forwarding table entry insertions is a cache enable indication. The second period of time is, in some embodiments, a soak period (T soak), and in alternative embodiments a plurality of soak periods, as discussed herein. At reference numeral 310, the cache can perform both the flow lookup and the forwarding table entry insertion, so the cache is fully enabled.

[0077] On the other hand, at reference numeral 312, when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions exceeds a second threshold over a configured period, the first cache is disabled for forwarding table entry insertion. The configured period can be one soak period (T soak ), and in alternative embodiments a plurality of soak periods, as discussed herein. Once the forwarding table entry insertion is disabled, the flow proceeds to reference numeral 314, where the period (T dry ) can be adjusted, and then at reference numeral 306, the first cache is again allowed to enable forwarding table entry insertion. In one embodiment, the period is adjusted to include a back-off time, as discussed above.

[0078] Figure 4 is a flow diagram illustrating the operation of disabling a cache of a network device, according to one embodiment. Figure 4 The operations in are an embodiment of reference numeral 302. At reference numeral 402, over a sampling period, it is determined whether the cache is overloaded. In one embodiment, the overloading condition can be determined based on evaluating one or more of a first count of cache evictions, a second count of forwarding table entry insertions, and a third count of cache hits. In one embodiment, the measure is calculated using equation (1), and when a first measure of the first count, the second count, and the third count exceeds a first threshold, the cache is determined to be overloaded, as discussed above.

[0079] When the cache is not overloaded, the flow proceeds to reference numeral 404, where the measure is reset. The measure reset can include resetting the measure counter used for the measure related to equation (1). The overloading counter will also be reset (e.g., to zero) only in the case where the cache is considered overloaded if the cache is overloaded over a plurality of consecutive sampling periods. The overloading counter will not be reset in the case where the cache is considered overloaded if the cache is overloaded over a majority of consecutive sampling periods, but the cache manager can delete counts due to earlier sampling periods (e.g., a sliding window count of how many of the most recent sampling periods were considered overloaded). After the reset, the cache undergoes another sampling period to determine whether the cache is overloaded. Note that some back-off time can be added before the next sampling is performed, so that the cache overloading monitoring does not take excessive resources of the network device.

[0080] When the cache is determined to be overloaded during the sampling period, the flow proceeds to reference numeral 406, where the overload counter is incremented (e.g., incremented by one). The flow then proceeds to reference numeral 408, where it is determined whether the overload counter has reached its threshold. If not, the flow can proceed to reset 404, where the measurement counter for the measurement related to equation (1) is reset, but the overload counter is not reset, so that earlier periods of overload remain in the calculation until the overload counter eventually reaches its threshold at reference numeral 408. Once the threshold is reached, the flow proceeds to reference numeral 409, where the cache is deemed to be overloaded and then disabled. Thus, in some embodiments, when the first measurement of the first count, the second count, and the third count exceeds the first threshold for a plurality of consecutive intervals or a substantial portion of consecutive intervals, the first cache is determined to be overloaded and thus disabled.

[0081] Figure 5A is a flowchart illustrating the operation of a network device to enable a cache according to one embodiment. The operation begins when the cache has been in a disabled state (e.g., in cache disabled state 204). The cache manager can cause the cache to remain in the disabled state for a certain period of time (e.g., T dry duration discussed above).

[0082] At reference numeral 502, the forwarding table entry insertion is enabled, while the flow lookup remains disabled. Then at reference numeral 504, during the soak period, it is determined whether the cache is functioning properly and ready to be enabled. The determination can be based on evaluating one or more of the first count of cache evictions and the second count of forwarding table entry insertions. In one embodiment, equation (2) is used to calculate the measurement, and when the second measurement of the first count and the second count (cache enable indication) is below the second threshold, the cache is determined to be in a proper operating state, as discussed above.

[0083] When the cache is not functioning properly, the flow proceeds to reference numeral 506, where the measurement is reset. The measurement reset can include resetting the measurement counter for the measurement related to equation (2). The enable counter will also be reset (e.g., reset to zero) only in the case where the cache is deemed to be functioning properly when the cache is functioning properly for a plurality of consecutive sampling periods. In the case where the cache is deemed to be functioning properly when the cache is functioning properly for a substantial portion of consecutive sampling periods, the enable counter will not be reset, similar to the overload counter discussed above.

[0084] At reference numeral 512, a backoff time is added to the timer for the next trial (e.g., new disabled timer = T dry + T backoffAs discussed herein). Then at reference 514, the forwarding table entry insertion is again disabled, and after the wait timer expires at the end of the next probation period, the flow again returns to reference numeral 502.

[0085] When the cache is functioning properly, the flow goes to reference numeral 508, where the enable counter is incremented (e.g., by one). The flow then goes to reference numeral 510, where it is determined whether the enable counter has reached its threshold. If not, the flow goes to reference numeral 514, and after the wait timer expires at the end of the next probation period, the flow again returns to reference numeral 502.

[0086] When it is determined that the enable counter has reached its threshold, the flow goes to reference numeral 520, where the cache is enabled, and it can perform flow lookups and forwarding table entry insertion as a properly functioning cache.

[0087] Figure 5B is a flow chart illustrating the operation of enabling a cache of a network device according to another embodiment. Figure 5B Similar to Figure 5A and the operations of similarly numbered reference numeral blocks are similar. One difference is that although Figure 5A the operations in Figure 5B make cache enablement determinations over multiple immersion periods.

[0088] In Figure 5B , after partially enabling the cache at reference numeral 552, the flow goes to reference numeral 553, which waits for the start of an immersion period. The immersion period includes multiple monitoring periods, which can be contiguous time intervals with or without time gaps therebetween. When time gaps are included, the time gaps can be distributed uniformly or non-uniformly based on one or more distribution functions.

[0089] At reference numeral 554, it is now determined whether the cache is functioning properly over the monitoring period.

[0090] If the cache fails to function properly, the flow goes to reference numerals 556, 562, and 564, which include operations similar to those in reference numerals 506, 512, and 514, respectively, of Figure 5A .

[0091] If the cache is working properly, the flow proceeds to reference numeral 558 and then to 560, which include operations similar to those in reference numerals 508 and 510, respectively. When it is determined that the enable counter has not reached its threshold, in the case where a time gap is set between monitoring periods, the flow proceeds to reference numeral 563 to wait for the next monitoring period within the immersion period. Otherwise, the flow proceeds directly to reference numeral 554, where the determination of whether the cache is working properly is performed again.

[0092] Embodiments determine when the cache is consistently being overloaded and use this information to dynamically disable the cache, thereby reducing or eliminating cache thrashing and improving packet switching performance. For example, embodiments of the invention can result in faster flow lookups and less resource consumption for flow lookups because more cache hits and fewer cache evictions can be generated by reducing cache thrashing. Embodiments of the invention create a self-learning system that automatically disables the cache if it does not provide performance benefits but instead causes degradation. Once the cache can work properly, the cache is enabled to improve flow lookup efficiency. Thus, embodiments provide optimal utilization of the cache hierarchy by reacting to changes in traffic patterns over time.

[0093] Network environment in which embodiments of the invention can operate

[0094] Figure 6A Connectivity between network devices (NDs) within an exemplary network and three exemplary implementations of the NDs are shown in accordance with some embodiments. Figure 6A NDs 600A-H and their line connections between 600A-600B, 600B-600C, 600C-600D, 600D-600E, 600E-600F, 600F-600G, and 600A-600G, and between 600H and each of 600A, 600C, 600D, and 600G are shown. These NDs are physical devices, and the connections between these NDs can be wireless or wired (often referred to as links). The additional lines extending from NDs 600A, 600E, and 600F illustrate that these NDs act as ingress and egress points for the network (and thus, these NDs are sometimes referred to as edge NDs; while other NDs can be referred to as core NDs).

[0095] Figure 6A Two exemplary ND implementations in are: 1) a special-purpose network device 602 that uses a custom application-specific integrated circuit (ASIC) and a special-purpose operating system (OS); and 2) a general-purpose network device 604 that uses a general-purpose off-the-shelf (COTS) processor and a standard OS.

[0096] The special-purpose network device 602 includes networking hardware 610, which includes a set of one or more processors 612, forwarding resources 614 (which typically include one or more ASICs and / or network processors), and physical network interfaces (NIs) 616 (through which network connections, such as those shown between the NDs 600A-H, are made), as well as a non-transitory machine-readable storage medium 618 in which networking software 620 is stored. During runtime, the networking software 620 can be executed by the networking hardware 610 to instantiate a set of one or more network software instances 622. Each networking software instance 622, as well as the portion of the networking hardware 610 executing that network software instance (either hardware dedicated to that networking software instance and / or time slices of hardware that is shared by that networking software instance with other networking software instances 622 over time), forms a separate virtual network element 630A-R. Each virtual network element (VNE) 630A-R includes a control communication and configuration module 632A-R (sometimes referred to as a local control module or control communication module) and forwarding tables 634A-R, such that a given virtual network element (e.g., 630A) includes that control communication and configuration module (e.g., 632A), a set of one or more forwarding tables (e.g., 634A), and the portion of the networking hardware 610 executing that virtual network element (e.g., 630A). In one embodiment, the networking software 620 contains a cache manager 150. The cache manager 150 can perform the operations described with reference to the previous figures. The cache manager 150 can generate one or more cache manager instances (CMIs) 653, each CMI for a virtual network element (e.g., a virtual switch), and the CMIs 653 manage cache disabling / enabling of the caches of the virtual network elements.

[0097] The special-purpose network device 602 is generally considered to include, physically and / or logically: 1) an ND control plane 624 (sometimes referred to as a control plane) that includes the processors 612 executing the control communication and configuration modules 632A-R; and 2) an ND forwarding plane 626 (sometimes referred to as a forwarding plane, a data plane, or a media plane) that includes the forwarding resources 614 with the forwarding tables 634A-R and the physical NIs 616. By way of example, in the case of the ND being a router (or implementing routing functionality), the ND control plane 624 (the processors 612 executing the control communication and configuration modules 632A-R) is generally responsible for participating in controlling how data (e.g., packets) are routed (e.g., next-hops for the data and outgoing physical NIs for the data) and storing that routing information in the forwarding tables 634A-R, and the ND forwarding plane 626 is responsible for receiving the data on the physical NIs 616 and forwarding the data out the appropriate physical NIs 616 based on the forwarding tables 634A-R.

[0098] Figure 6BAn exemplary way of implementing the special-purpose network device 602 is shown in accordance with some embodiments. Figure 6B A special-purpose network device is shown that includes cards 638 (typically hot-pluggable). Although in some embodiments the cards 638 are of two types (one or more used as ND forwarding plane 626 (sometimes referred to as line cards), one or more used to implement the ND control plane 624 (sometimes referred to as control cards)), alternative embodiments can combine functionality onto a single card and / or include additional card types (e.g., an additional type of card referred to as a service card, resource card, or multi-application card). A service card can provide specialized processing (e.g., layer 4 through layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), Point to Point (P2P), Voice over IP (VoIP) session border controller, mobile wireless gateway (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) gateway). By way of example, a service card can be used to terminate IPsec tunnels and perform the attendant authentication and encryption algorithms. The cards are coupled together (e.g., a first full mesh of line cards, a second full mesh of all cards) through one or more interconnection mechanisms (as represented by backplane 636).

[0099] Returning to Figure 6A, the general-purpose network device 604 includes hardware 640 comprising a set of one or more processors 642 (typically COTS processors) and physical NIs 646, and a non-transitory machine-readable storage medium 648 having stored therein software 650. During runtime, the processors 642 execute the software 650 to instantiate one or more sets of one or more applications 664A-R. While one embodiment does not implement virtualization, alternative embodiments can use different forms of virtualization. For example, in one such alternative embodiment, the virtualization layer 654 represents a kernel of an operating system (or a shim program executing on a base operating system) that allows multiple instances 662A-R, called software containers, each of which can be used to execute a set (or sets) of applications 664A-R; where the multiple software containers (also called virtualized engines, virtual private servers, or jails) are user spaces (typically virtual memory spaces) that are isolated from each other and from the kernel space running the operating system; and where a set of applications running in a given user space cannot access the memory of other processes unless explicitly allowed. In another such alternative embodiment, the virtualization layer 654 represents a system hypervisor (sometimes called a virtual machine monitor (VMM)) or a system hypervisor executing on top of a host operating system, and each of the multiple sets of applications 664A-R run on top of a guest operating system within an instance 662A-R called a virtual machine (which in some cases can be viewed as a tightly isolated form of software container) that can be unaware that it is running on a virtual machine rather than on a “bare metal” host electronic device, or through paravirtualization, the operating systems and / or applications can be aware of the presence of virtualization for optimization purposes. In other alternative embodiments, one, some or all of the applications are implemented as one or more unikernels, which can be generated by directly compiling an application with a limited set of libraries (e.g., a library operating system (LibOS) from drivers / libraries including OS services) that provide the specific OS services needed by that application. Since unikernels can be implemented to run directly on the hardware 640, to run directly on a system hypervisor (in which case unikernels are sometimes described as running in a LibOS virtual machine), or to run in a software container, embodiments can be implemented entirely through unikernels running directly on a system hypervisor represented by the virtualization layer 654, through unikernels running within software containers represented by the instances 662A-R, or as a combination of unikernels and the above techniques (e.g., unikernels and virtual machines both running directly on a system hypervisor, unikernels and multiple sets of applications running in different software containers). Note that the networking software 650 includes the cache manager 150, the operation of which is discussed herein.In some embodiments, the cache manager 150 can be instantiated in the virtualization layer 654 and / or the applications 664.

[0100] The instantiation of one or more applications 664A-R, one or more groups of applications 664A-R, and virtualization, if implemented, collectively is referred to as a software instance 652. Each group of applications 664A-R, the corresponding virtualized construct (e.g., instances 662A-R), if implemented, and that portion of the hardware 640 on which they are executed (assuming hardware dedicated to that execution and / or time slices of time-shared hardware) form an individual virtual network element 660A-R.

[0101] The virtual network elements 660A-R perform similar functions to the virtual network elements 630A-R, e.g., similar to the control communication and configuration modules 632A and forwarding tables 634A (such virtualization of the hardware 640 is sometimes referred to as network function virtualization (NFV)). Thus, NFV can be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in Data centers, NDs, and Customer Premises Equipment (CPE). While embodiments are illustrated by a one-to-one correspondence between each instance 662A-R and one VNE 660A-R, alternative embodiments can implement this correspondence at a more granular level of detail (e.g., line card virtual machines virtualize line cards, control card virtual machines virtualize control cards, etc.); it should be understood that techniques described herein with reference to this one-to-one correspondence between instances 662A-R and VNEs also apply to embodiments that use such a more granular level of detail and / or single-core.

[0102] In particular embodiments, the virtualization layer 654 includes a virtual switch that provides forwarding services similar to a physical Ethernet switch. Specifically, the virtual switch forwards traffic between instances 662A-R and the physical NIs 646 and, optionally, between instances 662A-R. Additionally, the virtual switch can enforce network isolation between VNEs 660A-R that are not allowed to communicate with each other, e.g., by adhering to virtual local area networks (VLANs).

[0103] Figure 6A A third exemplary ND implementation in the group is a hybrid network device 606 that includes both custom ASICs / dedicated OS and COTS processors / standard OS in a single ND or a single card in a ND. In particular embodiments of such hybrid network devices, a platform VM (i.e., a VM that implements the functionality of a dedicated network device 602) can provide paravirtualization for the networking hardware present in the hybrid network device 606.

[0104] Regardless of the above exemplary implementations of an ND, when considering a single VNE of the multiple VNEs implemented by the ND (e.g., only one of the VNEs is part of a given virtual network), or only a single VNE is currently implemented by the ND, a network element (NE) is sometimes used to refer to that VNE. Also in all of the above exemplary implementations, each VNE (e.g., those in VNEs 630A-R, VNEs 660A-R, and those in hybrid network device 606) receives data on a physical NI (e.g., 616, 646) and then forwards that data out an appropriate physical NI (e.g., 616, 646). For example, a VNE that implements an IP router function forwards IP packets based on certain IP header information in the IP packets; where the IP header information includes source IP address, destination IP address, source port, destination port (where "source port" and "destination port" refer to protocol ports as opposed to physical ports of the ND), transport protocol (e.g., User Datagram Protocol (UDP), Transmission Control Protocol (TCP), and Differentiated Services Code Point (DSCP) value).

[0105] Figure 6C Various exemplary ways in which VNEs can be coupled are shown in accordance with some embodiments. Figure 6C VNEs 670A.1-670A.P (and optionally VNEs 670A.Q-670A.R) implemented in ND 600A and VNE 670H.1 implemented in ND 600H are shown in Figure 6C In the middle, VNEs 670A.1-P are separate from each other in that they can receive packets from outside ND 600A and forward packets outside ND 600A; VNE 670A.1 is coupled with VNE 670H.1, and thus they transfer packets between the respective NDs; VNEs 670A.2-670A.3 can optionally forward packets between themselves without forwarding the packets outside ND 600A; and VNE 670A.P can optionally be the first in a chain of VNEs that includes VNE 670A.Q (followed by VNE 670A.R) (this is sometimes referred to as a dynamic service chain, where each VNE in the series of VNEs provides a different service, such as one or more Layer 4-7 network services). Although Figure 6C Various exemplary relationships between VNEs are shown, but alternative embodiments can support other relationships (e.g., more / less VNEs, more / less dynamic service chains, multiple different dynamic service chains with some common VNEs and some different VNEs).

[0106] For example, Figure 6ANDs can form part of the Internet or a private network; other electronic devices (not shown; e.g., end user devices including workstations, laptops, netbooks, tablets, palmtops, cellphones, smartphones, phablets, multimedia phones, Voice Over Internet Protocol (VOIP) phones, terminals, portable media players, GPS units, wearable devices, gaming systems, set-top boxes, Internet-enabled home appliances) can be coupled to the network (directly or through other networks such as access networks) to communicate with each other (directly or through servers) and / or access content and / or services on the network (e.g., the Internet or a virtual private network (VPN) overlaying (e.g., through tunneling) the Internet). Such content and / or services are typically provided by one or more servers (not shown) belonging to a service / content provider or one or more end user devices (not shown) participating in peer-to-peer (P2P) services, and can include, for example, public webpages (e.g., free content, storefronts, search services), private webpages (e.g., webpages providing username / password access to email services), and / or corporate networks over VPNs. For example, end user devices can be coupled (e.g., through customer premises equipment coupled to an access network (wired or wirelessly)) to edge NDs, which are coupled (e.g., through one or more core NDs) to other edge NDs, which are coupled to electronic devices acting as servers. However, through compute and storage virtualization, one or more electronic devices running as NDs in Figure 6A one or more software instances 662A-R can act as servers; the same is true for the hybrid network device 606; in the case of the special-purpose network device 602, one or more such servers can also run on the virtualization layer executed by the processor(s) 612); in this case, the servers are considered to be co-located with the VNEs of that ND.

[0107] A virtual network is a logical abstraction of a physical network (e.g., the physical network in Figure 6A A virtual network is a logical abstraction of a physical network (e.g., the physical network in

[0108] A network virtualization edge (NVE) resides at the edge of an underlay network and participates in implementing network virtualization; the network-facing side of an NVE uses the underlay network to tunnel frames back and forth between other NVEs; the outward-facing side of an NVE exchanges data with systems outside the network. A virtual network instance (VNI) is a particular instance of a virtual network on an NVE (e.g., a NE / VNE on an ND, a portion of a NE / VNE on an ND, where a NE / VNE is divided into multiple VNEs by emulation); one or more VNIs can be instantiated on an NVE (e.g., as different VNEs on an ND). A virtual access point (VAP) is a logical connection point on an NVE for connecting external systems to a virtual network; a VAP can be a physical or virtual port identified by a logical interface identifier (e.g., a VLAN ID).

[0109] Examples of network services include: 1) an Ethernet LAN emulation service (an Ethernet-based multipoint service similar to an Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) service), in which external systems are interconnected across a network over an underlay network through a LAN environment (e.g., an NVE provides separate L2 VNIs (virtual switching instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunnel encapsulation across the underlay network); and 2) a virtualized IP forwarding service (similar to an IETF IP VPN (e.g., Border Gateway Protocol (BGP) / MPLS IP VPN) from a service definition perspective), in which external systems are interconnected across a network over an underlay network through an L3 environment (e.g., an NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks and provides L3 (e.g., IP / MPLS) tunnel encapsulation across the underlay network). Network services can also include quality of service functionality (e.g., traffic classification marking, traffic policing and scheduling), security functionality (e.g., filters to protect customer premises from network-originated attacks to avoid malformed route advertisements), and management functionality (e.g., complete detection and handling).

[0110] Figure 6D A network with a single network element on each ND of Figure 6A is shown, and within this direct forwarding approach, a traditional distributed approach (typically used by traditional routers) is contrasted with a centralized approach for maintaining reachability and forwarding information (also referred to as network control). Specifically, Figure 6D Network elements (NEs) 670A-H with the same connectivity as NDs 600A-H of Figure 6A are shown.

[0111] Figure 6DThe distributed approach 672 is shown to distribute responsibilities for generating reachability and forwarding information across the NEs 670A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[0112] For example, if a special-purpose network device 602 is used, the control communication and configuration modules 632A-R of the ND control plane 624 typically include reachability and forwarding information modules to implement one or more routing protocols (e.g., exterior gateway protocols (such as Border Gateway Protocol (BGP)), interior gateway protocols (IGPs) (e.g., Open Shortest Path First (OSPF), Intermediate-System-to-Intermediate-System (IS-IS), Routing Information Protocol (RIP), Label Distribution Protocol (LDP), Resource Reservation Protocol (RSVP) (including RSVP Traffic Engineering (TE): RSVP extensions for LSP tunnels and Generalized Multi-Protocol Label Switching (GMPLS) signaling RSVP-TE)), to communicate with other NEs to exchange routes, and then select those routes based on one or more routing metrics. Thus, the NEs 670A-H (e.g., the processors 612 executing the control communication and configuration modules 632A-R) fulfill their responsibilities for participating in controlling how data (e.g., packets) is routed (e.g., next hop for the data and egress physical NIs for that data) by determining reachability within the network and computing their respective forwarding information distributedly. The routes and adjacencies are stored in one or more routing structures (e.g., Routing Information Base (RIB), Label Information Base (LIB), one or more adjacency structures) on the ND control plane 624. The ND control plane 624 uses the information (e.g., adjacency and routing information) based on the routing structures to program the ND forwarding plane 626. For example, the ND control plane 624 programs the adjacency and routing information into one or more forwarding tables 634A-R (e.g., Forwarding Information Base (FIB), Label Forwarding Information Base (LFIB), and one or more adjacency structures) on the ND forwarding plane 626. For Layer 2 forwarding, the ND can store one or more bridging tables that are used to forward data based on Layer 2 information in the data. Although the above example uses a special-purpose network device 602, the same distributed approach 672 can be implemented on a general-purpose network device 604 and a hybrid network device 606.

[0113] Figure 6DA centralized approach 674 (also known as software defined networking (SDN)) is shown in which the system makes decisions about where to send traffic from the underlying system that forwards traffic to selected destinations. The centralized approach 674 shown is responsible for generating reachability and forwarding information in a centralized control plane 676 (sometimes referred to as an SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization authority, or management control entity) and thus the process of neighbor discovery and topology discovery is centralized. The centralized control plane 676 has a southbound interface 682 to a data plane 680 (sometimes referred to as an infrastructure layer, network forwarding plane, or forwarding plane (which should not be confused with the ND forwarding plane)) that includes NEs 670A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 676 includes a network controller 678 that includes a centralized reachability and forwarding information module 679 that determines reachability within the network and distributes forwarding information on the southbound interface 682 (which can use the OpenFlow protocol) to the NEs 670A-H of the data plane 680. Thus, network intelligence is centralized in the centralized control plane 676 that is typically separate from the NDs. In one embodiment, the centralized reachability and forwarding information module 679 includes a cryptographic coordinator 675 that can coordinate the obtaining of key material (e.g., keys and cryptographic algorithms) necessary to secure communications between intermediate network devices and endpoint (client or server) network devices, as described above.

[0114] For example, in the case of using special-purpose network devices 602 in the data plane 680, each control communication and configuration module 632A-R of the ND control plane 624 typically includes a control agent that provides the VNE side of the southbound interface 682. In this case, the ND control plane 624 (the processor 612 executing the control communication and configuration modules 632A-R) performs its responsibility for participating in controlling how data (e.g., packets) is routed (e.g., the next hop for the data and the outbound physical NI for that data) by communicating with the centralized control plane 676 to receive forwarding information (and in some cases, reachability information) from the control agent of the centralized reachability and forwarding information module 679 (it should be understood that in some embodiments, in addition to communicating with the centralized control plane 676, the control communication and configuration modules 632A-R can also play some role in determining reachability and / or computing forwarding information, although less than in the case of a distributed approach; such embodiments are typically considered to be of the centralized approach 674, but can also be considered to be hybrid approaches).

[0115] While the above examples use specialized network devices 602, the same centralized approach 674 can be implemented with general-purpose network devices 604 and hybrid network devices 606 (e.g., each VNE 660A-R fulfills its responsibility for controlling how data (e.g., packets) is routed (e.g., next hop for the data and egress physical NI for that data) by communicating with the centralized control plane 676 to receive forwarding information (and in some cases reachability information) from the centralized reachability and forwarding information module 679); it should be understood that in some embodiments, in addition to communicating with the centralized control plane 676, the VNEs 660A-R can also play a role in determining reachability and / or computing forwarding information, albeit a lesser role than in the case of the distributed approach). In fact, the use of SDN technology can enhance the NFV technology that is typically used in general-purpose network device 604 or hybrid network device 606 implementations, as NFV can support SDN by providing the infrastructure on which the SDN software can run, and both NFV and SDN aim to leverage commodity server hardware and physical switches.

[0116] Figure 6D The centralized control plane 676 is also shown to have a northbound interface 684 to an application layer 686 in which applications 688 reside. The centralized control plane 676 has the ability to form virtual networks 692 for the applications 688 (sometimes referred to as logical forwarding planes, network services, or overlay networks (with the NEs 670A-H of the data plane 680 as the underlay network). Thus, the centralized control plane 676 maintains a global view of all NDs and configured NEs / VNEs and effectively maps virtual networks to the underlying NDs (including maintaining these mappings as the physical network changes through hardware (ND, link, or ND component) failure, addition, or removal).

[0117] While the above examples use specialized network devices 602, the same centralized approach 674 can be implemented with general-purpose network devices 604 and hybrid network devices 606 (e.g., each VNE 660A-R fulfills its responsibility for controlling how data (e.g., packets) is routed (e.g., next hop for the data and egress physical NI for that data) by communicating with the centralized control plane 676 to receive forwarding information (and in some cases reachability information) from the centralized reachability and forwarding information module 679); it should be understood that in some embodiments, in addition to communicating with the centralized control plane 676, the VNEs 660A-R can also play a role in determining reachability and / or computing forwarding information, albeit a lesser role than in the case of the distributed approach). In fact, the use of SDN technology can enhance the NFV technology that is typically used in general-purpose network device 604 or hybrid network device 606 implementations, as NFV can support SDN by providing the infrastructure on which the SDN software can run, and both NFV and SDN aim to leverage commodity server hardware and physical switches. Figure 6D A distributed approach 672 is shown separate from the centralized approach 674, but in some embodiments, the work of network control can be distributed differently, or a combination of the two. For example: 1) an embodiment can generally use the centralized approach (SDN) 674, but have specific functionality delegated to the NEs (e.g., the distributed approach can be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and / or topology discovery); or 2) an embodiment can perform neighbor discovery and topology discovery via both the centralized control plane and the distributed protocol, and compare the results to raise an exception when they do not agree. Such an embodiment would generally be considered to be of the centralized approach 674, but can also be considered to be a hybrid approach.

[0118] While the above examples use specialized network devices 602, the same centralized approach 674 can be implemented with general-purpose network devices 604 and hybrid network devices 606 (e.g., each VNE 660A-R fulfills its responsibility for controlling how data (e.g., packets) is routed (e.g., next hop for the data and egress physical NI for that data) by communicating with the centralized control plane 676 to receive forwarding information (and in some cases reachability information) from the centralized reachability and forwarding information module 679); it should be understood that in some embodiments, in addition to communicating with the centralized control plane 676, the VNEs 660A-R can also play a role in determining reachability and / or computing forwarding information, albeit a lesser role than in the case of the distributed approach). In fact, the use of SDN technology can enhance the NFV technology that is typically used in general-purpose network device 604 or hybrid network device 606 implementations, as NFV can support SDN by providing the infrastructure on which the SDN software can run, and both NFV and SDN aim to leverage commodity server hardware and physical switches. Figure 6DThe simple case of each ND 600A-H implementing a single NE 670A-H is shown, but it should be understood that references to Figure 6D The network control methods described also apply to networks in which one or more NDs 600A-H implement multiple VNEs (such as VNEs 630A-R, VNEs 660A-R, those in hybrid network device 606). Alternatively or additionally, network controller 678 can also emulate the implementation of multiple VNEs in a single ND. Specifically, in addition to (or instead of) implementing multiple VNEs in a single ND, network controller 678 can also present the implementation of a VNE / NE in a single ND as multiple VNEs in a virtual network 692 (all in the same virtual network 692, each in a different virtual network 692, or some combination). For example, network controller 678 can cause a ND to implement a single VNE (NE) in the underlay network, and then logically partition the resources of that NE within centralized control plane 676 to present different VNEs in a virtual network 692 (where these different VNEs in the overlay network are sharing the resources of the single-VNE / NE implementation on the ND in the underlay network).

[0119] In another aspect, Figure 6E and 6F Exemplary abstractions of NEs and VNEs that network controller 678 can present as part of different virtual networks 692 are shown, respectively. Figure 6E A simple case in which each ND 600A-H implements a single NE 670A-H (see Figure 6D ) but centralized control plane 676 has abstracted multiple NEs (NEs 670A-C and G-H) in different NDs into a single NE 670I in one of the virtual networks 692 (denoted as Figure 6D ) is shown. Figure 6E In this virtual network, NE 670I is shown coupled to NEs 670D and 670F, both of which are still coupled to NE 670E.

[0120] Figure 6F A case in which multiple VNEs (VNE 670A.1 and VNE 670H.1) are implemented on different NDs (ND 600A and ND 600H) and are coupled to each other and in which centralized control plane 676 has abstracted these multiple VNEs so that they appear as a single VNE 670T in one of the virtual networks 692 (denoted as Figure 6D ) is shown, according to some embodiments. Thus, the abstraction of a NE or VNE can span multiple NDs.

[0121] While some embodiments implement the centralized control plane 676 as a single entity (e.g., a single instance of software running on a single electronic device), alternative embodiments can distribute the functionality across multiple entities for redundancy and / or scalability purposes (e.g., multiple instances of software running on different electronic devices).

[0122] Similar to the network device implementations, the electronic devices running the centralized control plane 676, and thus the network controller 678 including the centralized reachability and forwarding information module 679, can be implemented in a variety of ways (e.g., a special purpose device, a general purpose (e.g., COTS) device, or a hybrid device). These electronic devices will similarly include a processor, a set of one or more physical NIs, and a non-transitory machine-readable storage medium having stored thereon the centralized control plane software.

[0123] Standards such as OpenFlow define a protocol for messages and a model for processing packets. The model for processing packets includes header parsing, packet classification, and making a forwarding decision. Header parsing describes how to interpret a packet based on a set of well-known protocols. Some protocol fields are used to construct a match structure (or key) that will be used in packet classification (e.g., the first key field can be the source Media Access Control (MAC) address, and the second key field can be the destination MAC address).

[0124] Packet classification involves performing a lookup in memory to classify a packet by determining which entry in a forwarding table (also referred to as a forwarding table entry or flow entry) matches the packet best based on the match structure or key of the forwarding table entry. Many flows represented in forwarding table entries can correspond / match to one packet; in such cases, the system is typically configured to determine one forwarding table entry from the multiple forwarding table entries according to a defined scheme (e.g., select the first forwarding table entry that matches). A forwarding table entry includes a set of specific match criteria (a set of values or wildcards, or an indication of which parts of the packet should be compared to a specific value / values / wildcard, as defined by the match function, for a particular field in the packet header, or some other packet content) and a set of one or more actions for the data plane to take when a matching packet is received. For example, for a packet using a particular port, the action can be to push a header onto the packet, flood the packet, or simply drop the packet. Thus, a forwarding table entry for IPv4 / IPv6 packets with a particular Transmission Control Protocol (TCP) destination port can contain an action specifying that these packets should be dropped.

[0125] Based on the forwarding table entry identified during packet classification, a forwarding decision is made and the actions identified in the matching forwarding table entry are performed on the packet.

[0126] However, when an unknown packet (e.g., a "miss packet" or "match miss" as used in OpenFlow parlance) arrives at the data plane 680, the packet (or a subset of the packet header and contents) is typically forwarded to the centralized control plane 676. The centralized control plane 676 then programs a forwarding table entry into the data plane 680 to accommodate packets belonging to the flow of the unknown packet. Once a particular forwarding table entry has been programmed into the data plane 680 by the centralized control plane 676, the next packet with matching credentials will match the forwarding table entry and take the set of actions associated with the matching entry.

[0127] A network interface (NI) can be physical or virtual; in the context of IP, an interface address is an IP address assigned to a NI (whether a physical NI or a virtual NI). A virtual NI can be associated with a physical NI, associated with another virtual interface, or can exist independently (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) can be numbered (with an IP address) or unnumbered (without an IP address). A loopback interface (and its loopback address) is a particular type of virtual NI (and IP address) of a NE / VNE (physical or virtual) that is often used for management purposes; such an IP address is referred to as a node loopback address. An IP address assigned to a NI of an ND is referred to as an IP address of the ND; at a more granular level, an IP address assigned to a NI (which is assigned to a NE / VNE implemented on an ND) can be referred to as an IP address of the NE / VNE.

[0128] A routing system next hop selection for a given destination can resolve to one path (i.e., a routing protocol can generate a next hop on the shortest path); however, if the routing system determines that there are multiple viable next hops (i.e., the routing protocol generated forwarding solution provides multiple next hops on the shortest path, i.e., multiple equal cost next hops), then some additional criteria are used, e.g., in connectionless networks Equal Cost Multi-Path (ECMP) (also referred to as Equal Cost Multi-Path Control, Multi-Path Forwarding, and IP Multi-Path) can be used (e.g., typical implementations use a particular header field as a criterion to ensure that packets of a particular packet flow are always forwarded on the same next hop to maintain packet flow ordering). A packet flow is defined for multi-path forwarding purposes as a set of packets that share ordering constraints. As one example, a set of packets in a particular TCP transport sequence need to arrive in order, otherwise TCP logic interprets out-of-order delivery as congestion and slows down the TCP transport rate.

[0129] Layer 3 (L3) link aggregation (LAG) links are links that directly connect two NDs with multiple IP addressed link paths (each link path is assigned a different IP address) and perform load distribution decisions across these different link paths at the ND forwarding plane; in this case, load distribution decisions will be made between the link paths.

[0130] Some NDs include functionality for authentication, authorization, and accounting (AAA) protocols (e.g., RADIUS (Remote Authentication Dial-In User Service), Diameter, and / or TACACS+ (Terminal Access Controller Access-Control System Plus)). AAA can be provided through a client / server model, where the AAA client is implemented on the ND, and the AAA server can be implemented locally on the ND or on a remote electronic device coupled to the ND. Authentication is the process of identifying and verifying a subscriber. For example, a subscriber can be identified by a combination of a username and password or by a unique key. Authorization determines what a subscriber can do after being authenticated, such as gaining access to certain electronic device information resources (e.g., through the use of access control policies). Accounting is the process of recording user activity. By way of a summary example, an end user device can be coupled (e.g., through an access network) through an edge ND (supporting AAA processing) coupled to a core ND coupled to electronic devices implementing servers of a service / content provider. The AAA processing is performed to retrieve a subscriber record stored in the AAA server for the subscriber. The subscriber record includes a set of attributes (e.g., subscriber name, password, authentication information, access control information, rate limit information, policing information) used during the processing of the subscriber's traffic.

[0131] A particular ND (e.g., a particular edge ND) represents end-user devices (or sometimes customer premises equipment (CPE), such as residential gateways (e.g., routers, modems)) that use subscriber circuits internally. A subscriber circuit uniquely identifies a subscriber session within the ND, and typically exists for the life of the session. Thus, the ND typically allocates a subscriber circuit when a subscriber connects to the ND, and deallocates the subscriber circuit accordingly when the subscriber disconnects. Each subscriber session represents a distinguishable flow of packets communicated between the ND and an end-user device (or sometimes CPE, such as a residential gateway or modem) using a protocol, such as a point-to-point protocol (PPPoX) (e.g., where X is Ethernet or asynchronous transfer mode (ATM)) based on another protocol, Ethernet, 802.1Q virtual LAN (VLAN), Internet Protocol, or ATM. Subscriber sessions can be initiated using various mechanisms (e.g., manually provisioned dynamic host configuration protocol (DHCP), DHCP / Clientless Internet Protocol Service (CLIPS), or media access control (MAC) address tracking). For example, point-to-point protocol (PPP) is commonly used for digital subscriber line (DSL) services, and requires installation of a PPP client that enables a subscriber to enter a username and password, which in turn can be used to select a subscriber record. When DHCP is used (e.g., for cable modem services), a username is typically not provided; but in this case, other information is provided (e.g., information including the MAC address of hardware in the end-user device (or CPE)). Using DHCP and CLIPS on the ND captures the MAC address, and these addresses are used to distinguish subscribers and access their subscriber records.

[0132] Virtual circuits (VCs), synonymous with virtual connections and virtual channels, are connection-oriented communication services that are transported with the aid of packet-mode communication. Virtual circuit communication is similar to circuit switching in that both are connection-oriented, which means that in both cases data is transported in the correct order and requires signaling overhead in the connection setup phase. Virtual circuits can exist at different layers. For example, at layer 4, connection-oriented transport layer data link protocols such as the Transmission Control Protocol (TCP) can rely on a connectionless packet-switched network layer protocol such as IP, where different packets can be routed on different paths and thus transported out of order. In the case of TCP used on top of an unreliable and connectionless IP protocol to establish a reliable virtual circuit, the virtual circuit is identified by a source and destination network socket address pair (i.e., the sender and receiver IP addresses and port numbers). However, virtual circuits are possible because TCP includes a segment number and reordering on the receiver side to prevent out-of-order delivery. Virtual circuits are also possible at layer 3 (network layer) and layer 2 (data link layer); such virtual circuit protocols are based on connection-oriented packet switching, which means that data is always transported along the same network path (i.e., through the same NEs / VNEs). In such protocols, packets are not individually routed and full addressing information is not provided in the header of each data packet; only a small virtual channel identifier (VCI) is needed in each packet; and routing information is transferred to the NEs / VNEs in the connection setup phase; switching involves only looking up the virtual channel identifier in a table, rather than analyzing full addresses. Examples of network layer and data link layer virtual circuit protocols where data is always transported on the same path are: X.25, where a VC is identified by a virtual channel identifier (VCI); Frame Relay, where a VC is identified by a VCI; Asynchronous Transfer Mode (ATM), where a circuit is identified by a virtual path identifier (VPI) and virtual channel identifier (VCI) pair; General Packet Radio Service (GPRS); and Multiprotocol Label Switching (MPLS), which can be used for IP over virtual circuits (each circuit is identified by a label).

[0133] Certain NDs (e.g., certain edge NDs) use a circuit hierarchy. Leaf nodes of the circuit hierarchy are subscriber circuits. Subscriber circuits have parent circuits in the hierarchy, which typically represent an aggregation of multiple subscriber circuits, and thus represent a network segment and network element used to provide the ND with an access network connection to these end user devices. These parent circuits can represent a physical or logical aggregation of subscriber circuits (e.g., a virtual local area network (VLAN), a permanent virtual circuit (PVC) (e.g., for asynchronous transfer mode (ATM)), a circuit group, a channel, a pseudo-wire, a physical NI of the ND, and a link aggregation group). A circuit group is a virtual construct that allows grouping of various circuit components together for configuration purposes; for example, aggregate rate control. A pseudo-wire is an emulation of a layer 2 point-to-point connection-oriented service. A link aggregation group is a virtual construct that consolidates multiple physical NIs for bandwidth aggregation and redundancy. Thus, parent circuits encapsulate subscriber circuits, either physically or logically.

[0134] Each VNE (e.g., a virtual router, a virtual bridge (which can act as a virtual switch instance in a virtual private LAN service (VPLS)) is typically independently manageable. For example, in the case of multiple virtual routers, each virtual router can share system resources but is segregated from other virtual routers in its administrative domain, AAA (authentication, authorization, and accounting) name space, IP address, and routing database. Multiple VNEs can be employed in an edge ND to provide direct network access and / or different classes of service for subscribers of a service and / or content provider.

[0135] Within certain NDs, an "interface" independent of a physical NI can be configured as part of a VNE to provide higher layer protocol and service information (e.g., layer 3 addressing). In addition to other subscriber configuration requirements, a subscriber record in an AAA server identifies to which context (e.g., which VNE / NE) a corresponding subscriber should be bound within the ND. As used herein, binding forms an association between a physical entity (e.g., a physical NI, a channel) or a logical entity (e.g., a circuit, such as a subscriber circuit or a logical circuit (a group of one or more subscriber circuits)) and a context interface through which network protocols (e.g., routing protocols, bridging protocols) are configured for that context. When some higher layer protocol interfaces are configured and associated with a physical entity, subscriber data flows over that physical entity.

[0136] Note that an electronic device uses machine-readable media, also known as computer-readable media, such as machine-readable storage media (e.g., magnetic disks, optical disks, solid state drives, read-only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustic, or other form of propagated signals, such as carrier waves, infrared signals), to store and transmit (internally and / or transmissively between other electronic devices) code (which includes software and is sometimes called computer program code or a computer program) and / or data. Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., where the processor is a microprocessor, a controller, a microcontroller, a central processing unit, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), other electronic circuitry, a combination of one or more of the above items), coupled to one or more machine-readable storage media to store code for execution on the set of processors and / or to store data. For example, an electronic device can include non-volatile memory that contains code because the non-volatile memory can retain the code / data even when the electronic device is turned off (when power is removed). The portion of the code that the processor(s) of the electronic device is to execute when the electronic device is turned on is typically copied from the slower non-volatile memory into the faster volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of the electronic device. A typical electronic device also includes a set of one or more physical network interfaces (NIs) to establish network connections with other electronic devices (to send and / or receive code and / or data using propagated signals). For example, the set of physical NIs (or the set of physical NIs in combination with the set of processors executing code) can perform any formatting, encoding, or converting to allow the electronic device to send and receive data over wired and / or wireless connections. In some embodiments, the physical NIs can include radio circuitry capable of (1) receiving data from other electronic devices over wireless connections and / or (2) sending data to other devices over wireless connections. The radio circuitry can include a transmitter, a receiver, and / or a transceiver suitable for radio frequency communications. The radio circuitry can convert digital data into radio signals having appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signals can then be transmitted through an antenna to the appropriate recipient. In some embodiments, the set of physical NIs can include a network interface controller (NIC) (also called a network interface card, network adapter, or local area network (LAN) adapter). The NIC can facilitate connecting the electronic device to other electronic devices, allowing them to communicate using wires by plugging cables into physical ports connected to the NIC. Different combinations of software, firmware, and / or hardware can be used to implement one or more parts of an embodiment.

[0137] Network nodes / devices are electronic devices. Some network devices are “multiservice network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, quality of service, and / or subscriber management), and / or provide support for multiple application services (e.g., data, voice, and video). Examples of network nodes also include NodeB, base station (BS), multi-standard radio (MSR) radio node such as MSR BS, eNodeB, gNodeB, MeNB, SeNB, integrated access backhaul (IAB) node, network controller, radio network controller (RNC), base station controller (BSC), relay, donor node controlled relay, base transceiver station (BTS), central unit (e.g., in a gNB), distributed unit (e.g., in a gNB), baseband unit, centralized baseband, C-RAN, access point (AP), transmission points, transmission nodes, RRU, RRH, nodes in distributed antenna system (DAS), core network nodes (e.g., MSC, MME, etc.), O&M, OSS, SON, positioning nodes (e.g., E-SMLC), etc.

[0138] A wireless network (also referred to as a cellular network) is a network of devices that communicate using radio waves (electromagnetic waves within the 30 KHz - 300 GHz frequency range). Wireless communication can follow a wireless communication standard, such as New Radio (NR), LTE (Long-Term Evolution), LTE-Advanced (LTE-A), Wideband Code-Division Multiple Access (WCDMA), High-Speed Packet Access (HSPA), WiFi (Wireless Fidelity), and Bluetooth. Further, communication between devices (e.g., network devices and mobile devices in a wireless communication network) can be performed in accordance with any suitable generation of communication protocols, including but not limited to first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G) communication protocols, and / or any other protocols that are currently known or that future become known.

[0139] A communication network includes one or more wireless and / or wired networks and it includes networks such as a wide-area network (WAN), metropolitan area network (MAN), local area network (LAN), Internet area network (IAN), campus area network (CAN), and virtual private network (VPN). The communication network can implement various management interfaces and protocols as discussed herein.

[0140] Alternative embodiments

[0141] While the application has been described in terms of numerous embodiments, those skilled in the art will recognize that the application can be practiced with modifications and alterations limited only by the spirit and scope of the appended claims. Accordingly, the description is to be regarded as illustrative in nature and not restrictive.

[0142] Embodiments

[0143] Embodiments of the application include:

[0144] 1. A method performed by a network device, the method comprising:

[0145] determining (302) that a first cache of the network device is overloaded, wherein the first cache is within a cache hierarchy that includes a plurality of caches, each cache including a set of forwarding table entries for use by the network device in forwarding packets of a traffic flow, and wherein a lower cache in the cache hierarchy is to be examined earlier in a flow lookup than a higher cache in the cache hierarchy; and

[0146] upon determining (304) that the first cache is overloaded,

[0147] disabling the first cache for use in flow lookups, wherein the flow lookups skip the first cache and are performed at a second cache higher in the cache hierarchy, and

[0148] disabling the first cache for use in forwarding table entry insertion when the forwarding table entry insertion is performed in one or more of the plurality of caches.

[0149] 2. The method of embodiment 1, wherein the first cache is determined to be overloaded based on one or more of a first count of cache evictions, a second count of forwarding table entry insertions, and a third count of cache hits.

[0150] 3. The method of embodiment 2, wherein the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold.

[0151] 4. The method of embodiment 2, wherein the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold for a plurality of consecutive intervals.

[0152] 5. The method of embodiment 1, further comprising:

[0153] after the first time period, enabling (310) the first cache to perform flow lookups when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions is below a second threshold for a second time period.

[0154] evicting (308) one or more forwarding table entries within the first cache when the first cache is deemed full and a forwarding table entry is to be inserted into the first cache.

[0155] 6. The method of embodiment 5, further comprising:

[0156] enabling (310) the first cache to perform flow lookups when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions is below a second threshold for a second time period.

[0157] 7. The method of embodiment 5, further comprising:

[0158] enabling (310) the first cache to perform flow lookups when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions is below a second threshold for a second time period.

[0159] 8. The method of embodiment 5, further comprising:

[0160] disabling (312) the first cache for forwarding table entry insertions when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions exceeds a second threshold for a configured time period; and

[0161] waiting (314) a third time period before enabling the first cache to perform forwarding table entry insertions when performing forwarding table entry insertions in one or more of the plurality of caches.

[0162] 9. A network device (602, 604), comprising:

[0163] a processor (612, 642) and a non-transitory computer-readable storage medium (618, 648) that provides instructions that, when executed by the processor, cause the network device to perform the method 1-8.

[0164] 10. A non-transitory computer-readable storage medium (618, 648) that provides instructions that, when executed, cause a network device to perform the method 1-8.

Claims

1. A method performed by a network device, the method comprising: determining (302) that a first cache of the network device is overloaded, wherein the first cache is within a cache hierarchy that includes a plurality of caches, each cache includes a set of forwarding table entries for the network device to forward packets of traffic flows, and wherein a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy; and upon determining (304) that the first cache is overloaded, disabling the first cache for the flow lookup, wherein the flow lookup skips the first cache and is performed at a second cache that is higher in the cache hierarchy, and upon performing a forwarding table entry insertion in one or more of the plurality of caches, disabling the first cache for the forwarding table entry insertion; after a first period of time, upon performing the forwarding table entry insertion in one or more other of the plurality of caches, enabling (306) the first cache to perform the forwarding table entry insertion; and upon the first cache being deemed full and a forwarding table entry is to be inserted into the first cache, evicting (308) one or more forwarding table entries within the first cache.

2. The method of claim 1, wherein, the first cache is determined to be overloaded based on evaluating one or more of a first count of cache evictions, a second count of forwarding table entry insertions, and a third count of cache hits.

3. The method of claim 2, wherein, the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold.

4. The method of claim 2, wherein, the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold for a plurality of consecutive intervals.

5. The method of claim 1, further comprising: upon a second measure of the first count of cache evictions and the second count of forwarding table entry insertions being below a second threshold for a second period of time, enabling (310) the first cache to perform the flow lookup.

6. The method of claim 1, further comprising: upon a second measure of the first count of cache evictions and the second count of forwarding table entry insertions being below a second threshold for a plurality of consecutive intervals, enabling (310) the first cache to perform the flow lookup.

7. The method of claim 1, further comprising: upon a second measure of the first count of cache evictions and the second count of forwarding table entry insertions exceeding a second threshold for a configured period of time, disabling (312) the first cache for the forwarding table entry insertion; and upon performing the forwarding table entry insertion in one or more of the plurality of caches, waiting (314) for a third period of time before enabling the first cache to perform the forwarding table entry insertion.

8. A network device (602, 604), comprising: ​ Processor (612, 642) and non-transitory computer-readable storage medium (618, 642) providing instructions 648), the instruction, when executed by the processor, causes the network device to perform: Determining (302) that the first cache of the network device is overloaded, wherein the first cache is within a cache hierarchy comprising multiple caches, each cache comprising a set of forwarding table entries for the network device to forward packets of traffic flows, and wherein lower caches in the cache hierarchy are checked earlier in flow lookups compared to higher caches in the cache hierarchy; and After determining (304) that the first cache is overloaded, The first cache is disabled for the streaming lookup, wherein the streaming lookup skips the first cache and is performed at a higher cache location in the cache hierarchy, and When a forwarding table entry insertion is performed in one or more of the plurality of caches, the first cache is disabled for the forwarding table entry insertion; After the first time period, when the forwarding table entry insertion is performed in one or more other caches among the plurality of caches, the first cache is allowed (306) to perform the forwarding table entry insertion; and When the first cache is considered full and a forwarding table entry is to be inserted into the first cache, one or more forwarding table entries in the first cache are evicted (308).

9. The network device of claim 8, wherein, The first cache is determined to be overloaded based on one or more of the following: a first count of cache evictions, a second count of forwarding table entry insertions, and a third count of cache hits.

10. The network device of claim 9, wherein, The first cache is determined to be overloaded when a first measurement of the first count, the second count, and the third count exceeds a first threshold.

11. The network device of claim 9, wherein, The first cache is determined to be overloaded when the first measurement of the first count, the second count, and the third count exceeds a first threshold within a plurality of consecutive intervals.

12. The network device of claim 8, wherein, The instructions, when executed by the processor, can also cause the network device to perform: When the second measurement of the first count of cache eviction and the second count of forwarding table entry insertion is lower than the second threshold during the second time period, the first cache is enabled (310) to perform the stream lookup.

13. The network device of claim 8, wherein, The instructions, when executed by the processor, can also cause the network device to perform: When a second measurement of the first count of cache eviction and the second count of forwarding table entry insertion is below a second threshold over multiple consecutive intervals, the first cache is enabled (310) to perform the stream lookup.

14. The network device of claim 8, wherein, The instructions, when executed by the processor, can also cause the network device to perform: When a second measurement of the first count of cache eviction and the second count of forwarding table entry insertion exceeds a second threshold within a configured time period, the first cache is prohibited (312) from being used for the forwarding table entry insertion; as well as when performing the forwarding table entry insertion in one or more of the plurality of caches, the first cache is disabled for the forwarding table entry insertion until a third period of time has elapsed.

15. A non-transitory computer-readable storage medium (618, 648) storing instructions that, when executed by a processor of a network device, are capable of causing the network device to perform: determining (302) a first cache overloading of the network device, wherein the first cache is within a cache hierarchy comprising a plurality of caches, each cache comprising a set of forwarding table entries for the network device to forward packets of traffic flows, and wherein a lower cache in the cache hierarchy is to be checked earlier in a flow lookup than a higher cache in the cache hierarchy; and upon determining (304) that the first cache is overloaded, disabling the first cache for the flow lookup, wherein the flow lookup skips the first cache and is performed at a second cache higher in the cache hierarchy, and when performing the forwarding table entry insertion in one or more of the plurality of caches, the first cache is disabled for the forwarding table entry insertion; after a first period of time, when performing the forwarding table entry insertion in one or more other of the plurality of caches, enabling (306) the first cache to perform the forwarding table entry insertion; and when the first cache is deemed full and a forwarding table entry is to be inserted into the first cache, evicting (308) one or more forwarding table entries within the first cache.

16. The non-transitory computer-readable storage medium of claim 15, wherein, the first cache is determined to be overloaded based on evaluating one or more of a first count of cache evictions, a second count of forwarding table entry insertions, and a third count of cache hits.

17. The non-transitory computer-readable storage medium of claim 16, wherein, the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold.

18. The non-transitory computer-readable storage medium of claim 16, wherein, the first cache is determined to be overloaded when a first measure of the first count, the second count, and the third count exceeds a first threshold for a plurality of consecutive intervals.

19. The non-transitory computer-readable storage medium of claim 15, wherein, the instructions, when executed by the processor, are further capable of causing the network device to perform: when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions is below a second threshold for a second period of time, enabling (310) the first cache to perform the flow lookup.

20. The non-transitory computer-readable storage medium of claim 15, wherein, the instructions, when executed by the processor, are further capable of causing the network device to perform: when a second measure of the first count of cache evictions and the second count of forwarding table entry insertions is below a second threshold for a plurality of consecutive intervals, enabling (310) the first cache to perform the flow lookup.

21. The non-transitory computer-readable storage medium of claim 15, wherein, the instructions, when executed by the processor, are further capable of causing the network device to perform: when a second measure of a first count of cache evictions and a second count of forwarding table entry insertions exceeds a second threshold over a configured period of time, disabling (312) the first cache for the forwarding table entry insertions; and when performing the forwarding table entry insertions in one or more of the plurality of caches, waiting (314) a third period of time before allowing the first cache to perform the forwarding table entry insertions.

Images