Flow Monitoring in Network Devices

Through the multi-level memory architecture and dynamic management of packet processors, the memory overflow problem of network devices when processing large amounts of streaming data is solved, efficient storage and update of streaming state information is realized, and the scalability and efficiency of the device are improved.

CN112825507BActive Publication Date: 2024-11-08MARVELL ISRAEL (M L S L) LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010297209.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-03-25
Filing Date
2020-04-15
Publication Date
2024-11-08
Estimated Expiration
2040-04-15

AI Technical Summary

Technical Problem

When existing network devices store and manage flow state information, it is difficult for them to effectively process large amounts of streaming data, resulting in memory overflow or need to frequently expand expensive memory, affecting the scalability and efficiency of the device.

Method used

Using a multi-level memory architecture, the packet processor dynamically manages the memory, and frees up space for new stream data by transferring older stream state information from faster memory to slower but large-capacity memory, thus realizing the stream state information. Efficient storage and updates.

Benefits of technology

The stream state information management of network devices between different levels of memory is realized, ensuring the efficiency and scalability of the device when processing large amounts of streams, and avoiding frequent expansion of faster but costly memory.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112825507B_ABST
    Figure CN112825507B_ABST
Patent Text Reader

Abstract

Based on a packet entering the network device, flow state information stored in a first memory of a plurality of memories is updated, and the plurality of memories are used to maintain flow state information at the network device. The memories are arranged in a hierarchical arrangement, in which memories at progressively higher hierarchical levels are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device. When it is determined that the fullness level of the first memory exceeds a first threshold, flow state information associated with at least one flow in the first flow set is transferred from the first memory to a second memory, and the flow state information for the first flow set is currently being maintained in the first memory, and the second memory is at a higher hierarchical level than the first memory. A new flow is instantiated in the space released in the first memory.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 938,836, filed on November 21, 2019, entitled “Flow Analysis and Management,” the disclosure of which is expressly incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to communication networks, and more particularly to flow information collection and management in network devices. Background Art

[0004] It is often useful to obtain traffic flow measurements in a communication network. For example, such information about a flow can be used to characterize the flow using parameters such as flow duration, flow rate, time, burstiness, etc. Traffic flow measurements can be made by a network device (such as a bridge, switch, or router, etc.) or some other measurement device. These measurements can then be used for various processes such as traffic metering, traffic analysis, traffic engineering, attack / intrusion detection, billing, QoS verification, etc. For example, a traffic analysis application can utilize flow measurements made at multiple different nodes in the network (e.g., routers, bridges, switches, etc.) in order to analyze traffic in different parts of the network.

[0005] Conventional network devices update flow state information maintained in a flow state information memory that supports line-speed updates. The network device periodically transmits the flow state information stored in the flow state information memory to a collector device, which typically collects such information from multiple network devices in the network. The flow state information is typically output from the network device to the collector device periodically, for example, according to a schedule defined by a flow monitoring protocol utilized by the network device. Once the flow state information stored in the flow state information memory of the network device is output to the collector device, the space released in the flow state information memory can be used to maintain flow state information for new flows processed by the network device. The flow state information memory in the network device typically needs to be both fast enough to allow the flow state information to be updated at line speed and large enough to maintain the flow state information for a sufficient number of flows that can be processed by the network device within the time interval between the cycles of outputting the flow state information to the collector device. Summary of the invention

[0006] In one embodiment, a method for monitoring flows in a network device includes: receiving, at a packet processor of the network device, packets entering via a plurality of network interfaces of the network device; using the packet processor, classifying the packets into corresponding flows based at least in part on information in corresponding headers of the packets; using the packet processor, based on the received packets, updating flow state information associated with the corresponding flows corresponding to the received packets, the flow state information being stored in a first memory of a plurality of memories, the plurality of memories being used to maintain the flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement, in which memories at progressively higher hierarchical levels are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a layer A first hierarchical level of the hierarchical arrangement; and dynamically managing multiple memories using a packet processor, including: using the packet processor to determine that a full level of the first memory exceeds a first threshold, in response to determining that the full level of the first memory exceeds the first threshold, transferring flow state information associated with at least one flow in a first flow set from the first memory to a second memory among the multiple memories to release space in the first memory, the flow state information for the first flow set is currently being maintained in the first memory, the second memory is at a second hierarchical level, the second hierarchical level is higher than the first hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the first memory to the second memory, instantiating a new flow in the space released in the first memory using the packet processor.

[0007] In another embodiment, a network device includes a plurality of network interfaces, and a packet processor coupled to the plurality of network interfaces. The packet processor is configured to: receive packets incoming via the plurality of network interfaces of the network device, classify the packets into respective flows based at least in part on information in respective headers of the received packets, update flow state information associated with the respective flows corresponding to the received packets based on the received packets, the flow state information being stored in a first memory of a plurality of memories, the plurality of memories being used to maintain the flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement, the memories at progressively higher hierarchical levels in the hierarchical arrangement being configured to maintain the flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a first hierarchical level of the hierarchical arrangement, and dynamically The invention manages multiple memories, and the packet processor is configured to: determine that a full level of a first memory exceeds a first threshold; in response to determining that the full level of the first memory exceeds the first threshold, transfer flow state information associated with at least one flow in a first flow set from the first memory to a second memory among the multiple memories to release space in the first memory, the flow state information for the first flow set is currently being maintained in the first memory, the second memory is at a second hierarchical level, the second hierarchical level is higher than the first hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the first memory to the second memory, instantiate a new flow in the space released in the first memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of an example network device configured for flow monitoring according to one embodiment.

[0009] Figure 2 According to one embodiment, Figure 1 Figure 1 is a diagram of an example flow state information table storing flow state information in a network device of FIG. 1 ;

[0010] Figure 3 According to one embodiment, Figure 1 Block diagram of a flow monitoring system for network devices.

[0011] Figure 4 According to another embodiment, a method for monitoring Figure 1 A flow chart of an example method for managing flows in a network device. DETAILED DESCRIPTION

[0012] In the embodiments described below, a network device utilizes several hierarchical levels of memory to maintain flow state information corresponding to the flows processed by the network device. In one embodiment, the memory on the gradually increasing hierarchical level is gradually larger, slower and less expensive. In one embodiment, the memory on the gradually increasing hierarchical level is configured to maintain the gradually larger latest and most active flow set being processed by the network device at any given time. In one embodiment, the network device is configured to manage the memory space of the memory based on the corresponding full level of monitoring one or more memories at different hierarchical levels of the memory. In one embodiment, when the memory of the lower hierarchical level reaches or approaches full capacity, the network device transfers the flow state information associated with one or more flows in the memory of the lower hierarchical level to the memory of the next hierarchical level. In one embodiment, then, the space released in the memory of the lower hierarchical level is used to instantiate newer flows in the memory of the lower hierarchical level. In at least some embodiments, transferring flow state information associated with older flows from a relatively lower hierarchy level of memory to a relatively higher hierarchy level of memory allows a network device to be scaled to maintain flow state information for any given number of flows, such as based on the location of the network device in the network, while maintaining flow state information associated with the newest and most active flows in faster and more expensive memory at a lower hierarchy level without having to expand other more expensive memory at the lower hierarchy level.

[0013] Figure 1 is a block diagram of an example network device 100 (such as a bridge, switch, router, etc.) including capabilities for flow monitoring according to one embodiment. The network device 100 includes a plurality of network interfaces 108 (sometimes referred to herein as "ports") configured to be coupled to a network communication link. The network device 100 also includes a packet processor 112 coupled to the plurality of ports 108. In one embodiment, the packet processor 112 is implemented on a single integrated circuit (IC) (sometimes referred to as a "chip"). For example, the packet processor 112 corresponds to a "system on a chip" (SoC) that integrates various components of the network device 100 (including the embedded processor 122) onto a single IC. In another embodiment, the packet processor 112 is implemented on a plurality of ICs corresponding to a multi-chip module (MCM), in which a plurality of ICs are integrated, for example, onto a unified substrate and integrated into a single IC package having IC pins. In one embodiment, the plurality of ICs are connected together internally (e.g., within the IC package) by fine wires and / or with conductors embedded in the unified substrate. In yet another embodiment, the packet processor 108 is implemented on multiple ICs corresponding to a system in a package (SiP). A SiP is generally similar to an MCM, but with a SiP, multiple ICs can be stacked vertically or tiled horizontally within a single IC package.

[0014] The packet processor 112 includes a forwarding engine 114 coupled to the plurality of ports 108. The forwarding engine 114 is configured to forward packets between the plurality of ports 108, for example, based on analyzing information in the header of the packet (e.g., one or more of a destination address, a virtual local area network (VLAN) identifier, etc.). The packet processor 112 also includes a flow monitoring engine 116. In one embodiment, the flow monitoring engine 116 includes a flow classifier 118 and a flow state update engine 116. The flow classifier 118 is configured to classify packets into flows, and the packet processor 112 is configured to maintain flow state information about the flows. A flow corresponds to a series of related packets. As an example, some flows (such as Internet Protocol (IP) Transmission Control Protocol (TCP) / User Datagram Protocol (UDP) flows) are typically defined in the network industry by a 5-tuple, such as {destination IP address, source IP address, L4 protocol, UDP / TCP destination port, UDP / TCP source port}. In other examples, a flow is identified only by a specific source address and / or a specific destination address in the header of the packet. For example, all packets with a specific IP source address correspond to a specific flow, or all packets with a specific IP destination address correspond to a specific flow. As another example, all packets with both a specific IP source address and a specific IP destination address correspond to a specific flow. Similarly, packets with a specific media access control (MAC) source address and / or a specific MAC destination address correspond to a specific flow. Additionally or alternatively, other information in the packet header is used to define the flow, such as one or more of a specific packet type, a specific virtual local area network (VLAN) identifier (ID), etc.

[0015] The network device 100 can process different numbers of flows in a given time, for example, depending on the location of the network device 100 in the network. By way of example only, if the network device 100 is used as an edge router in a network, the network device 100 can process a relatively large number of flows in a given time compared to network devices that are not located at the edge of the network. In one embodiment, the network device 100 utilizes a hierarchical arrangement of memory 130, wherein the memory 130 at the lowest level in the hierarchy is the fastest, most expensive, and has the smallest storage capacity, and the memory 130 at progressively higher levels in the hierarchy is progressively slower, less expensive, and has a larger storage capacity. As described in more detail below, the network device 100 is configured to utilize memory at different hierarchical levels as needed, extending to monitor any suitable number of flows processed by the network device 100, for example, in a given time, depending on the location of the network device 100 in the network.

[0016] In one embodiment, one or more memories 130 are internal to the packet processor 112. For example, in the illustrated embodiment, the packet processor 112 includes a first memory 130-1 and a second memory 130-2 that are internal to the packet processor 112 (e.g., implemented on the same chip as the packet processor 112). Figure 1 100 as including two memories 130 internal to the packet processor 112, but in other embodiments, the packet processor 112 includes other numbers (e.g., 3, 4, 5, etc.) of memories 130 internal to the packet processor 112. In another embodiment, the packet processor 112 includes only a single memory 130 internal to the packet processor 112. In one embodiment, the network device 100 also includes at least one memory 130 external to the packet processor 112. For example, in one embodiment, the network device 100 includes a third memory 130-3 external to the packet processor 112. In one embodiment, the memory 130-3 is implemented on one or more chips separate from the chip on which the packet processor 112 is implemented. In one embodiment, the memory 130-3 is coupled to an external processor 138, such as a central processing unit (CPU) of the network device 100.

[0017] In one embodiment, the memory 130 is arranged in a hierarchical arrangement, in which the memory 130 at the lowest level in the hierarchy is the fastest (e.g., has the highest memory access bandwidth), the most expensive and has the smallest storage capacity, and the memory at the progressively higher levels in the hierarchy is progressively slower, less expensive and has a larger storage capacity. In one embodiment, the memory 130-1 is at the first lowest level of the hierarchy, the memory 130-2 is at the second level of the hierarchy, which is the next level relative to the first level of the hierarchy, and in the illustrated embodiment, the memory 130-3 is at the third highest level of the hierarchy. In one embodiment, the memory 130-1 is a ternary content addressable memory (TCAM) or another type of memory configured to operate at line speed. In one embodiment, the memory 130-2 is a memory that is slower than the memory 130-1 but has a large storage capacity. In various embodiments, the memory 130-2 is a random access memory (RAM) or another suitable type of memory. Similarly, in one embodiment, the memory 130-3 is slower than the memory 130-2 and has a large storage capacity. In one embodiment, memory 130 - 3 is included in a database 140 coupled to CPU 138 of network device 100 .

[0018] As described in more detail below, packet processor 112 is configured to manage flow state information in memory 130 such that memory 130 at the lowest hierarchical level stores flow state information corresponding to a first set of the newest and / or most active flows being processed by packet processor 112, and, as needed, one or more memories at higher hierarchical levels are utilized to store progressively larger sets of the newest and / or most active flows processed by packet processor 112. In one embodiment, because memory 130 at the lowest hierarchical level stores the most active and / or recent flows processed by packet processor 112, packet processor 112 is able to update flow state information corresponding to the most active and / or recent flows at line speed, while scaling up as needed to maintain flow state information corresponding to a larger set of the most active and / or recent flows in one or more progressively slower and larger memories 130 without having to expand memory 130 at lower hierarchical levels. Thus, in one embodiment, network device 100 is scalable to use a minimum hierarchical level to maintain flow state information corresponding to a suitable maximum number of flows that can be processed by network device 100 without having to expand fixed-size memories at lower hierarchical levels.

[0019] Continue to refer Figure 1 , the flow monitoring engine 116 typically determines whether the incoming packet belongs to an existing flow (i.e., a flow previously identified by the flow monitoring engine and a flow currently known to the flow monitoring engine 116) or to a currently unknown flow (sometimes referred to as a "new flow" in this disclosure). For example, the flow monitoring engine 116 classifies the packet into a flow and checks whether there is currently an entry corresponding to the flow in the memory 130-1. In one embodiment, when the flow monitoring engine 116 determines that the incoming packet does not belong to an existing flow, for example, when there is currently no entry corresponding to the flow in the flow information state table stored in the flow memory 130-1, the flow is instantiated in the memory 130-1. In one embodiment, instantiating the flow in the memory 130-1 involves associating an entry of the flow information state table used to maintain the flow state information in the memory 130-1 with the flow, for example, by adding information identifying the flow to the flow identification field in the entry in the memory 130-1. In one embodiment, the flow monitoring engine 116 instantiates a new flow in the flow memory 130-1. In another embodiment, another entity (e.g., embedded processor 122 or CPU 138) instantiates a new flow in flow memory 130. For example, when flow monitoring engine 116 determines that an incoming packet does not belong to a known flow, flow monitoring engine 116 mirrors the packet to embedded processor 122 or CPU 138, and embedded processor 122 or CPU 138 instantiates a new flow in memory 130-1.

[0020] On the other hand, according to one embodiment, when the flow monitoring engine 116 determines that the incoming packet belongs to an existing flow, for example, when there is currently an entry corresponding to the flow in the memory 130-1, the flow monitoring engine 116 updates the flow state information in (or associated with) the entry corresponding to the flow based on the incoming packet. For example, in one embodiment, the flow monitoring engine 116 reads the current flow state information from the entry corresponding to the flow in the memory 130-1, and updates the value of one or more fields in the entry (e.g., number of bytes, number of packets, time of the last packet, etc.) based on the received packet.

[0021] In one embodiment, the memory 130-1 supports the following operations: i) identifying a flow in the memory 130-1, and ii) updating the flow state information of the identified flow at line speed based on the incoming packets corresponding to the flow. However, in one embodiment, the memory 130-1 is limited to maintaining a maximum of k flows, where k is a positive integer. In one embodiment, in order to address the limitation on the number of flows that can be maintained in the memory 130-1, the network device 100 is configured to monitor the full level of the memory 130-1, and when the full level of the memory 130-1 exceeds a threshold, the flow state information corresponding to one or more flows is transferred from the memory 130-1 to the memory 130-2. In one embodiment, for example, the embedded processor 122 is configured to monitor the full level of the memory 130-1, and when the full level of the memory 130-1 exceeds a threshold, the flow state information corresponding to one or more flows is transferred from the memory 130-1 to the memory 130-2. In one embodiment, when the number of available entries in the flow state information table in memory 130-1 for holding flow state information is zero or almost zero (e.g., 1, 2, 3, etc.), embedded processor 122 determines that the full level of memory 130-1 exceeds a threshold. In other embodiments, other suitable memory full criteria are utilized.

[0022] In one embodiment, in response to determining that the full level of memory 130-1 exceeds a threshold, embedded processor 122 identifies the oldest stream held in memory 130-1 (i.e., a stream instantiated in memory 130-1 before any other stream currently held in memory 130-1), and transfers stream state information corresponding to the oldest stream from memory 130-1 to memory 130-2. In another embodiment, embedded processor 122 utilizes different selection criteria to identify the streams for which to transfer stream state information from memory 130-1 to memory 130-2.

[0023] In one embodiment, program memory 124 coupled to embedded processor 122 stores a set of machine-readable instructions 126 that, when executed by embedded processor 122, cause embedded processor 122 to monitor a fullness level of memory 130-1 and, when the fullness level of memory 130-1 exceeds a threshold, i) identify in memory 130-1 a flow for which flow state information is to be transferred from memory 130-1, ii) read flow state information corresponding to the identified flow from memory 130-1, and iii) store the flow state information in memory 130-2. In one embodiment, storing the flow state information in memory 130-2 includes determining whether a record corresponding to the flow already exists in memory 130-2, such as whether a record corresponding to the flow was created in memory 130-2 when flow state information corresponding to the same flow was previously transferred from memory 130-1. In response to determining that a record corresponding to the flow already exists in memory 130-2, embedded processor 122 updates the record in memory 130-2 based on the flow state information currently being transferred from memory 130-1. On the other hand, in one embodiment, in response to determining that memory 130-2 does not include a record corresponding to the flow, embedded processor 122 creates a record corresponding to the flow in memory 130-2 and stores the flow state information in the created record in memory 130-2.

[0024] In one embodiment, after the flow state information corresponding to a particular flow is transferred from memory 130-1 to memory 130-2, a new flow is instantiated in the freed space in memory 130-1. For example, in one embodiment, flow monitoring engine 116, embedded processor 122, or CPU 138 instantiates the new flow in memory 130-2 as described above. In one embodiment, flow state update engine 120 of flow monitoring engine 116 then begins to update the flow state information corresponding to the new flow based on new incoming packets corresponding to the flow received by network device 100 at line speed.

[0025] In one embodiment, memory 130-2 is at a lower level relative to memory 130-1, and memory 130-2 is configured to maintain a greater number of flows than the maximum number of flows that can be maintained in memory 130-1. However, in one embodiment, memory 130-2 is limited to maintaining a maximum number of flows, which is greater than the maximum number of flows that can be maintained in memory 130-1. For example, in one embodiment, although memory 130-1 is limited to maintaining a maximum of k flows, memory 130-2 is limited to maintaining a maximum of n*k flows, where n and k are positive integers. In order to address the limitation on the number of flows that can be maintained in memory 130-2, network device 100 is configured to monitor the full level of memory 130-2, and when the full level of memory 130-2 exceeds a threshold, the flow state information corresponding to one or more flows is transferred from memory 130-2 to CPU 138. For example, in one embodiment, the set of instructions 142, when executed by the embedded processor 122, further causes the embedded processor 142 to monitor a fullness level of the memory 130-2, and when the fullness level of the memory 130-2 exceeds a threshold, transfer flow state information corresponding to one or more flows from the memory 130-2 to the CPU 138. In one embodiment, the CPU 138 is configured to receive the flow state information transferred from the memory 130-2 to the CPU 138, and store the flow state information in the memory 130-3.

[0026] In one embodiment, when the number of available entries in the flow state information table for maintaining flow state information in memory 130-2 is zero or nearly zero (e.g., 1, 2, 3, etc.), embedded processor 122 determines that the full level of memory 130-2 exceeds a threshold. In other embodiments, other suitable memory full level criteria are utilized. In one embodiment, in response to determining that the full level of memory 130-2 exceeds a threshold, embedded processor 122 identifies the oldest flow maintained in memory 130-2 (i.e., a flow instantiated in memory 130-2 before any other flow currently maintained in memory 130-2), and transfers the flow state information corresponding to the oldest flow from memory 130-2 to CPU 138. In another embodiment, embedded processor 122 utilizes different selection criteria to identify the flow for which flow state information is to be transferred from memory 130-1 to CPU 128. In one embodiment, after the flow state information corresponding to a particular flow is transferred from memory 130-2 to CPU 138, a record corresponding to the new flow is created in the freed space in memory 130-2 to store the flow state information corresponding to the new flow transferred from memory 130-1.

[0027] In one embodiment, program memory 142 coupled to CPU 138 stores a set of machine-readable instructions that, when executed by CPU 138, cause CPU 138 to receive flow state information transferred from memory 130 and store the flow state information in memory 130-3. In one embodiment, storing the flow state information in memory 130-3 includes determining whether a record corresponding to the flow already exists in memory 130-3, such as whether a record corresponding to the flow was created in memory 130-3 when the flow state information corresponding to the flow was previously transferred from memory 130-2. In response to determining that a record corresponding to the flow already exists in memory 130-3, CPU 138 updates the record in memory 130-3 based on the flow state information currently being transferred from memory 130-3. On the other hand, in response to determining that memory 130-2 does not include a record corresponding to the flow, CPU 138 creates a record corresponding to the flow in memory 130-3 and stores the flow state information in the created record in memory 130-3.

[0028] In one embodiment, memory 130-3 is at the next level relative to memory 130-2, and memory 130-3 is configured to maintain a greater number of flows than the maximum number of flows that can be maintained in memory 130-2. For example, in one embodiment, although memory 130-2 is limited to maintaining a maximum of n*k flows, memory 130-3 is configured to maintain a maximum of m*n*k flows, where m, n, and k are positive integers. In one embodiment, memory 130-3 is configured to maintain flow state information corresponding to the maximum number of flows for which flow monitoring is supported by a network device. In one embodiment, CPU 138 is configured to periodically transfer flow state information from memory 130-3 to an external flow monitoring entity (such as an external collector device). For example, in one embodiment, program memory 142 stores a machine-readable instruction set 146, which, when executed by CPU 138, causes CPU 138 to periodically transfer flow state information from memory 130-3 to an external flow monitoring entity. In one embodiment, periodically transferring the flow state information from memory 130-3 to the external flow monitoring entity includes transferring the flow state information from memory 130-3 according to a predetermined schedule, such as a specific flow state transfer schedule supported by the external flow monitoring entity.

[0029] Figure 22 is a diagram of an example flow state information table 200 for storing flow state information according to one embodiment. In one embodiment, each memory 130 of network device 100 includes a flow state information table, such as flow state information table 200. Flow state information table 200 includes a plurality of entries 202 for storing flow state information corresponding to different flows. In one embodiment, flow state information table 200 includes a plurality of entries that are limited by the storage capacity of the memory including flow state information table 200. For example, referring to Figure 1 and 2 In one embodiment, the flow state information table (such as flow state information table 200) included in memory 130-1 includes at most k entries 202, the flow state information table (such as flow state information table 200) included in memory 130-2 includes at most n*k entries 202, and the flow state information table (such as flow state information table 200) included in memory 130-3 includes at most m*n*k entries 202, where m, n, and k are positive integers. Each entry 202 of the flow state information table 200 includes a flow identifier field 204 that associates the entry 202 with a particular flow, and a plurality of flow state fields 206 that store flow state information corresponding to the identified flow. Field 206 includes: a byte count field 206-1, which is used to record the count of bytes in the received packets belonging to the identified flow; a packet count field 206-2, which is used to record the count of received packets belonging to the identified flow; a start time field 206-3, which is used to record the start time of the flow (for example, the time when the first packet belonging to the flow is received by the network device), and a last packet time field 206-4, which is used to record the time when the most recent packet belonging to the flow is received by the network device. In some embodiments, the flow state information table 200 omits Figure 2 One of the plurality of flow state information fields 206 shown, and / or including Figure 2 One or more flow state information fields 206 not shown.

[0030] In one embodiment, instantiating a new flow or creating a record for a new flow in the flow state information table 200 includes recording identification information associated with the new flow in the flow identifier information field 204 of an available entry 202 of the flow state information table 200, and recording current flow state information in the flow state information field 206 of the entry 202 (e.g., based on the first incoming packet of the flow being instantiated in the flow state information table 200, or based on flow state information being transferred to the flow state information table 200). In one embodiment, the flow state update engine 122 is configured to update the flow state information based on an incoming packet belonging to an existing flow for which an entry already exists in the flow state information table 200 (e.g., stored in the memory 130-1) by: i) searching the flow state information table 200 to identify an entry 202, identifying the flow identifier information field 204 of the flow in the entry 202, ii) reading the current flow state information from one or more flow state information fields 206 of the identified entry 202, iii) calculating updated values ​​of the one or more flow state information fields 206 based on the current flow state information read from the identified entry 202 and the incoming packet belonging to the flow, and iv) writing the updated values ​​to the corresponding one or more flow state information fields. Similarly, in one embodiment, the embedded processor 122 / CPU 138 updates the flow state information corresponding to the flow being transferred to the memory 130-2 / 130-2 by: i) searching the flow state information table 200 to identify the entry 202, identifying the flow identifier information field 204 of the flow in the entry 202, ii) reading the current flow state information from one or more flow state information fields 206 of the identified entry 202, iii) calculating updated values ​​of the one or more flow state information fields 206 based on the current flow state information read from the identified entry 202 and the flow state information being transferred, and iv) writing the updated values ​​to the corresponding one or more flow state information fields.

[0031] Figure 3 is a block diagram of a network 300 with flow monitoring capabilities according to one embodiment. In one embodiment, the network 300 includes a plurality of network devices 304, including at least a first network device 304-1, a second network device 304-2, and a third network device 304-3. In one embodiment, each network device 304 is connected to Figure 1304-1 is the same or similar to the network device 104 of the network 300. In one embodiment, different ones of the network devices 304 are located at different locations or levels in the network 300, and process different numbers of flows in a given time interval depending on the location or level at which the network devices are divided in the network 300. For example, in an embodiment where the network 300 is an internal network (e.g., a campus network) connected to an external network (e.g., the Internet), the network device 304-1 is an edge network device, such as an edge router that connects the network 300 to other networks, and the network devices 302-2 and 302-3 are non-edge network devices, such as non-edge switches or routers operating in the network 300. In this embodiment, the network device 304-1 generally processes a larger number of flows than the number of flows processed by the non-edge network device 302-2 or the non-edge network device 302-3. As another example, in an embodiment where network devices 304 are arranged in a multi-level architecture in network 300, network device 304-1 is a core network device in network 300, while network devices 304-2, 304-3 are leaf or backbone network devices in network 300. In this embodiment, network device 304-1 operating at the core of network 300 typically handles more flows in a given time than network devices 304-2, 304-3 operating at non-core levels of network 300.

[0032] The network device 304 is configured to collect flow state information related to flows traversing the network device 304 as described herein, and output the flow state information to the collector device 310. In one embodiment, the network device 304 is configured to output the flow state information to the collector device 301 over the network according to a suitable protocol, such as the IPFIX protocol or another suitable protocol. In one embodiment, the network device 304 outputs the flow state information to the collector device 310 according to a predetermined schedule, such as periodically at predetermined time intervals. In one embodiment, the collector device 310 stores the flow state information in the flow database 310 and / or provides the flow state information to the application server 314 for subsequent analysis by the application server 314. In various embodiments, the application server 314 uses the flow state information received from the network device 304 for various operations, such as network planning, optimizing network resource utilization or service performance, detection and prevention of network attacks / intrusions, quality of service (QoS) monitoring, usage-based billing, and the like.

[0033] In one embodiment, each network device 304 utilizes one or more hierarchical levels of memory as described herein to scale to the number of flows for which the network device 304 maintains flow state information at the network device 304 without having to scale lower hierarchical levels of memory that are faster but more expensive in terms of cost, power consumption, etc. Thus, for example, by using the hierarchical memory architecture described herein, a network device 304-1 operating at the edge or core of a network scales to monitor a greater number of flows than network device 304-2, even though, in one embodiment, network device 304-1 utilizes the same amount of hardware memory (e.g., TCAM) and the same amount of embedded memory as network device 304-2.

[0034] Figure 4 According to one embodiment, a Figure 1 Flow chart of an example method 400 implemented by a network device of the network device 104 of FIG. 100 ( Figure 1 ) describes the method 400. In other embodiments, the method 400 is different from Figure 1 A suitable network device implementation of network device 104.

[0035] At block 402, packets entering via a plurality of network interfaces of the network device are received by a packet processor of the network device, and at block 404, the received packets are classified into respective flows by the packet processor. In one embodiment, packets are classified into packet flows based at least in part on information in respective headers of the packets. By way of example only, in one embodiment, packets are classified into flows, which are defined as flows of packets having headers, all of which have the same 5-tuple value, including: i) source IP address, ii) source port number, iii) destination IP address, iv) destination port number, and v) specific IP protocol. In other embodiments, packets are classified into flows according to other suitable classification criteria.

[0036] In box 406, the flow state information associated with the corresponding flow corresponding to the received packet is updated based on the received packet. In one embodiment, the flow state information is stored in a first memory of a plurality of memories, and the plurality of memories are used to maintain the flow state information at the network device. In one embodiment, the flow state information is stored in the memory 130-1. In another embodiment, the flow state information is stored in a suitable memory different from the memory 130-1. In one embodiment, the plurality of memories for maintaining the flow state information at the network device are arranged in a hierarchical arrangement, in which the memories at gradually higher hierarchical levels are configured to maintain the flow state information corresponding to the gradually larger flow sets processed by the network device. In one embodiment, the first memory is at the first hierarchical level of the hierarchical arrangement. In one embodiment, the first memory has the highest access bandwidth among the plurality of memories. In one embodiment, the first memory supports updating the flow state information at line speed.

[0037] At block 408, it is determined that the fullness level of the first memory exceeds a first threshold. For example, in one embodiment, it is determined that the number of available entries in the flow state information table for holding flow state information in the first memory is zero or nearly zero (e.g., 1, 2, 3, etc.). In other embodiments, other suitable memory fullness criteria are utilized.

[0038] At block 410, in response to determining that the fullness level of the first memory exceeds a first threshold, flow state information associated with at least one flow in the first flow set for which flow state information is currently being maintained in the first memory is transferred from the first memory to a second memory of the plurality of memories to free up space in the first memory. In one embodiment, the second memory is at a second hierarchical level of the hierarchical arrangement, the second hierarchical level being higher than the first hierarchical level of the hierarchical arrangement. In one embodiment, transferring the flow state information associated with the flow from the first memory to the second memory includes: identifying the oldest flow in the first flow set, the oldest flow having been instantiated in the first memory before any other flow in the first flow set for which flow state information is currently being maintained in the first memory, and transferring the flow state information associated with the oldest flow identified in the first flow set to the second memory. In one embodiment, the second memory has a memory access bandwidth that is less than a memory access bandwidth of the first memory. In one embodiment, the second memory does not support updating the flow state information at line rate. In one embodiment, flow state information associated with the oldest flows is transferred from a first memory to a second memory to ensure that flow state information for the newest and most active flows being processed by the network device at any given time is maintained in a first, faster memory of the network device, while expanding the total number of flows being monitored by the network device.

[0039] At block 412, after transferring the flow state information associated with at least one flow from the first memory to the second memory, a new flow is instantiated in the freed space in the first memory. The flow state information corresponding to the new flow is then updated at line speed based on subsequent packets corresponding to the flow received by the network device.

[0040] In one embodiment, a method for monitoring flows in a network device includes: receiving, at a packet processor of the network device, packets entering via a plurality of network interfaces of the network device; using the packet processor to classify the packets into corresponding flows based at least in part on information in corresponding headers of the packets; using the packet processor to update, based on the received packets, flow state information associated with the corresponding flows corresponding to the received packets, the flow state information being stored in a first memory of a plurality of memories, the plurality of memories being used to maintain the flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement, in which memories at progressively higher hierarchical levels are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a hierarchical level. A first hierarchical level of the hierarchical arrangement; and dynamically managing multiple memories using a packet processor, including: using the packet processor to determine that a full level of the first memory exceeds a first threshold, in response to determining that the full level of the first memory exceeds the first threshold, transferring flow state information associated with at least one flow in a first flow set from the first memory to a second memory among the multiple memories to release space in the first memory, the flow state information for the first flow set is currently being maintained in the first memory, the second memory is at a second hierarchical level, the second hierarchical level is higher than the first hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the first memory to the second memory, instantiating a new flow in the space released in the first memory using the packet processor.

[0041] In other embodiments, the method further includes one of the following features, or any suitable combination of two or more of the following features.

[0042] The first memory has a first access bandwidth, and transferring flow state information associated with at least one flow from the first memory to the second memory includes: transferring the flow state information to the second memory having a second memory access bandwidth, the second memory access bandwidth being lower than the first memory access bandwidth.

[0043] Based on the received packet, updating flow state information associated with a flow corresponding to the packet includes searching a ternary content addressable memory (TCAM) to identify an entry corresponding to the flow, and updating one or more fields of the entry corresponding to the flow based on the received packet.

[0044] Transferring the flow state information associated with the at least one flow includes transferring the flow state information associated with the at least one flow to a random access memory (RAM).

[0045] Transferring flow state information associated with at least one flow from a first memory to a second memory includes identifying an oldest flow in a first set of flows, the oldest flow having been instantiated in the first memory before any other flow in the first set of flows for which flow state information is currently being maintained in the first memory, and transferring to the second memory the flow state information associated with the oldest flow identified in the first set of flows.

[0046] Dynamically managing multiple memories also includes: using a packet processor to determine that a full level of a second memory exceeds a second threshold, in response to determining that the full level of the second memory exceeds the second threshold, transferring flow state information associated with at least one flow in a second flow set from the second memory to a third memory among the multiple memories to free up space in the second memory, the flow state information for the second flow set currently being maintained in the second memory, the third memory being at a third hierarchical level, the third hierarchical level being higher than a second hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the second memory to the third memory, creating a record in the space freed in the second memory, the record being used to store the flow state information associated with the new transferred flow from the first memory to the second memory.

[0047] Transferring the flow state information associated with the at least one flow from the second memory to the third memory includes transferring the flow state information associated with the at least one flow to a memory external to the packet processor.

[0048] Transferring the flow state information associated with the at least one flow from the second memory to the third memory includes transferring the flow state information associated with the at least one flow to a memory coupled to a central processing unit (CPU) of the network device.

[0049] Transferring flow state information associated with at least one flow from a second memory to a third memory includes identifying an oldest flow in a second set of flows, the oldest flow having been instantiated in the second memory before any other flow in the second set of flows for which flow state information is currently being maintained in the second memory, and transferring to the third memory the flow state information associated with the oldest flow identified in the second set of flows.

[0050] The method also includes: periodically outputting the flow state information from the third memory to a collector device, which is outside the network device, and after outputting the flow state information from the third memory to the collector device, making the memory space occupied by the flow state information in the third memory available for storing new flow state information that is subsequently transferred from the second memory to the third memory.

[0051] In another embodiment, a network device includes a plurality of network interfaces, and a packet processor coupled to the plurality of network interfaces. The packet processor is configured to: receive packets incoming via the plurality of network interfaces of the network device, classify the packets into respective flows based at least in part on information in respective headers of the received packets, update flow state information associated with the respective flows corresponding to the received packets based on the received packets, the flow state information being stored in a first memory of a plurality of memories, the plurality of memories being used to maintain the flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement, the memories at progressively higher hierarchical levels in the hierarchical arrangement being configured to maintain the flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a first hierarchical level of the hierarchical arrangement, and dynamically The invention manages multiple memories, and the packet processor is configured to: determine that a full level of a first memory exceeds a first threshold; in response to determining that the full level of the first memory exceeds the first threshold, transfer flow state information associated with at least one flow in a first flow set from the first memory to a second memory among the multiple memories to release space in the first memory, the flow state information for the first flow set is currently being maintained in the first memory, the second memory is at a second hierarchical level, the second hierarchical level is higher than the first hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the first memory to the second memory, instantiate a new flow in the space released in the first memory.

[0052] In other embodiments, the network device further includes one of the following features, or any suitable combination of two or more of the following features.

[0053] The first memory has a first access bandwidth.

[0054] The second memory has a second memory access bandwidth, which is lower than the first memory access bandwidth.

[0055] The first memory includes a ternary content addressable memory (TCAM) that associates flow state information with a corresponding flow.

[0056] The second memory includes a random access memory (RAM) that associates flow state information with corresponding flows.

[0057] The packet processor is configured to update flow state information associated with a flow corresponding to the packet based on the received packet by at least searching the TCAM to identify an entry corresponding to the flow and updating one or more fields of the entry corresponding to the flow based on the received packet.

[0058] The packet processor is configured to: identify an oldest flow in the first set of flows, which has been instantiated in the first memory before any other flow in the first set of flows for which flow state information is currently being maintained in the first memory, and transfer flow state information associated with the oldest flow from the first memory to the second memory.

[0059] The packet processor is also configured to: determine that a full level of the second memory exceeds a second threshold, in response to determining that the full level of the second memory exceeds the second threshold, transfer flow state information associated with at least one flow in the second flow set from the second memory to a third memory among the multiple memories to free up space in the second memory, the flow state information for the second flow set currently being maintained in the second memory, the third memory being at a third hierarchical level of the hierarchical arrangement, and after transferring the flow state information associated with at least one flow from the second memory to the third memory, create a record in the space freed in the second memory, the record being used to store the flow state information associated with the flow subsequently transferred from the first memory to the second memory.

[0060] The packet processor is configured to: identify the oldest flow in the second set of flows for which flow state information is currently being maintained in the second memory, the oldest flow having been instantiated in the second memory before any other flow in the second set of flows, and transfer the flow state information associated with the oldest flow from the second memory to the third memory.

[0061] The third memory is external to the packet processor.

[0062] The network device also includes an external processor, wherein the third memory is included in a database coupled to the external processor.

[0063] The external processor is configured to: periodically output the flow state information stored in the third memory to the collector device, and after outputting the flow state information from the third memory to the collector device, make the memory space occupied by the flow state information in the third memory available for storing new flow state information that is subsequently transferred from the second memory to the third memory.

[0064] At least some of the various blocks, operations, and techniques described above may be implemented using hardware, a processor executing firmware instructions, a processor executing software instructions, or any combination thereof. When implemented using a processor executing software or firmware instructions, the software or firmware instructions may be stored in any computer-readable memory, such as a disk, optical disk, or other storage medium, RAM or ROM or flash memory, a processor, a hard drive, an optical drive, a tape drive, etc. The software or firmware instructions may include machine-readable instructions that, when executed by one or more processors, cause the one or more processors to perform various actions.

[0065] When implemented in hardware, the hardware may include one or more of the following: discrete components, integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), and the like.

[0066] Although the present invention has been described with reference to specific examples, which are merely illustrative and not restrictive, changes, additions and / or deletions may be made to the disclosed embodiments without departing from the scope of the present invention.

Claims

1. A method for monitoring flows in a network device, the method comprising: receiving, at a packet processor of the network device, packets incoming via a plurality of network interfaces of the network device; classifying, with the packet processor, the packets into respective flows based at least in part on information in respective headers of the packets; updating, with the packet processor, flow state information associated with the corresponding flow corresponding to the received packet based on the received packet, the flow state information being stored in a first memory of a plurality of memories for maintaining flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement in which memories at progressively higher hierarchical levels are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a first hierarchical level of the hierarchical arrangement; as well as Dynamically managing the plurality of memories using the packet processor comprises: determining, using the packet processor, that a full level of the first memory exceeds a first threshold, responsive to determining that the full level of the first memory exceeds the first threshold, transferring flow state information associated with at least one flow in a first set of flows from the first memory to a second memory of the plurality of memories to free up space in the first memory, flow state information for the first set of flows currently being held in the first memory, the second memory being at a second hierarchy level, the second hierarchy level being higher than the first hierarchy level of the hierarchical arrangement, and After transferring flow state information associated with the at least one flow from the first memory to the second memory, a new flow is instantiated, using the packet processor, in the freed space in the first memory.

2. The method according to claim 1, wherein: The first memory has a first access bandwidth, and Transferring flow state information associated with the at least one flow from the first memory to the second memory includes transferring the flow state information to the second memory having a second memory access bandwidth that is lower than the first memory access bandwidth.

3. The method of claim 1 , wherein updating flow state information associated with a flow corresponding to the packet based on the received packet comprises: searching a ternary content addressable memory (TCAM) to identify an entry corresponding to the flow, and One or more fields of the entry corresponding to the flow are updated based on the received packet.

4. The method of claim 3, wherein transferring flow state information associated with at least one flow comprises: The flow state information associated with the at least one flow is transferred to a random access memory (RAM).

5. The method of claim 1 , wherein transferring flow state information associated with the at least one flow from the first memory to the second memory comprises: identifying an oldest flow in the first set of flows, the oldest flow having been instantiated in the first memory before any other flow in the first set of flows for which flow state information is currently being maintained in the first memory, and Flow state information associated with the oldest flow identified in the first set of flows is transferred to the second memory.

6. The method of claim 1, wherein dynamically managing the plurality of memories further comprises: determining, using the packet processor, that a full level of the second memory exceeds a second threshold, in response to determining that the full level of the second memory exceeds the second threshold, transferring flow state information associated with at least one flow in a second set of flows from the second memory to a third memory of the plurality of memories to free up space in the second memory, the flow state information for the second set of flows currently being held in the second memory, the third memory being at a third hierarchy level, the third hierarchy level being higher than the second hierarchy level of the hierarchical arrangement, and After transferring the flow state information associated with the at least one flow from the second memory to the third memory, a record is created in the space released in the second memory, the record being used to store the flow state information associated with the new transferred flow from the first memory to the second memory.

7. The method of claim 6, wherein transferring flow state information associated with the at least one flow from the second memory to the third memory comprises: The flow state information associated with the at least one flow is transferred to a memory external to the packet processor.

8. The method of claim 6, wherein transferring flow state information associated with the at least one flow from the second memory to the third memory comprises: The flow state information associated with the at least one flow is transferred to a memory coupled to a central processing unit (CPU) of the network device.

9. The method of claim 6, wherein transferring flow state information associated with the at least one flow from the second memory to the third memory comprises: identifying an oldest flow in the second set of flows, the oldest flow having been instantiated in the second memory before any other flow in the second set of flows for which flow state information is currently being maintained in the second memory, and Flow state information associated with the oldest flow identified in the second set of flows is transferred to the third memory.

10. The method according to claim 6, further comprising: periodically outputting the flow state information from the third memory to a collector device, the collector device being external to the network device, and After the flow state information is output from the third memory to the collector device, the memory space occupied by the flow state information of the third memory is made available for storing new flow state information that is subsequently transferred from the second memory to the third memory.

11. A network device comprising: Multiple network interfaces, and a packet processor coupled to the plurality of network interfaces, wherein the packet processor is configured to: receiving packets incoming via a plurality of network interfaces of the network device, classifying the received packets into respective flows based at least in part on information in respective headers of the packets, updating flow state information associated with the corresponding flow corresponding to the received packet based on the received packet, the flow state information being stored in a first memory of a plurality of memories, the plurality of memories being used to maintain the flow state information at the network device, the plurality of memories being arranged in a hierarchical arrangement in which memories at progressively higher hierarchical levels are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device, the first memory being at a first hierarchical level of the hierarchical arrangement, and Dynamically managing the plurality of memories, the packet processor is configured to: determining that a fullness level of the first memory exceeds a first threshold; responsive to determining that the full level of the first memory exceeds the first threshold, transferring flow state information associated with at least one flow in a first set of flows from the first memory to a second memory of the plurality of memories to free up space in the first memory, flow state information for the first set of flows currently being held in the first memory, the second memory being at a second hierarchy level, the second hierarchy level being higher than the first hierarchy level of the hierarchical arrangement, and After transferring the flow state information associated with the at least one flow from the first memory to the second memory, a new flow is instantiated in the freed space in the first memory.

12. The network device according to claim 11, wherein: The first memory has a first access bandwidth, and The second memory has a second memory access bandwidth, which is lower than the first memory access bandwidth.

13. The network device according to claim 11, wherein: The first memory includes a ternary content addressable memory (TCAM) that associates flow state information with corresponding flows, and The second memory includes a random access memory (RAM) that associates flow state information with a corresponding flow.

14. The network device of claim 13, wherein the packet processor is configured to update flow state information associated with a flow corresponding to the packet based on the received packet by at least: searching the TCAM to identify an entry corresponding to the flow, and One or more fields of the entry corresponding to the flow are updated based on the received packet.

15. The network device of claim 11, wherein the packet processor is configured to: identifying an oldest flow in the first set of flows, the oldest flow having been instantiated in the first memory before any other flow in the first set of flows for which flow state information is currently being maintained in the first memory, and Flow state information associated with the oldest flow is transferred from the first memory to the second memory.

16. The network device of claim 11, wherein the packet processor is further configured to: determining that a full level of the second memory exceeds a second threshold, in response to determining that the full level of the second memory exceeds the second threshold, transferring flow state information associated with at least one flow in a second set of flows from the second memory to a third memory of the plurality of memories to free up space in the second memory, the flow state information for the second set of flows currently being held in the second memory, the third memory being at a third hierarchical level of the hierarchical arrangement, and After transferring the flow state information associated with the at least one flow from the second memory to the third memory, a record is created in the space released in the second memory, the record being used to store the flow state information associated with the flow subsequently transferred from the first memory to the second memory.

17. The network device of claim 16, wherein the packet processor is configured to: identifying an oldest flow in the second set of flows for which flow state information is currently being maintained in the second memory, the oldest flow having been instantiated in the second memory before any other flow in the second set of flows, and Flow state information associated with the oldest flow is transferred from the second memory to the third memory.

18. The network device of claim 16, wherein the third memory is external to the packet processor.

19. The network device of claim 16, further comprising an external processor, wherein the third memory is included in a database coupled to the external processor.

20. The network device of claim 19, wherein the external processor is configured to: periodically outputting the flow state information stored in the third memory to the collector device, and After the flow state information is output from the third memory to the collector device, the memory space occupied by the flow state information in the third memory is made available for storing new flow state information that is subsequently transferred from the second memory to the third memory.