Intelligent flow state synchronization to improve availability and performance of redundant network security devices

Abstract

Embodiments of the present disclosure relate to intelligent flow state synchronization that improves availability and performance of redundant network security devices. An example security system for use between at least one upstream router and at least one downstream router is described. A group or pool of security devices can be used to provide stateful security to bidirectional packet flows between the upstream router and the downstream router. Packets of the bidirectional flows are forwarded to particular security devices based on a consistent hashing ring process. Bidirectional state information is synchronized between some, but not all, of the security devices for a given flow. The security devices that share such bidirectional flow state information are determined using the same consistent hashing ring process.

Description

§1. BACKGROUND

[0001] §1.1 FIELD OF TECHNOLOGY

[0002] This specification relates to communication networks and security. More specifically, this specification relates to providing security to a communication network.

[0003] §1.2 BACKGROUND INFORMATION

[0004] Current network security devices typically provide various stateful network security services (e.g., Layer 3 to Layer 7 (L3 to L7)) by processing received Internet Protocol (IP) packets. Current network security devices are used to monitor network traffic and classify it as legitimate or suspicious. Suspicious traffic can be dropped and / or blocked, and / or other security processing applied. In a typical L3 network topology, the security devices are connected to neighbor routers via a dynamic routing protocol (e.g., BGP).

[0005] To increase the resiliency, availability, and performance of network security services, it is desirable to group multiple security devices together so that one or more backup security devices can provide redundancy in the event that a primary security device fails or goes down (e.g., reboots) for some reason. It can also be desirable for multiple security devices to share the workload of providing security services by operating in parallel.

[0006] Figure 1 An example network topology 100 is illustrated in which a pool or group of N security devices (SDs) 110 provide security for upstream routers 120 (e.g., Rl and R2 connecting to a public network 125 or a portion of the public network 125) and downstream routers 130 (e.g., R3 and R4 connecting to an internal network 135 or a portion of the internal network 135). The security devices, upstream routers, and / or downstream routers can share control information via a protocol such as the Border Gateway Protocol (BGP) 130. Each of the upstream routers 120 can use Equal Cost Multi-Path (ECMP) 140 to provide forwarding to the pool of security devices 110. For example, the routers implement packet-based forwarding. In selecting an ECMP next hop, the routers can use a variety of different algorithms. One such algorithm is a hash algorithm whose inputs or keys include information extracted from the header of an IP packet (e.g., source IP address and / or destination IP address). Similarly, each of the downstream routers 130 can use ECMP 150 to provide forwarding to the pool of security devices 110.

[0007] The concept of this policy of providing a pool of security devices appears promising. Unfortunately, however, when the number of redundant security devices becomes large (assuming that "stateful" security is to be provided), it has scaling problems. If these security devices are to provide stateful security, the scaling problem arises from the need for flow state synchronization between the security devices. More specifically, routers typically treat redundant security devices as ECMP next hops. That is, routers typically use an ECMP path selection algorithm (e.g., using a hash function that is provided with the IP address of a packet as input) to map IP packets to security devices. The routers use the mapping results to select a security device as the next hop for an IP packet and send it to the selected security device.

[0008] In fact, a security device can go down (e.g., "off-line") and later come back up (e.g., "on-line"). The potential changes in the security devices while they are off-line would cause the routers to change the mapping between IP packet flows and their next hops of security devices. Without knowing the results of such mapping changes by the upstream routers and / or downstream routers, the security devices in the group would assume the worst case scenario that the mapping would change randomly. Thus, under this worst case assumption, each security device would need to be backed up by any and all of its peer devices. That is, under this worst case assumption, each security device would synchronize its flow state information with all of the peer security devices in the group. This amount of synchronization of flow state information would cause the security devices to spend excessive bandwidth sending state to each of the peer security devices (and receiving state) and to allocate large amounts of memory to store copies of the state received by all of the peer security devices in the group, especially as N becomes large. This problem makes it technically and economically impractical to scale the number of redundant security devices proportionally.

[0009] In view of the foregoing, it would be useful to provide a security service with high resiliency, availability, and performance. It would be useful to do so while avoiding scaling problems by providing many redundant security devices.

[0010] Some security devices apply bidirectional rules and, thus, track information about data traffic in both upstream and downstream directions. Thus, it would be useful to provide stateful bidirectional security. §2. SUMMARY

[0011] Example embodiments in accordance with the present specification address the challenge of providing many redundant security devices while avoiding scaling problems. Example embodiments in accordance with the present specification can be implemented by forwarding packets of bidirectional flows to specific security devices based on a consistent hashing ring procedure. Bidirectional state information is synchronized between some but not all of the security devices for a given flow. The security devices that share such bidirectional flow state information use the same consistent hashing ring procedure to determine.

[0012] In one example embodiment, a security system for use between at least one upstream router and at least one downstream router is provided. Such a security system includes: a first plurality (N) of security devices arranged between the at least one upstream router and the at least one downstream router, each of the first plurality of security devices configured to provide bidirectional security services for a plurality of bidirectional flows between the at least one upstream router and the at least one downstream router. For each bidirectional flow of the plurality of bidirectional flows, (1) one of the first plurality of security devices is assigned as a primary security device to the bidirectional flow using a consistent hashing ring algorithm for generating a consistent hashing ring, and (2) a set of one or more (M, where M is at least 1) of the first plurality of security devices are assigned as backup security device(s) to the bidirectional flow using the consistent hashing ring algorithm, where M+1 is less than N. Each of the first plurality of security devices is configured with a state synchronization procedure, where for each bidirectional flow of the plurality of bidirectional flows, the primary security device assigned to the bidirectional flow and the set of M backup security devices assigned to the bidirectional flow synchronize bidirectional flow state information with each other but not with all of the N security devices.

[0013] In some such security systems, for each flow of the plurality of flows, the primary security device assigned to the bidirectional flow and the set of M backup security devices assigned to the bidirectional flow synchronize bidirectional flow state information with each other but not with any other of the N security devices.

[0014] In some such security systems, for each bidirectional flow of the plurality of bidirectional flows, the M backup security devices assigned to the bidirectional flow are the M next unique security devices on the consistent hashing ring (e.g., in a clockwise direction) after the primary security device assigned to the bidirectional flow.

[0015] In some such security systems, upon failure of the primary security device assigned to a given bidirectional flow, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to the next available (e.g., UP) backup security device of the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring. In some such security systems, upon recovery of the primary security device assigned to a given bidirectional flow, in response to the recovery, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to the primary security device assigned to the given bidirectional flow.

[0016] In some such security systems, if M is greater than 1, then upon failure of both (1) the primary security device assigned to a given bidirectional flow and (2) the next backup security device in the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring, at least one upstream router and at least one downstream router forwards any packets belonging to the given bidirectional flow to the next available backup security device in the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring. §3. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An example network topology in which a pool or group of security devices provide security for upstream routers and downstream routers is illustrated.

[0018] Figure 2 The assignment of packet flows and security devices on a consistent hashing ring is illustrated.

[0019] Figure 3 The association of a packet flow with its primary security device and N backup security devices on a consistent hashing ring is illustrated.

[0020] Figure 4A and Figure 4B are flow diagrams of example methods performed by an upstream router or downstream router and a security device, respectively.

[0021] Figure 5 An example data forwarding system that can be used as a node in a communication network, such as a communication network employing routers and security devices, coupled via communication links is illustrated.

[0022] Figure 6 is a block diagram of a router that can be used in a communication network, such as a communication network employing routers and security devices.

[0023] Figure 7 is an example architecture of an ASIC that can be distributed in packet forwarding components to divide responsibility for packet forwarding.

[0024] Figure 8A and Figure 8B are examples of operations of the example architecture of Figure 7

[0025] Figure 9 is a flow diagram of an example method for providing packet forwarding in an example router.

[0026] Figure 10 is a block diagram of an example machine that can execute one or more of the processes described and / or store information used and / or generated by such processes.

[0027] Figure 11 ​Figure illustrates synchronization of flow state for a backup security device.

[0028] Figure 12 Figure illustrates the relationship between a restored security device and its peer security devices.

[0029] Figure 13 Figure illustrates the placement of virtual security devices (associated with physical security devices) on a consistent hashing ring.

[0030] Figure 14 Figure illustrates synchronization or sharing of flow state information between virtual security devices (associated with physical security devices) placed on a consistent hashing ring.

[0031] Figure 15 Figure illustrates operations by example upstream and downstream routers with respect to a pool of security devices. §4. DETAILED DESCRIPTION

[0032] The present disclosure can relate to novel methods, apparatuses, message formats, and / or data structures to efficiently reduce or minimize the overhead of state synchronization for (e.g., many) redundant security devices. The following description is presented to enable any person skilled in the art to make and use the described embodiments, and is provided in the context of particular applications and their requirements. Thus, the following description of example embodiments is provided for illustrative purposes so as to describe and describe, but is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Various modifications will be apparent to those skilled in the art, and the general principles

[0033] §4.1 Definitions

[0034] A "consistent hashing ring algorithm" is an algorithm or process shared among multiple devices such that each device can independently determine a consistent location on a consistent hashing ring or hash ring according to one or more given inputs.

[0035] A "security device" is a device that provides security functionality (and possibly also provides routing functionality). Security devices are intended to include instances of both hardware and software (also referred to as "physical" or "real") security devices as well as virtual security devices. For example, a virtualized security device can be launched on a public provider cloud. Thus, a security device can include hardware, software, or some combination of hardware and software. A security device can be referred to as an "application-level gateway."

[0036] An "available" security device can provide security functionality, and possibly also provide routing functionality. An "available" security device can include a state in which the security device has recovered from a failure, a state in which the security device has completed a reboot, and the like. An "available" security device can be in an "UP" state or in a "COLD" state. More specifically, when a security device is in an UP state, the security device is operational, ready to provide stateful security, and has all of the required flow state information. A security device can be considered to be in a partial start or "COLD" state when it is receiving some of the required flow state information (e.g., with incomplete synchronization). For example, a "recovered" security can have no packet flow state information (COLD), incomplete packet flow state information (COLD), or complete packet flow state information (UP). Thus, a recovered security device can be said to be "COLD" in the case that it has no packet flow state information or incomplete packet flow state information, but "warms up" as the security device receives more packet flow state information. Thus, a recovered SD can not yet be ready to provide security functionality, and can consider the COLD state to be a transitional state between a DOWN state and an UP state. A security device that is not available (e.g., not UP and not COLD) is said to be "unavailable."

[0037] §4.2 Overview of Configuration System

[0038] In example systems according to the present specification, a security device of a packet connects to a downstream router and an upstream router to exchange control information via a routing protocol. The security device of the packet also provides network security service(s) for IP packets that transition these network security services from the upstream router to the downstream router or from the downstream router to the upstream router. The network security services are "stateful" in accordance with flows of the bidirectional IP packets of the transitioning security device. Thus, the security device creates, stores, updates, and deletes state for each flow of bidirectional IP packets that are being processed. In some example embodiments according to the present specification, a flow is defined by a tuple of IP packet header data (e.g., source IP address, destination IP address, and the like).

[0039] To prevent potential failure of the security device(s) from interrupting "stateful" network security services, a redundant copy of the state information for each flow is needed. This redundant copy of the state information can be stored on another device, such as on another security device. As mentioned above, the routers can use ECMP forwarding towards multiple security devices for each packet flow to achieve network redundancy. The security devices synchronize the state information for each flow with the appropriate peer devices to ensure redundancy of the stateful network security services. From the perspective of the routers, each packet flow is sent to a primary security device through a primary ECMP path, or to one of several (or more generally, at least one) backup security devices through one of several (or more generally, at least one) backup ECMP paths. With the 1:M redundancy model of network security services, each security device is backed up by M devices; the primary security device for a flow synchronizes its flow state to the M backup security devices, where M is equal to or greater than 1, but less than N. When the total number of security devices (denoted by N) is large, M is much smaller than N as a practical matter. Therefore, the sharing of each flow state information among the M security devices is scaled independently of the total number of security devices N in the pool.

[0040] By sharing the flow state information among the M security devices, these M security devices and their neighboring routers have the same (or almost the same) view of the primary security device and the backup security devices for each packet flow. In example embodiments according to the present specification, this is achieved by having these routers and security devices share and use the same consistent hashing algorithm to determine the same set of primary security devices and backup security devices for each packet flow. With this same view, the security devices with the 1:M redundancy model limit the scope of the synchronization of the state of each packet flow to between the primary device and the M backup devices assigned to the flow, rather than to all N devices. Therefore, the efficiency of the flow state synchronization is significantly improved (especially in the case of M « N), and the load of the flow state synchronization of each security device is independent of the total number of security devices (N). Conventionally, the state synchronization among N devices is an O(N2) problem. However, in the case of example embodiments according to the present specification, the state synchronization of each security device is limited to M devices, and the state synchronization of the total number of security devices (N) is an O(NM) problem, where M « N. As can be seen from the above, the larger the N:M ratio, the more the efficiency of the flow state synchronization is improved. 2 - ) problem. However, in the case of example embodiments according to the present specification, the state synchronization of each security device is limited to M devices, and the state synchronization of the total number of security devices (N) is an O(NM) problem, where M « N. As can be seen from the above, the larger the N:M ratio, the more the efficiency of the flow state synchronization is improved.

[0041] Applying a consistent hashing algorithm permits routers to reduce the flow of packet shuffling across security devices after a failure (entering a DOWN state; that is, becoming unavailable) and recovery (entering an UP state; that is, becoming available). Generally, routers only redirect flows that are located on segments between an SD entering UP and DOWN and its immediate neighbor SDs on the consistent hashing ring; not flows that are located outside of the segment. Thus, using a consistent hashing algorithm reduces or limits the portion of packet flows that are redirected compared to a regular hashing algorithm. Applying a consistent hashing algorithm also permits security devices to easily identify the primary security device and backup device(s) for a given packet flow. The input to the consistent hashing algorithm is an IP packet flow defined by a tuple of header data of a packet (e.g., source IP address, destination IP address, etc.).

[0042] To ensure the required symmetric routing for security devices, downstream routers and upstream routers use the same information as well as data extracted from the header of an IP packet as input to the consistent hashing algorithm. For example, in one example embodiment according to the present specification, the source IP address and the destination IP address are maintained as input to the consistent hashing algorithm of an upstream (or downstream) router, but are reversed before being input to the consistent hashing algorithm of the upstream (or downstream) router. Thus, for packets of the same bidirectional flow, a downstream router and an upstream router obtain the same result (output) of the consistent hashing function. Further, a security device receives packets from both a downstream router and an upstream router, and can extract different data from the header of the packets to use as input to the consistent hashing algorithm applied by the security device. Thus, when a security device receives a packet from a downstream router or an upstream router, the security device extracts the same data from the header of the packet as the router, and uses the data extracted from the header of the packet as input to the same consistent hashing algorithm. Thus, regardless of whether a neighbor router is upstream or downstream of a security device, the security device and its neighbor router obtain the same result of the consistent hashing function for packets sent from the router to the device.

[0043] The result or output of the consistent hashing algorithm is a location on a consistent hashing ring. All N security devices are assigned (e.g., uniformly) distributed locations on the consistent hashing ring according to their unique identifier(s) (e.g., their IP address (e.g., loopback address), their device ID, security device configuration information, Layer 2 (e.g., MAC) address, etc.). All security devices and neighbor routers share the assignment (as these security devices and neighbor routers apply the same consistent hashing algorithm with the same information as input). Figure 2 A consistent hashing ring is illustrated in FIG. 1. In FIG. 1, SDi represents the i-th security device of N security devices. Pi and Ki represent a packet flow (P) with the same hash key (K), respectively. Figure 2 In FIG. 1, the consistent hashing ring is illustrated as a circle. However, the consistent hashing ring can be any shape (e.g., a line, a square, a rectangle, etc.).

[0044] First assume a scenario in which no security devices fail. In this scenario, for each packet flow, the primary security device for the given packet flow is the security device that is located after the flow's location on the consistent hashing ring in a given direction (e.g., clockwise) (by applying data extracted from the flow packet's header as input to a consistent hashing algorithm). The M backup security devices for a given flow are the next M security devices that are located after the primary security device on the consistent hashing ring in the given (e.g., clockwise) direction. Unless the primary security device fails and up until the primary security device fails, the packet flow is said to be "owned" by its primary security device.

[0045] Now assume that a security device fails (or otherwise goes down; that is, becomes unavailable). In response to its failure, the security device is treated as removed from the consistent hashing ring. Thus, the packet flow that was previously "owned" by the failed security device will be "owned" (in order) by the next available backup security device in the (e.g., clockwise) direction on the consistent hashing ring. (Note that if "stateful" security is not strictly required, then the packet flow previously owned by the failed security device will be owned by the next available security device in the (e.g., clockwise) direction on the consistent hashing ring out of N security devices.)

[0046] Now assume that the failed security device recovers (or otherwise enters an UP state; that is, becomes available). In response, the security device is treated as added back to the consistent hashing ring. Thus, ownership of the packet flow that was previously migrated from the recovered security device to a backup security device will be returned to the recovered primary security device.

[0047] Figure 3 The association of a packet flow with its primary security device and M backup security devices on the consistent hashing ring is illustrated. As shown, SD1 is assigned as the primary security device for packet flow PI, and devices SD2 through SD M+1 are assigned as the M backup security devices for packet flow PI. The order in which the backup security devices receive the order of packets for flow PI when its primary security device SD1 is DOWN or otherwise unavailable is ordered based on their location relative to other backup security devices on the consistent hashing ring (e.g., in the clockwise direction). That is, security devices D2, D3, and D M+1 are the first, second, and Mth backup devices assigned to packet flow PI.

[0048] The upstream and downstream routers apply a consistent hashing algorithm to identify a primary security device that "owns" a given packet flow. The identified primary security device is used as the next hop (e.g., ECMP) for the packets of the flow. When the primary security device to which the packet flow is assigned is functioning properly (that is, is UP or otherwise available), the router sends the packets of the flow toward its assigned primary security device. Otherwise, the router redirects the packets of the flow toward a backup security device that owns the packet flow (that is, the next available (on the consistent hashing ring) backup security device for the packet flow).

[0049] §4.3 Example Method(s)

[0050] Figure 4A and Figure 4B are flowcharts of example methods 400 and 450, respectively, performed by the upstream or downstream routers and the security devices. Referring first to Figure 4A Different branches of the example method 400 are executed in response to the occurrence of different events. (Event branch point 405) For example, in response to one or more conditions for updating the forwarding tables of the routers being satisfied, the example method 400 executes the left-hand side branch of the example method 400. Note that various conditions can be used to trigger and update the forwarding tables. Such conditions can include, for example, the receipt of packets, the receipt of Interior Gateway Protocol (IGP) information, the receipt of network topology information in a protocol advertisement, etc. Referring to the left-hand side branch, the example method 400 uses a consistent hashing ring algorithm 499 to determine, for each bidirectional flow, a primary security device and M (where M is at least one) secondary security devices. (Block 410) The example method 400 then stores, for each bidirectional flow, the determined primary security device as a next hop and the M secondary security devices as backup next hop(s). (Block 415) The example method 400 then branches back to the event branch point 405 via node A (448).

[0051] Referring back to event branch point 405, the right branch of example method 400 is executed in response to receiving a packet (e.g., from public network 125 in the case that the router running method 400 is upstream router 120, or from internal network 135 in the case that the router running method 400 is downstream router 130). More specifically, example method 400 determines the bidirectional flow to which the packet belongs and determines a next hop (e.g., ECMP). (Block 420) That is, in effect, example method 400 determines the one of the plurality of N security devices to which the determined bidirectional flow is assigned as the primary security device. Next, it is determined whether the primary security device is available. If the primary security device is available (decision 425 = yes), then method 400 forwards the packet via the primary security device (block 430) before the method branches back to event branch point 405 via node A (448). On the other hand, if it is determined that the primary security device is not available (decision 425 = no), then example method 400 determines whether any of the M security devices on the consistent hashing ring (that is, any of the backup next hops) is available. If the primary security device is available (decision 435 = yes), then example method 400 forwards the received new packet to the determined next available security device of the plurality of M security devices on the consistent hashing ring (block 440) before example method 400 branches back to event branch point 405 via node A. On the other hand, if it is determined that none of the M security devices on the consistent hashing ring (that is, any of the backup next hops) is available (decision 445 = no), then example method 400 can invoke one of the following: (A) a no security option in which the packet is forwarded without any security (e.g., by bypassing the pool of security devices, which can not be a practical option as it opens a security hole), (B) a non-stateful security option in which the packet is forwarded by a security device other than the primary device or the M backup devices (e.g., the next available security device of the N security devices) that will not have state information for the flow to which the packet belongs, or (C) an extreme security option in which the packet is discarded (block 445) before example method 400 branches back to event branch point 405 via node A (448).

[0052] Referring next to Figure 4B different branches of example method 450 are executed in response to the occurrence of different events. (Event branch point 452) For example, in response to one or more conditions for configuring a security device being met, the leftmost branch of example method 450 is executed. More specifically, example method 450 uses consistent hashing ring algorithm 499 (which is the same as consistent hashing ring algorithm 499 used to generate consistent hashing ring 499) to determine a primary security device of the plurality of N security devices to which the determined bidirectional flow is assigned. Next, it is determined whether the primary security device is available. If the primary security device is available (decision 455 = yes), then method 450 forwards the packet via the primary security device (block 460) before the method branches back to event branch point 452 via node A (468). On the other hand, if it is determined that the primary security device is not available (decision 455 = no), then example method 450 determines whether any of the M security devices on the consistent hashing ring (that is, any of the backup next hops) is available. If the primary security device is available (decision 465 = yes), then example method 450 forwards the received new packet to the determined next available security device of the plurality of M security devices on the consistent hashing ring (block 470) before example method 450 branches back to event branch point 452 via node A. On the other hand, if it is determined that none of the M security devices on the consistent hashing ring (that is, any of the backup next hops) is available (decision 465 = no), then example method 450 can invoke one of the following: (A) a no security option in which the packet is forwarded without any security (e.g., by bypassing the pool of security devices, which can not be a practical option as it opens a security hole), (B) a non-stateful security option in which the packet is forwarded by a security device other than the primary device or the M backup devices (e.g., the next available security device of the N security devices) that will not have state information for the flow to which the packet belongs, or (C) an extreme security option in which the packet is discarded (block 475) before example method 450 branches back to event branch point 452 via node A (468). Figure 4Athe same as used by the example method 400 of FIG. 4), assigning a security device to a location on the consistent hashing ring (block 460) and determining the next M security devices on the consistent hashing ring (block 462). The example method 450 then branches back to the event branch point 452 via node B (495).

[0053] Referring back to the event branch point 452, in response to receiving a packet from (A) one of the at least one upstream router or (B) one of the at least one downstream router, the example method 450 performs a second branch from the left. More specifically, the example method 450 determines a bidirectional flow to which the packet belongs (block 470), updates a state of a given bidirectional flow to which the received packet belongs (or creates a new state if one does not already exist) (block 472), and applies security to the received packet (e.g., forwards the packet, drops the packet, duplicates the packet, redirects the packet, etc.) (block 474). The example method 450 then branches back to the event branch point 452 via node B (495).

[0054] Referring back to the event branch point 452, in response to one or more conditions for sharing / synchronizing flow state information being met, a second branch from the right of the example method 450 is performed. More specifically, the example method 450 causes the stored state of a given bidirectional flow for which a security device is a primary security device (or a security device is a backup security device that receives packets in response to the primary security device being DOWN) to be synchronized with a set of M backup security devices assigned to the given bidirectional flow that are UP, that is, the security devices that are in the next M security devices of the primary security device on the consistent hashing ring that are in an UP state (e.g., in a clockwise direction), but not all of the N security devices. (Block 480) The example method 450 then branches back to the event branch point 452 via node B (495).

[0055] Referring back to event branch point 452, in response to a security device entering an "UP" state, the rightmost branch of example method 450 is executed. More specifically, example method 450 uses a consistent hashing ring algorithm 499 to find both (1) a first set of M available security devices that precede the security device on the consistent hashing ring and (2) a second set of M available security devices that follow the security device on the consistent hashing ring. (Block 490) Example method 450 then synchronizes state information for any of the plurality of bidirectional flows to which each of the first set of M security devices is assigned as a primary security device with the UP security device (block 492) and synchronizes state information for any of the plurality of bidirectional flows to which the UP security device executing method 450 is assigned as a primary security device with the most recent security device in the second set of M security devices that is in an UP state (block 494). Example method 450 then branches back to event branch point 452 via node B branch 495.

[0056] Referring back to block 490, although this action of using a consistent hashing ring algorithm 499 to find both (1) a first set of M security devices that precede the security device on the consistent hashing ring and (2) a second set of M security devices that follow the security device on the consistent hashing ring in response to a security device entering an "UP" state can be performed, this finding action can be performed regardless of whether a security device enters an UP state.

[0057] Note that when a security device enters a DOWN state, example method 450 need not do anything because this will be recognized by upstream and downstream routers. (Recall, e.g., blocks 425 and 435 of Figure 4A

[0058] §4.4 Example Apparatus

[0059] Figure 5 Two data forwarding systems 510 and 520 are coupled via a communication link 530. The link can be a physical link or a "wireless" link. For example, data forwarding systems 510 and 520 can be a router and a security device, respectively. Note that security device 520 can also include the functionality of a router. Data forwarding systems 510, 520 can each include a control component (e.g., a routing engine) 514, 524 and a forwarding component 512, 522, respectively. Each data forwarding system 510, 520 includes one or more interfaces 516, 526 that terminate one or more communication links 530.

[0060] Still referring to Figure 5 ​The control component 514 of the router 510 can be configured to share the router ID, the total number of routers, and the properties of the consistent hashing algorithm with the network security device(s) 520. The control component 524 of the network security device 520 can be configured to share the device ID, the total number of security devices, and the properties of the consistent hashing algorithm with peer routers. The forwarding component 512 of the router 510 can apply the consistent hashing algorithm to select ECMP paths to forward packets of the flow(s) to the security device(s). Finally, the forwarding component 522 of the network security device 520 can (1) receive packets from the router and update the state of the corresponding flow, (2) synchronize the flow state with the working backup security device for a given flow, and (3) detect recovery of a peer security device and synchronize the state(s) of the flow(s) to the recovered peer security device. Example embodiments in accordance with the present specification can be integrated in the control plane and data plane of network routers and security devices, and can be implemented with software and / or hardware as advanced features of routers and security devices.

[0061] As just discussed above and with reference to Figure 6 Some example data forwarding devices, such as the security device and router 600, include a control component (e.g., a routing engine) 610 and a packet forwarding component (e.g., a packet forwarding engine) 690.

[0062] The control component 610 can include an operating system (OS) kernel 620, routing protocol process(es) 630, label-based forwarding protocol process(es) 640, interface process(es) 650, user interface (e.g., command line interface) process(es) 660, and chassis process(es) 670, and can store routing table(s) 639, label forwarding information 645, and forwarding (e.g., routing-based and / or label-based) table(s) 680. As shown, the routing protocol process(es) 630 can support routing protocols such as Routing Information Protocol (“RIP”) 631, Intermediate-System-to-Intermediate-System Protocol (“IS-IS”) 632, Open Shortest Path First Protocol (“OSPF”) 633, Enhanced Interior Gateway Routing Protocol (“EIGRP”) 634, and Border Gateway Protocol (“BGP”) 635, and the label-based forwarding protocol process(es) 640 can support protocols such as BGP 635, Label Distribution Protocol (“LDP”) 636, Resource Reservation Protocol (“RSVP”) 637, EVPN 638, and L2VPN 639. One or more components (not shown) can permit a user 665 to interact with the user interface process(es) 660. Similarly, one or more components (not shown) can permit external devices to interact with one or more of the router protocol process(es) 630, the label-based forwarding protocol process(es) 640, the interface process(es) 650, and the chassis process(es) 670 via SNMP 685, and such processes can send information to external devices via SNMP 685.

[0063] The packet forwarding component 690 can include a microkernel 692, interface process(es) 693, ASIC drivers 694, chassis process(es) 695, and forwarding (e.g., routing-based and / or label-based) table(s) 696 over hardware components (e.g., ASICs, switch fabric, optics, etc.) 691.

[0064] In Figure 6In the example router 600, the control component 610 handles tasks such as executing routing protocols, executing label-based forwarding protocols, controlling packet processing, etc., which frees the packet forwarding component 690 to quickly forward received packets. That is, received control packets (e.g., routing protocol packets and / or label-based forwarding protocol packets) are not fully processed on the packet forwarding component 690 itself, but are passed to the control component 610, thereby reducing the workload that the packet forwarding component 690 must perform and freeing it to process packets to be efficiently forwarded. Thus, the control component 610 is primarily responsible for running routing protocols and / or label-based forwarding protocols, maintaining routing tables and / or label forwarding information, sending forwarding table updates to the packet forwarding component 690, and performing system management. The example control component 610 can process routing protocol packets, provide a management interface, provide configuration management, perform accounting, and provide alarms. The processes 630, 640, 650, 660, and 670 can be modular and can interact with the OS kernel 620. That is, almost all of the processes directly communicate with the OS kernel 620. Using modular software that cleanly separates the processes from each other isolates problems with a given process so that such problems do not affect other processes that can be running. Additionally, using modular software facilitates easier scaling.

[0065] Still referring to Figure 6 The example OS kernel 620 can contain an application programming interface ("API") system for external program invocation and scripting capabilities. The control component 610 can be an Intel PCI platform running the OS from flash memory, with alternate copies stored on the router's hard disk. The OS kernel 620 layers on top of the Intel PCI platform and establishes communication between the Intel PCI platform and the processes of the control component 610. The OS kernel 620 also ensures that the forwarding tables 696 used by the packet forwarding component 690 are synchronized with those 680 in the control component 610. Thus, in addition to providing the underlying infrastructure for the control component 610 software processes, the OS kernel 620 also provides the link between the control component 610 and the packet forwarding component 690.

[0066] Referring to Figure 6Routing protocol process(es) 630 provide routing and routing control functions within the platform. In this example, RIP 631, ISIS 632, OSPF 633, and EIGRP 634 (and BGP 635) protocols are provided. Naturally, other routing protocols can additionally or alternatively be provided. Similarly, label-based forwarding protocol process(es) 640 provide label forwarding and label control functions. In this example, LDP 636, RSVP 637, EVPN 638, and L2VPN 639 (and BGP 635) protocols are provided. Naturally, other label-based forwarding protocols (e.g., MPLS, SR, etc.) can additionally or alternatively be provided. In the example router 600, routing table(s) 639 are produced by routing protocol process(es) 630, while label forwarding information 645 is produced by label-based forwarding protocol process(es) 640.

[0067] Still referring to Figure 6 Interface process(es) 650 perform configuration and encapsulation of physical interfaces.

[0068] The example control component 610 can provide several ways to manage the router. For example, the example control component 610 can provide user interface process(es) 660 that allow system operator 665 to interact with the system through configuration, modification, and monitoring. SNMP 685 allows SNMP-capable systems to communicate with the router platform. This also allows the platform to provide necessary SNMP information to external agents. For example, SNMP 685 can permit management of the system from a network management station running software such as Hewlett-Packard's OpenView through a framework such as Hewlett-Packard's Network Node Manager ("HP-NNM"). Accounting of packets (commonly referred to as traffic statistics) can be performed by the control component 610, thereby avoiding slowing traffic forwarding by the packet forwarding component 690.

[0069] Although not shown, the example router 600 can provide out-of-band management, RS-232 DB9 ports for serial console and remote management access, and tertiary storage using removable PC cards. Further, although not shown, a service interface positioned in front of the chassis provides an external view of the internal workings of the router. This can be used as a troubleshooting tool, a monitoring tool, or both. The service interface can include LED indicators, alarm indicators, control component ports, and / or a display screen. Finally, the service interface can provide interaction with a command line interface ("CLI") 660 via a console port, an auxiliary port, and / or a management Ethernet port.

[0070] The packet forwarding component 690 is responsible for outputting received packets correctly and as quickly as possible. If there is no entry in the forwarding table for a given destination or given label and the packet forwarding component 690 cannot perform the forwarding on its own, the packet forwarding component 690 can send the packet to the control component 610 for processing to the unknown destination. The example packet forwarding component 690 is designed to perform layer 2 and layer 3 switching, routing lookup, and fast packet forwarding.

[0071] As shown in Figure 6 The example packet forwarding component 690 has an embedded microkernel 692 over hardware components 691, interface process(es) 693, ASIC drivers 694, and rack process(es) 695, and stores forwarding (e.g., routing-based and / or label-based) table(s) 696. The microkernel 692 interacts with the interface process(es) 693 and the rack process(es) 695 to monitor and control these functions. The interface process(es) 692 have direct communication with the OS kernel 620 of the control component 610. This communication includes forwarding exception and control packets to the control component 610, receiving packets to be forwarded, receiving forwarding table updates, providing information to the control component 610 about the health of the packet forwarding component 690, and permitting configuration of the interface of the user interface (e.g., CLI) process(es) 660 from the control component 610. The stored forwarding table(s) 696 are static until a new forwarding table is received from the control component 610. The interface process(es) 693 use the forwarding table(s) 696 to look up next hop information. The interface process(es) 693 also have direct communication with the distributed ASICs. Finally, the rack process(es) 695 can communicate directly with the microkernel 692 and the ASIC drivers 694.

[0072] Referring back to Figure 5 , certain functions of a router according to the present specification can be performed by the control component 514, while other functions are performed by the forwarding component 512. Similarly, certain functions of a network security appliance according to the present specification can be performed by the control component 524, while other functions are performed by the forwarding component 522.

[0073] Figure 7 is an example of how ASICs can be distributed in the packet forwarding component 690 to divide the responsibility for packet forwarding. As shown in Figure 7 , the ASICs of the packet forwarding component 690 can be distributed on physical interface cards (“PICs”) 710, flexible PIC concentrators (“FPCs”) 720, midplane or backplane 730, and system control board(s) 740 (for switching and / or forwarding). The switching fabric is also shown as a system switch board (“SSB”) or switch and forwarding module (“SFM”) 750 (which can be as inFigure 8A and 8B the switching fabric 750') shown in FIG. 7B. Each PIC 710 includes one or more PIC I / O managers 715. Each FPC 720 includes one or more I / O managers 722 with associated memory 724 (which can be RDRAM 724' as shown in FIG. 7B, for example) respectively. The midplane / backplane 730 includes buffer managers 735a, 735b. Finally, the system control board 740 includes an Internet processor 742 and an instance of a forwarding table 744 (recall, for example, 696 of FIG. 6). Figure 8A and Figure 8B the switching fabric 750') shown in FIG. 7B. Each PIC 710 includes one or more PIC I / O managers 715. Each FPC 720 includes one or more I / O managers 722 with associated memory 724 (which can be RDRAM 724' as shown in FIG. 7B, for example) respectively. The midplane / backplane 730 includes buffer managers 735a, 735b. Finally, the system control board 740 includes an Internet processor 742 and an instance of a forwarding table 744 (recall, for example, 696 of FIG. 6). Figure 6

[0074] Still referring to FIG. 7B, the PIC 710 contains interface ports. Each PIC 710 can be plugged into the FPC 720. Each individual PIC 710 can contain an ASIC that handles medium-specific functions such as framing or encapsulation. Some example PICs 710 provide SDH / SONET, ATM, Gigabit Ethernet, Fast Ethernet, and / or DS3 / E3 interface ports. Figure 7 The FPC 720 can contain one or more PICs 710 and can carry signals from the PICs 710 as shown in FIG. 7B to the midplane / backplane 730.

[0075] Figure 7 The midplane / backplane 730 holds line cards. When a line card is inserted from the front into the chassis of an example router, the line card can be connected into the midplane / backplane 730. The control components (e.g., routing engines) 610 can be inserted from the back of the chassis into the back of the midplane / backplane 730. The midplane / backplane 730 can carry electrical (or optical) signals and power to each line card and control component 610.

[0076] The system control board 740 can perform forwarding lookups. The system control board 740 can also communicate errors to the routing engines. Further, the system control board 740 can also monitor the condition of the router based on information it receives from sensors. If an abnormal condition is detected, the system control board 740 can immediately notify the control components 610.

[0077] Referring to FIG. 7B,

[0078] FIG. 7B, Figure 7 FIG. 7B, Figure 8A and Figure 8B ​​In some example routers, each of the PICs 710, 710' contains at least one I / O manager ASIC 715 responsible for media-specific tasks such as encapsulation. Packets traverse these I / O ASICs in the process of entering and exiting the router. The I / O manager ASICs 715 on the PICs 710, 710' are responsible for managing connections with the I / O manager ASICs 722 on the FPCs 720, 720', managing link layer framing and creating bit streams, performing cyclic redundancy checks (CRCs), and detecting link layer errors and generating alarms when appropriate. The FPCs 720 include another I / O manager ASIC 722. This ASIC 722 (shown as Layer 2 / 3 packet processing component 710' / 720') takes packets from the PICs 710 and breaks them into (e.g., 74 byte) memory blocks. The FPC I / O manager ASIC 722 (shown as Layer 2 / 3 packet processing component 710' / 720') sends the blocks to a first distributed buffer manager (DBM) 735a (shown as switch interface component 735a') that decodes encapsulation and protocol-specific information, counts packets and bytes for each logical circuit, verifies packet integrity, and applies class of service (CoS) rules to the packets. At this point, the packets are first written to memory. More specifically, the example DBM ASIC 735 / 735a' manages packets across all FPCs 720 and writes them to shared memory 724. In parallel, the first DBM ASIC 735 / 735a' also extracts information about the destination of the packets and passes this forwarding-related information to an Internet processor 742 / 742'. The Internet processor 742 / 742' performs a route lookup using a forwarding table 744 and sends information to a second DBM ASIC 735b'. The Internet processor ASIC 742 / 742' also collects exceptional packets (i.e., packets that do not have a forwarding table entry) and sends them to the control component 610. The second DBM ASIC 725 (shown as queuing and memory interface component 735b') then takes this information and 74 byte blocks and forwards them to the I / O manager ASIC 722 of the egress FPC 720 / 720' (or multiple egress FPCs) for reassembly (in the case of multicast).

[0079] The I / O manager ASIC 722 on the egress FPC 720 / 720' can perform several value-added services. In addition to incrementing the Time-to-Live ("TTL") value and repackaging packets for processing by the PIC 710, it can also apply Class of Service (CoS) rules. To do this, it can queue pointers to packets in one of the available queues, each sharing the link bandwidth, before applying the rules. Queuing can be based on various rules. Therefore, the I / O manager ASIC 722 on the egress FPC 720 / 720' can be responsible for receiving blocks from the second DBM ASIC 735 / 735', incrementing the TTL value, queuing pointers to packets, repackaging blocks if necessary, and sending the encapsulated packets to the PIC I / O manager ASIC 715 before applying CoS rules.

[0080] Figure 9 This is a flowchart of an example method 900 for providing packet forwarding in an example router. When a packet is received on an ingress (incoming) port or interface, the main actions of method 900 are triggered. (Event 910) The required checksum, type, and frame checks for the medium type it serves are performed, and the packet is output as a serial bit stream. (Box 920) The packet is then decapsulated and parsed into (e.g., 64-byte) blocks. (Box 930) The packet is written to a buffer memory and forwarding information is passed on the Internet processor. (Box 940) The route is then looked up in the forwarding table using the passed forwarding information. (Box 950) Note that the forwarding table can typically handle unicast packets without the set options (e.g., accounting) as well as multicast packets that already have cached entries. Therefore, if these conditions are determined to be met (the "yes" branch of decision 960), the packet forwarding component looks for the next hop and egress interface, and forwards (or queues) the packet to the next hop via the egress interface before leaving method 900 (node ​​990) (Box 970). Otherwise, if these conditions are not met (the "No" branch of decision 960), then forwarding information is sent to control component 610 for advanced forwarding resolution (box 980) before leaving method 900 (node ​​990).

[0081] Returning to reference box 970, you can group and queue the data. In fact, as previously referenced... Figure 7As stated, pointers to packets can be queued. The packets themselves can remain in shared memory. Thus, all queuing decisions and CoS rules can be applied in the absence of actual packets. When the pointer for a packet reaches the front of the line, the I / O manager ASIC 722 can send a request for the packet to the second DBM ASIC 735b. The DBM ASIC 735 reads the block from shared memory and sends it to the I / O manager ASIC 722 on the FPC 720, which then serializes the bits and sends them to the media specific ASIC of the egress interface. The I / O manager ASIC 715 on the egress PIC 710 can apply physical layer framing, perform CRC, and send the bit stream out over the link.

[0082] Referring back to Figure 9 block 980 and Figure 7 regarding the transmission of control and exception packets, the system control board 740 handles almost all exception packets. For example, the system control board 740 can pass exception packets to the control component 610.

[0083] While example embodiments in accordance with the present description can be implemented on example routers and / or security appliances of Figure 5 or Figure 6 , at least some portions of embodiments in accordance with the present description can be implemented on communication network nodes such as routers (real and / or virtualized) having different architectures, security appliances (real and / or virtualized), switches, etc. More generally, embodiments in accordance with the present description can be implemented on an example system 900 as illustrated in Figure 10 .

[0084] Figure 10 are capable of performing the described methods (review, for example, Figure 4A and Figure 4BA block diagram of an example machine 1000 capable of implementing one or more methods of the present specification and / or storing information used and / or generated by such processes is shown in FIG. 1. The example machine 1000 includes one or more processors 1010, one or more input / output interface units 1030, one or more storage devices 1020, and one or more system buses and / or networks 1040 for supporting communications within the coupled elements. One or more input devices 1032 and one or more output devices 1034 can be coupled with the one or more input / output interfaces 1030. The one or more processors 1010 can execute machine executable instructions (e.g., C or C++ running on a Linux operating system that is widely available from multiple vendors) to implement one or more aspects of the present specification. At least a portion of the machine executable instructions can be stored (temporarily or more permanently) on the one or more storage devices 1020 and / or can be received from an external source via the one or more input interface units 1030. The machine executable instructions can be stored as various software modules, each of which performs one or more operations. Functional software modules are examples of components of the present specification.

[0085] In some embodiments according to the present specification, the processor 1010 can be one or more real and / or virtualized microprocessors and / or ASICs. The bus 1040 can include a system bus. The storage device 1020 can include system memory, such as read-only memory (ROM) and / or random access memory (RAM). The storage device 1020 can also include a hard disk drive for reading from and writing to a hard disk, a magnetic disk drive for reading from or writing to a (e.g., removable) magnetic disk, an optical disk drive for reading from or writing to a removable (magnetic) optical disk (such as a compact disk or other (magnetic) optical media), or a solid-state nonvolatile storage device.

[0086] Some example embodiments according to the present specification can also be provided as machine-readable media for storing the machine executable instructions. The machine-readable media can be non-transitory and can include, but not limited to, flash memory, optical disks, CD-ROMs, DVD ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards or any other type of media suitable for storing electronic instructions. For example, example embodiments according to the present specification can be downloaded as a computer program which can be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of a communication link (e.g., a modem or network connection) and stored on a non-transitory storage medium. The machine-readable media can also be referred to as a processor-readable medium.

[0087] Some or all parts of the example embodiments according to the present specification (or components or modules thereof) can be implemented in hardware (such as one or more field-programmable gate arrays ("FPGAs"), one or more integrated circuits (such as ASICs), one or more network processors, etc.). Alternatively or additionally, embodiments according to the present specification (or components or modules thereof) can be implemented as stored program instructions that are executed by a processor. Such hardware and / or software can be provided in a packet-switched data (e.g., packet, cell, etc.) forwarding device (e.g., a switch, router, etc.), a laptop computer, a desktop computer, a tablet computer, a mobile phone, or any device having computing and networking capabilities.

[0088] §4.5 Improvements, Alternatives, and Extensions

[0089] §4.5.1 Example Way of Determining M Backup Security Devices

[0090] In some example embodiments according to the present specification, a primary security device identifies M (where M is at least one) backup security devices for any packet flow(s) that the primary security device "owns" as follows. When determining its M backup security devices, the primary security device can (conceptually) remove itself from a consistent hashing ring and use a consistent hashing ring algorithm to find backup devices for the packet flow(s) that it "owns." If M is greater than one, the primary device can (conceptually) remove itself and any backup security device(s) that have already been found from the consistent hashing ring and then use the consistent hashing algorithm again to identify the next backup security device. This can be repeated until all M backup security devices are identified for the packet flow(s) that the primary security device "owns." After all M backup devices have been identified, if the backup devices are functioning properly, the primary security device only synchronizes its state for its packet flow(s) to the M backup devices (recall, e.g., block 480 of FIG. 4). Figure 4B

[0091] Note that the state of each of the security devices can be tracked by one or more upstream routers and one or more downstream routers. Note that the UP / DOWN / COLD state of a security device is different from the flow state or bidirectional flow state.

[0092] In example embodiments according to the present specification, consistent hashing is maintained even when security devices (and / or links) are removed or added.

[0093] Note that in some example embodiments according to the present specification, the security devices are capable of running a routing protocol, such as eBGP, OSPF, IS-IS, etc.

[0094] §4.5.1 Example Way of Determining M Backup Security Devices ​

[0095] Security device(s) when multiple virtual security devices are associated with a physical (or real) security device

[0096] The distribution of security devices on the consistent hashing ring can be uniform or non-uniform. The distribution of security devices on the consistent hashing ring can even be augmented with virtual security devices associated with each real (that is, physical) security device to enhance the results of the consistent hashing algorithm. For example, one augmented example embodiment uses virtual security devices of real security devices to more uniformly distribute flows to all security devices in a consistent hashing mechanism. Another augmented example embodiment makes the number of virtual security devices of each real security device proportional to the capacity of the corresponding real security device to better balance flows or workloads among the various security devices (according to the individual capacities of the various security devices).

[0097] To more uniformly distribute flows in a consistent hashing mechanism, in some example embodiments according to the present specification, a physical network security device can be mapped to multiple virtual network security devices, and each of these virtual security devices assigned a location on a consistent hashing ring. Figure 13 illustrates multiple virtual security devices (three in this illustrative example) mapped to each physical network security device. Still referring to Figure 13 , these virtual security devices are assigned on a consistent hashing ring. In , SD i.0 , SD i.1 , and SD i.2 are these virtual security devices associated with physical security device SD i , where i is an integer from 0 to n.

[0098] Figure 13 Using virtual security devices as just described does not require changing the way primary security devices are identified for packet flows using a consistent hashing mechanism. More specifically, when virtual security devices are used, the method for identifying M backup SDs of a primary SD along the consistent hashing ring ignores or skips virtual security devices that overlap (that is, are associated with) the primary SD and its previously identified backup SD(s). Thus, M unique physical or real SDs will be identified for backup primary SDs. For example, referring to , packet flow PI 1305 has a corresponding hash key K1. The primary SD for packet flow PI is SD1 (identified by its first associated virtual security device SD 1.0 1310. Further, the two unique backup SDs for packet flow PI are SD2 (identified by virtual security device SD 2,2 1320), and then SD3 (identified by virtual security device SD 3.2identified P1 is ignored or skipped when backing up SD 1.1 and SD 2.0 because SD 1.1 and SD 2.0 are respectively associated with the same physical or real secure device as the physical or real secure device of the previously identified backup SD 1.0 and SD 2.2 .

[0099] Figure 14 Fig. illustrates how the ownership of the consistent hashing ring is divided into six physical or real SDs (SD1 to SD6) (mapping each of these physical or real SDs to three virtual nodes (SD 1.i to SD 6.i with i = 0, 1 or 2)) and also illustrates the state synchronization between the SDs. As an example, the three dotted circle marks can be the portion of the consistent hashing ring owned by the physical secure device SD2, mapping this portion to three virtual secure devices; i.e., SD 2.0 , SD 2.1 and SD 2.2 . Each of these virtual secure devices (1) synchronizes its flow state to its two backup unique SDs and (2) receives the flow state synchronized from the two unique SDs whose virtual SDs are backed up by the virtual nodes of SD2. More specifically, virtual secure device SD 2.0 (mapped from physical secure device SD2) synchronizes its flow state to virtual secure devices SD 4.2 and SD 1.1 1410a, virtual secure device SD 2.1 (mapped from physical secure device SD2) synchronizes its flow state to virtual secure devices SD 4.0 and SD 6.2 1410b, and virtual secure device SD 2.2 (mapped from physical secure device SD2) synchronizes its flow state to virtual secure devices SD 6.0 and SD 4.1 1410c. Further, virtual secure devices SD 5.1 and SD 6.1 synchronize their flow state to virtual secure device SD 2.0 1420a, virtual secure devices SD 3.0 and SD 5.2 synchronize their flow state to virtual secure device SD 2.1 1420b, and virtual secure devices SD 5.0 and SD 1.2 synchronize their flow state to virtual secure device SD2.2 1420c.

[0100] §4.5.2 Example ways of sharing flow state information among peers

[0101] In some example embodiments according to the present specification, synchronization of flow state among suitable security devices can be accomplished via known network protocols, such as (for example) back-fiber, BGP peering, IP network, TCP (layer 2), UDP, etc. In other example embodiments, a proprietary protocol can be used to share such flow state information among suitable security devices. In some example embodiments according to the present specification, flow state information can be provided as a "data object" delivered from a "generator" to devices of interest (or "peers" of the "generator"). For example, BGP can be used to find a path to a peering security device, and UDP can be used as a transport layer to carry the link state data object. In some example embodiments according to the present specification, the communication channel between peering security devices can be safeguarded or protected by an IPsec VPN tunnel, such that an outside device cannot observe the link state data object.

[0102] In some example embodiments according to the present specification, when M is greater than one and a backup security device receives packets redirected by a router from a failed primary device, the backup security device synchronizes the state(s) of the flow(s) that have been assigned to the remaining M backup security devices. In this case, the backup security device can use a consistent hashing ring algorithm to identify the primary security device and M backup security devices for the packet flow(s), assuming no security device has failed. Then, for each given flow, the backup security device then only synchronizes the state(s) of these flows to the remaining identified M backup devices that are functioning. For example, as illustrated in Figure 11 l the primary security device SD l and the first backup security device SD2 are both unavailable (e.g. both failed). Thus, the second backup security device SD3 for P1 now "has" or "owns" the packet flow P1 and receives its packets from the router. Then, the security device SD3 synchronizes the flow state S1 of P1 to the remaining backup devices SD4 to SD M+1 .

[0103] §4.5.1 Example ways of determining primary and backup security devices as next hop

[0104] ​In some example embodiments according to the present specification, the hash function can be the same as the hash function used in Equal Cost Multi Path (ECMP) forwarding. ECMP is a network routing policy that allows traffic of the same session or flow (that is, traffic with the same source and destination) to be transmitted across multiple equal cost paths. It is a mechanism that allows one to load balance traffic and increase bandwidth by making more efficient use of otherwise unused bandwidth on links to the same destination.

[0105] When forwarding packets, routing techniques decide which next hop path to use. In making that decision, the device considers the packet header fields that identify the flow. When ECMP is used, the equal cost next hop paths are identified based on a routing metric calculation and a hash algorithm. That is, the equal cost routes have the same preference and metric values and the same network cost. The ECMP process identifies a set of routes, each of which is a legal equal cost next hop towards the destination. The identified routes are referred to as an ECMP set. Because it addresses only the next hop destination, ECMP can be used with most routing protocols. An Equal Cost Multi Path (ECMP) set is formed when the routing table contains multiple next hop addresses for the same destination that are equal cost (equal preference and metric values). If an ECMP set exists for an active route, a hash algorithm can be used to select one of the next hop addresses in the ECMP set to install in the forwarding table. Some router operating systems, such as Junos from Juniper Networks of Sunnyvale, California, can be configured to install multiple next hop entries in the ECMP set in the forwarding table. On such Juniper Networks devices, per-flow load balancing can be performed to spread traffic across multiple paths between the routing devices. On some Juniper Networks security devices, the source IP address and destination IP address and protocol are examined to determine individual traffic flows. Packets of the same flow are forwarded on the same interface; the interface does not change when there are additions or changes to the ECMP set. This is important for features such as source Network Address Translation (NAT), where the translation is performed only during the first path of the session establishment of IDP, Application Layer Gateway (ALG), and routing-based Virtual Private Network (VPN) tunnels. If a packet arrives on a given interface in the ECMP set, the security device ensures that the reverse traffic is forwarded through the same interface.

[0106] Further, on some security appliances from Juniper Networks, the maximum number of next-hop addresses that can be installed in an ECMP set in the forwarding table is 16. If there are more than 16 next-hop addresses in the ECMP set, only the first 16 addresses are used. In a "rack cluster" deployment, a "local" interface is an interface on the same node as the interface that the packet arrived on, and a "remote" interface is an interface on another rack cluster node. If an ECMP route has both a local and a remote interface in a rack cluster, the local interface is preferred for the next hop.

[0107] If a next-hop address is no longer part of an ECMP set, or if the next-hop address is removed from the routing table due to a route change, the flows using the next hop are re-routed and the session is unaffected. Re-routing of flows also occurs if there is a configuration change that removes the next-hop address or if an administrator removes the next-hop interface without deleting the next-hop interface. If a next-hop address is removed from the routing table due to an interface being deleted or a session being intentionally cleared, the session is terminated and not re-routed.

[0108] To configure ECMP flow-based forwarding on some security appliances from Juniper Networks, first define a load-balancing routing policy by including one or more policy statements under the {edit policy options} hierarchy level, where actions are load-balanced per packet. Then apply the routing policy to the routes exported from the routing table to the forwarding table. To do this, include the forwarding table and export configuration statements under the {edit routing options} hierarchy level.

[0109] ECMP on SRX series devices and vSRX instances from Juniper Networks can be configured to implement per-flow load balancing to spread traffic across multiple paths between routing devices. Equal-cost routes have the same preference and metric values. These devices examine the source IP address, destination IP address, and protocol to determine individual traffic flows. Traffic permitted by a security policy with the same source IP address, destination IP address, and protocol number is forwarded to the same next hop. Junos OS on these Juniper Networks devices uses flow information in its hashing logic.

[0110] For Junos OS SRX series devices and vSRX instances from Juniper Networks, an ECMP set is formed when the routing table contains multiple next-hop addresses for an equal-cost same destination. ECMP allows multiple next-hop entries in the ECMP set to be installed in the forwarding table. Packets of the same flow are forwarded on the same interface; the interface does not change when there are additions or changes to the ECMP set.

[0111] If an ECMP set exists for an active route, Junos OS uses a hashing algorithm to select one of the next-hop addresses in the ECMP set to install in the forwarding table.

[0112] In some versions of Junos OS from Juniper Networks, if ECMP support is enabled for reverse traffic, SRX Series devices use a hashing algorithm to determine the interface for reverse traffic in a flow. The process is similar to asymmetric routing, where a packet traverses from a source to a destination in one path, and a different path is taken when the packet returns to the source. If the feature is not enabled, SRX Series devices select a route in the ECMP set that leads to the incoming interface for reverse traffic, which is the default behavior.

[0113] With some routers and security appliances from Juniper Networks, the allow reverse ecmp configuration statement in the {edit security flow} hierarchy can be used to configure ECMP flow-based forwarding to use a hashing algorithm when selecting a route in an ECMP set for reverse traffic. That is, with the feature enabled, instead of selecting a route that leads to the incoming interface, the device uses a hashing algorithm to select a route in the ECMP set for reverse traffic. If the ECMP flow-based policy is zone-based, ECMP reverse lookup support ensures that the egress interface for reverse traffic is in the same zone as the ingress interface for the traffic.

[0114] Example operations of the (multiple) example methods

[0115] When a security appliance recovers, before the recovered appliance is ready to receive and process packets from its neighbor routers, the security appliance synchronizes the (multiple) flow state from peer security appliances and installs the (multiple) flow state in local storage. (Recall, e.g., the rightmost branch of Figure 4B See Figure 12 , according to the consistent hashing ring algorithm, the recovered security appliance SD k will be the primary security appliance for packets for which the recovered security appliance "owns" the flow. (Recall, e.g., block 494 of Figure 4B and with respect to SD Figure 12 in k See SD k+1 to SD k+M ). On the other hand, the recovered appliance SD k will back up, clockwise on the consistent hashing ring, M peer security appliances (SD k-M to SD k-1 ) that precede itself. (Recall block 492 of Figure 4B ).

[0116] Therefore, the recovered device SD k Synchronize and install (1) from the restored device SD card k The following M security devices (SD) k+1 To SD k+M (2) The flow states of all the most recently working devices between (1) and (2) the security device SD located on the consistent hash ring that has been recovered. k The previous M security devices (SD) k-M To SD k-1 The state of the stream owned by any security device in the M security devices (SDs), where each security device originates from one of these M security devices (SDs). k-M To SD k-1 The most recently operating secure device between the upstream and downstream routers. After the restored device completes the aforementioned flow state synchronization, the device begins receiving and processing packets from the upstream and / or downstream routers.

[0117] Figure 15 The diagram illustrates the interaction between upstream router 1520 and downstream router 1530. Figure 4A The operation corresponding to the left branch. Figure 15 In this context, network portion 1500 includes a pool or group of twelve security devices (SDs) 1510 arranged between upstream router 1520 and downstream router 1530. Upstream router 1520 is arranged between security devices 1510 and a first network 1525 (such as...). Figure 1 The downstream router 1530 is positioned between the security device 1510 and the second network 1535 (such as, for example, the public network 125). Figure 1 The internal network 155). The pool or group of SD 1510 includes twelve SDs (SD1 to SD2). 12 Therefore, N = 12. For this example, assume M = 2.

[0118] Suppose upstream router 1520 receives packets from flow A to flow B (e.g., from IP source address A to IP destination address B) (referred to as flow AB). Information from the packet headers is extracted and provided to a hash function (e.g., ECMP) 1522 configured with parameters 1599 of multiple consistent hashing algorithms. As a result of the hashing, upstream router 1520 installs forwarding information for flow AB, where the primary next hop is set to SD3, the first backup next hop is SD4, and the second backup next hop is SD5. These three SDs are enclosed within a dashed ellipse.

[0119] Further assume that upstream router 1520 receives a packet from flow C to D (e.g., from IP source address C to IP destination address D), referred to as flow CD. The information from the packet(s) header(s) is extracted and provided to the hash function 1522 configured with the consistency hash algorithm parameter(s) 1599. Assume that as a result of the hash, upstream router 1520 installs forwarding information for flow CD with the primary next hop set to SD7, the first backup next hop to SD8, and the second backup next hop to SD9. These three SDs are enclosed in a solid line ellipse.

[0120] Finally, assume that the upstream router receives a packet from flow A to D (e.g., from IP source address A to IP destination address D), referred to as flow AD. The information from the packet(s) header(s) is extracted and provided to the hash function 1522 configured with the consistency hash algorithm parameter(s) 1599. Assume that as a result of the hash, upstream router 1520 installs forwarding information for flow AD with the primary next hop set to SD5, the first backup next hop to SD6, and the second backup next hop to SD7. These three SDs are enclosed in a dashed-dotted line ellipse.

[0121] Note that some SDs have different responsibilities for different flows. For example, SD5 is the second backup next hop for flow AB, but it is the primary next hop for flow AD. As another example, SD7 is the primary next hop for flow CD, but it is the second backup next hop for flow AD.

[0122] It is desirable to have bidirectional flow state information tracked by the appropriate SDs among the SDs. To this end, refer to downstream router 1530, an exchange function 1536 is used to exchange the information extracted from the packet header(s) before providing the information to the hash function 1532 configured with the consistency hash algorithm parameter(s) 1599. Assume that downstream router 1530 receives a packet from flow B to A (e.g., from IP source address B to IP destination address A), referred to as flow BA. The information from the packet(s) header(s) is extracted, exchanged, and provided to the hash function (e.g., ECMP) 1532 configured with the consistency hash algorithm parameter(s) 1599. As a result of the hash, downstream router 1530 installs forwarding information for flow BA with the primary next hop set to SD3, the first backup next hop to SD4, and the second backup next hop to SD5, since the extracted header information after the exchange from flow BA will be the same as the header information from flow AB. These three SDs are enclosed in a dashed line ellipse. As this example illustrates, the same SD has the same responsibilities for upstream to downstream flow AB and downstream to upstream flow BA. This will allow the corresponding SD to maintain bidirectional flow state information.

[0123] Next, assume that downstream router 1530 receives a packet from flow D to C (e.g., from IP source address D to IP destination address C), referred to as flow DC. The information from the packet(s) header(s) is extracted, the information is exchanged, and the information is provided to hash function 1532 configured with the consistency hash algorithm parameter(s) 1599. Since the extracted header information from flow DC after the exchange will be the same as the header information from flow CD, as a result of the hash, downstream router 1530 installs forwarding information for flow DC with the primary next hop set to SD7, the first backup next hop to SD8, and the second backup next hop to SD9. These three SDs are enclosed in a solid ellipse. As this example illustrates, the same SDs have the same responsibility for upstream to downstream flow CD and downstream to upstream flow DC. This will allow the corresponding SDs to maintain bidirectional flow state information.

[0124] Finally, assume that downstream router 1530 receives a packet from flow D to A (e.g., from IP source address D to IP destination address A), referred to as flow DA. The information from the packet(s) header(s) is extracted, the information is exchanged, and the information is provided to hash function 1532 configured with the consistency hash algorithm parameter(s) 1599. Since the extracted header information from flow DA after the exchange will be the same as the header information from flow AD, as a result of the hash, downstream router 1530 installs forwarding information for flow DA with the primary next hop set to SD5, the first backup next hop to SD6, and the second backup next hop to SD7. These three SDs are enclosed in a dashed ellipse. As this example illustrates, the same SDs have the same responsibility for upstream to downstream flow AD and downstream to upstream flow DA. This will allow the corresponding SDs to maintain bidirectional flow state information.

[0125] §4.7 Conclusion

[0126] With example embodiments according to this specification, a large population of security devices can efficiently collaborate and align with a neighbor router in determining a subset of all security devices that will receive packets for any given flow from the neighbor router using a shared consistency hash algorithm. In view of this determination, the state of the flow is synchronized among the security devices within this subset (and preferably only among the security devices within this subset) rather than among all of the security devices. Thus, such example embodiments address one or more of the unmet needs discussed in the background. Such example embodiments can be used to provide an efficient, practical, and scalable approach to support high resiliency, availability, and performance of network security services with a large population of security devices collaborating with a neighbor router.

Claims

1. A security system comprising: a) at least one upstream router; b) at least one downstream router; and c) a first plurality of security devices disposed between the at least one upstream router and the at least one downstream router, each security device of the first plurality of security devices providing bidirectional security services for a plurality of bidirectional flows between the at least one upstream router and the at least one downstream router, wherein the first plurality of security devices is N security devices, wherein a consistent hashing ring algorithm for generating a consistent hashing ring is used to assign each bidirectional flow of the plurality of bidirectional flows to a primary security device of the first plurality of security devices and a set of one or more security devices of the first plurality of security devices acting as a plurality of backup security devices, wherein M+1 is less than N, wherein the set of one or more security devices of the first plurality of security devices is M backup security devices, and wherein for each bidirectional flow of the plurality of bidirectional flows, the primary security device and the set of M backup security devices of the first plurality of security devices synchronize bidirectional flow state information with each other but not with all of the N security devices.

2. The system of claim 1, wherein for each bidirectional flow of the plurality of bidirectional flows, the primary security device and the set of M backup security devices of the first plurality of security devices synchronize only the bidirectional flow state information with each other but not with any other security device of the N security devices.

3. The system of claim 1, wherein for each bidirectional flow of the plurality of bidirectional flows, the M backup security devices are the M next unique security devices after the primary security device on the consistent hashing ring.

4. The system of claim 3, wherein for each bidirectional flow of the plurality of bidirectional flows, the M backup security devices are the M next unique security devices after the primary security device on the consistent hashing ring in a clockwise direction.

5. The system of claim 1, wherein upon failure of a primary security device assigned to a given bidirectional flow, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to a next available backup security device of the M backup security devices, the next available backup security device being assigned to the given bidirectional flow on the consistent hashing ring.

6. The system of claim 5, wherein upon recovery of the primary security device assigned to the given bidirectional flow, in response to the recovery, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to the primary security device assigned to the given bidirectional flow.

7. The system of claim 1, wherein M is greater than 1, and ​ wherein upon failure of both the primary security device assigned to a given bidirectional flow and the next backup security device among the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to the next available backup security device among the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring.

8. A security system for use between at least one upstream router and at least one downstream router, the security system comprising: a first plurality of security devices arranged between the at least one upstream router and the at least one downstream router, each security device of the first plurality of security devices configured to provide bidirectional security services for a plurality of bidirectional flows between the at least one upstream router and the at least one downstream router, wherein the first plurality of security devices is N security devices, wherein for each bidirectional flow of the plurality of bidirectional flows, one security device of the first plurality of security devices is assigned to the bidirectional flow as a primary security device using a consistent hashing ring algorithm for generating a consistent hashing ring, and a set of one or more security devices of the first plurality of security devices is assigned to the bidirectional flow as a plurality of backup security devices using the consistent hashing ring algorithm, wherein M+1 is less than N, wherein the set of one or more security devices of the first plurality of security devices is M backup security devices, each security device of the first plurality of security devices is configured with a state synchronization procedure, wherein for each bidirectional flow of the plurality of bidirectional flows, the primary security device assigned to the bidirectional flow and the set of M backup security devices assigned to the bidirectional flow synchronize bidirectional flow state information with each other, but do not synchronize bidirectional flow state information with all of the N security devices.

9. The security system of claim 8, wherein for each flow of the plurality of bidirectional flows, the primary security device assigned to the bidirectional flow and the set of M backup security devices assigned to the bidirectional flow synchronize only the bidirectional flow state information with each other, but do not synchronize the bidirectional flow state information with any other security device of the N security devices.

10. The security system of claim 8, wherein for each bidirectional flow of the plurality of bidirectional flows, the M backup security devices assigned to the bidirectional flow are M next unique security devices on the consistent hashing ring after the primary security device assigned to the bidirectional flow.

11. The security system of claim 10, wherein for each bidirectional flow of the plurality of bidirectional flows, the M backup security devices assigned to the bidirectional flow are M next unique security devices on the consistent hashing ring in a clockwise direction after the primary security device assigned to the bidirectional flow.

12. The security system of claim 8, wherein upon failure of a primary security device assigned to a given bidirectional flow, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to a next available backup security device of the M backup security devices, the next available backup security device assigned to the given bidirectional flow on the consistent hashing ring.

13. The security system of claim 12, wherein upon recovery of the primary security device assigned to the given bidirectional flow, in response to the recovery, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to the primary security device assigned to the given bidirectional flow.

14. The security system of claim 8, wherein M is greater than 1, and wherein upon failure of both the primary security device assigned to a given bidirectional flow and a next backup security device of the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring, the at least one upstream router and the at least one downstream router forward any packets belonging to the given bidirectional flow to a next available backup security device of the M backup security devices assigned to the given bidirectional flow on the consistent hashing ring.

15. A computer-implemented method for a system having at least one upstream router, at least one downstream router, and a first plurality of security devices arranged between the at least one upstream router and the at least one downstream router, wherein the first plurality of security devices is N security devices, each security device of the first plurality of security devices is configured to provide bidirectional security services for packets received from the at least one upstream router and packets received from the at least one downstream router, the computer-implemented method comprising: a) using a consistent hashing ring algorithm for generating a consistent hashing ring, for each bidirectional flow of a plurality of bidirectional flows, assigning one security device of the first plurality of security devices as a primary security device to the bidirectional flow, and assigning a set of one or more security devices of the first plurality of security devices as a plurality of backup security devices to the bidirectional flow, wherein the set of one or more security devices of the first plurality of security devices is M backup security devices, wherein M+1 is less than N; b) receiving a packet belonging to a given bidirectional flow from either one of the at least one upstream router or the at least one downstream router; c) in response to receiving the packet, 1) creating or updating a state of the given bidirectional flow to which the received packet belongs, and 2) storing the state of the given bidirectional flow; and furthermore d) synchronizing the stored state of the given bidirectional flow between the primary security device assigned to the given bidirectional flow and the set of M backup security devices assigned to the given bidirectional flow, but not synchronizing the stored state with all of the N security devices.

16. The computer-implemented method of claim 15, wherein the act of synchronizing the stored state of the given bidirectional flow between the primary security device assigned to the given bidirectional flow and the set of M backup security devices assigned to the given bidirectional flow, but not synchronizing the stored state with all of the N security devices is performed only between the primary security device assigned to the given bidirectional flow and the set of M backup security devices assigned to the given bidirectional flow.

17. The computer-implemented method of claim 15, wherein the M backup security devices assigned to the given bidirectional flow are the M next unique security devices on the consistent hashing ring after the primary security device assigned to the bidirectional flow.

18. The computer-implemented method of claim 17, wherein the M backup security devices assigned to the given bidirectional flow are the M next unique security devices on the consistent hashing ring in a clockwise direction after the primary security device assigned to the bidirectional flow.

19. The computer-implemented method of claim 15, further comprising: e) using the consistent hashing ring algorithm to find both a first set of M different security devices before the one security device on the consistent hashing ring and a second set of M different security devices after the one security device on the consistent hashing ring; f) in response to one of the N security devices entering an available state, 1) synchronizing state information of any bidirectional flow of the plurality of bidirectional flows to which each of the first set of M security devices is assigned as a primary security device with the one security device, and 2) synchronizing state information of any bidirectional flow of the plurality of bidirectional flows to which the one security device is assigned as a primary security device with the most recent security device of the second set of M security devices that is in an UP state.

20. The computer-implemented method of claim 15, further comprising: e) receiving, by one of the at least one upstream router or one of the at least one downstream router, a new packet; f) determining a bidirectional flow to which the new packet belongs; g) determining one of the N security devices to which the determined bidirectional flow is assigned as a primary security device; h) determining whether the primary security device is available; i) in response to determining that the primary security device is not available, 1) determining a next available security device of the N security devices on the consistent hashing ring, and 2) assigning the next available security device as a primary security device to the bidirectional flow. 2) forwarding the received new packet to the determined next available one of the N secure devices on the consistent hashing ring.

Images