Traffic distribution device, traffic distribution system, traffic distribution method, and traffic distribution program

Abstract

A distribution device (10) is used in a traffic distribution system (100) in which the distribution device (10) distributes traffic to each node. The distribution device (10) comprises: a processing unit for calculating a hash function (11) from a key of traffic to be distributed and determining, on the basis of the result of the calculation, a node that is a distribution destination of the traffic; and a storage unit for storing a hash function (11) that is used for determining a node at which to place a state that is to be used by the traffic before the processing unit distributes the traffic.

Description

Traffic sorting device, traffic sorting system, traffic sorting method, and traffic sorting program

[0001] The present invention relates to a traffic distribution device, a traffic distribution system, a traffic distribution method, and a traffic distribution program.

[0002] In recent years, network interface cards (NICs) equipped with programmable hardware accelerators have become widespread (see Non-Patent Documents 1 and 2). Network function virtualization (NFV) processing is offloaded from the CPU to the NIC, enabling efficient virtualization of network device functions. While existing hardware accelerators offer extremely high processing performance, they lack sufficient memory capacity to hold the information (hereinafter referred to as "state") required for traffic processing. The memory capacity refers to the storage capacity of, for example, block random access memory (BRAM) or static random access memory (SRAM).

[0003] Therefore, when it is necessary to maintain a large amount of state, it is necessary to increase the number of units (scale out). In a configuration where hardware accelerators are added by scaling out, a distribution device is provided in front of the hardware accelerator to distribute traffic to the appropriate hardware accelerator. The traffic distribution process performed by the distribution device is classified into rule-based distribution process and hash-based distribution process.

[0004] 13 is an explanatory diagram illustrating rule-based distribution processing. A traffic distribution system 100x includes a client 30x that requests traffic processing, nodes 21x to 29x that process traffic using a built-in hardware accelerator, and a distribution device 10x that distributes the traffic to one of the nodes 21x to 29x. Each of the nodes 21x to 29x stores states ST1 to ST10000.

[0005] The distribution device 10x refers to a key table 11x to identify destination nodes 21x to 29x that are pre-assigned to each key calculated from traffic. A key is, for example, a value included in a packet such as an outer IP address of a VPN (Virtual Private Network) or a 5-tuple. VPNs include, for example, Generic Routing Encapsulation (GRE), Security Architecture for Internet Protocol (IPsec), and Virtual eXtensible Local Area Network (VXLAN).

[0006] In the key table 11x, correspondence information between keys and each of the nodes 21x to 29x is registered in advance so that the states held by each of the nodes 21x to 29x can be used by the traffic to be distributed. By adding (scaling out) the nodes 21x to 29x, the number of states that can be held by the entire traffic distribution system 100x increases. This allows each of the nodes 21x to 29x to prepare in advance the states to be used by the traffic to be distributed by the key table 11x, enabling stateful traffic processing.

[0007] Fig. 14 is an explanatory diagram showing hash-based distribution processing. Similar to Fig. 13, a traffic distribution system 100y includes a client 30y that requests traffic processing, nodes 21y to 29y that process traffic using a built-in hardware accelerator, and a distribution device 10y that distributes the traffic to one of the nodes 21y to 29y. Each of the nodes 21y to 29y stores states ST1 to ST10000.

[0008] 13, the process in which the distribution device 10x identifies the distribution destination by referring to the key table 11x is replaced with a process in which the distribution device 10y inputs a key into the hash function 11y and identifies the distribution destination based on the output result (hash value) in FIG. 14. This type of hash-based traffic distribution process using the hash function 11y is used in ECMP (Equal Cost Multi Path) and the like, as described in Non-Patent Documents 3 and 4. This eliminates the need for the distribution device 10y to manage individual states using the hash function 11y, eliminating the need to develop the distribution device 10y as a dedicated device and allowing it to be implemented in commercially available programmable hardware accelerators.

[0009] Intel, "Intel(R) FPGA SmartNIC N6000-PL Platform," [online], [searched August 7, 2023], Internet <URL: https: / / www.intel.co.jp / content / www / jp / ja / products / details / fpga / platforms / smartnic / n6000-pl-platform.html> ("Intel" is a registered trademark). Intel, "Intel(R) Infrastructure Processing Unit (Intel(R) IPU)," [online], [searched August 7, 2023], Internet <URL: https: / / www.intel.co.jp / content / www / jp / ja / products / details / network-io / ipu.html>. Yurufuwa Network, "NSX-T BGP ECMP", [online], [Retrieved August 7, 2023], Internet <URL: https: / / y-network.jp / 2020 / 08 / 31 / nsx-t-036 / >, 2020.08.31 C. Hopps, NextHop Technologies, "Analysis of an Equal-Cost Multi-Path Algorithm", [online], [Retrieved August 7, 2023], Internet <URL: https: / / www.rfc-editor.org / rfc / rfc2992>, RFC2992, November 2000

[0010] There are two needs for traffic distribution systems: (Requirement 1) It is required to guarantee that the state used by the traffic exists at the node to which the traffic is distributed, and to execute stateful traffic processing (mainly NFV processing). (Requirement 2) It is required to simplify the state management of the traffic distribution device and to realize the distribution device at low cost.

[0011] Regarding (Request 1), the hash-based distribution process in Fig. 14 does not achieve this because the process of allocating each state to each destination node is unrelated to the hash function 11y. Generally, hash-based distribution processes are used on the assumption that all subsequent nodes have the same state or are used for stateless processing.

[0012] 13, the rule-based distribution process does not achieve (Request 2) because the key table 11x makes state management of the traffic distribution device complex and large-capacity for each key. In other words, rule-based distribution process requires a dedicated device specialized for distribution that is more powerful than the node's hardware accelerator, which is expensive and takes time to respond to updates due to specification changes, resulting in high costs.

[0013] Therefore, a main object of the present invention is to provide a distribution device that distributes stateful traffic processing at low cost.

[0014] In order to solve the above problems, the traffic distribution device of the present invention has the following features: The present invention is a traffic distribution device used in a traffic distribution system that distributes traffic from the traffic distribution device to each node, characterized by comprising: a processing unit that calculates a hash function from a key of the traffic to be distributed and determines a node to which the traffic should be distributed based on the calculation result, and a storage unit that stores the hash function used to determine a node to which a state used by the traffic should be allocated before the processing unit distributes the traffic.

[0015] According to the present invention, a distribution device that distributes stateful traffic processing can be provided at low cost.

[0016] 1 is a configuration diagram of a traffic distribution system according to the present embodiment. FIG. 2 is an explanatory diagram showing a first stage process associated with scaling in the number of nodes according to the present embodiment. FIG. 3 is an explanatory diagram showing a second stage process associated with scaling in the number of nodes according to the present embodiment. FIG. 4 is an explanatory diagram showing a third stage process associated with scaling in the number of nodes according to the present embodiment. FIG. 5 is an explanatory diagram showing scale-in in pattern 1 according to the present embodiment. FIG. 6 is a configuration diagram showing the contents of a key table for pattern 1 according to the present embodiment. FIG. 7 is a flowchart showing state reallocation processing in pattern 1 according to the present embodiment. FIG. 8 is a configuration diagram showing the rule table of FIG. 8 according to the present embodiment. FIG. 9 is a configuration diagram showing the contents of an allocation table for pattern 2 according to the present embodiment. FIG. 10 is a flowchart showing state reallocation processing in pattern 2 according to the present embodiment. FIG. 11 is a hardware configuration diagram of each device constituting the traffic distribution system according to the present embodiment. FIG. 12 is an explanatory diagram showing rule-based distribution processing. FIG. 13 is an explanatory diagram showing hash-based distribution processing.

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0018] FIG. 1 is a configuration diagram of a traffic distribution system 100. The traffic distribution system 100 includes a client 30 that requests processing of traffic (U-plane signals), nodes (nodes 21 to 29) that process traffic using a built-in hardware accelerator, and a distribution device (traffic distribution device) 10 that distributes traffic to one of the nodes. Each node stores states (states ST1 to ST10000). Each state is, for example, the outer-IP address of a VPN and routing information. VPNs come in various formats, such as GRE, IPsec, and VXLAN.

[0019] Furthermore, the traffic sorting system 100 has an additional controller 40 compared to the traffic sorting system 100x in Fig. 13 and the traffic sorting system 100y in Fig. 14. While Fig. 1 shows an example in which the controller 40 and the sorting device 10 are separate devices, the controller 40 and the sorting device 10 may be housed in the same housing. The controller 40 distributes a hash function 11 to the sorting device 10 (dashed arrow from the controller 40 to the sorting device 10), so that the controller 40 and the sorting device 10 have the same hash function 11.

[0020] The controller 40 then registers each state in each node where each state is to be registered (recorded), which is calculated by inputting the key of each state into the hash function 11. The distribution device 10 then distributes the traffic to each node using the hash function 11. In other words, because the distribution device 10 and the controller 40 use the same hash function 11, the traffic distributed to each node can use the state registered in advance (stateful processing can be executed).

[0021] 1 distributes traffic from the distribution device 10 to each node. The distribution device 10 has the following components: A processing unit 13 that calculates a hash function 11 from the key of the traffic to be distributed and determines the node to which the traffic should be distributed based on the calculation result; and A storage unit 14 that stores the hash function 11 (the same as the hash function 11 used by the processing unit 13 in the calculation to determine the node to which the state used by the traffic should be allocated) that is used to determine the node to which the state used by the traffic should be allocated before the processing unit 13 distributes the traffic.

[0022] Here, there may be cases where the number of nodes needs to be reduced (scaled in) due to a decrease in traffic volume, etc. Below, an example of scaling in the number of nodes from three to two will be described with reference to Figures 2 to 4. Figure 2 is an explanatory diagram showing the first stage of processing associated with scaling in the number of nodes. First, the controller 40 selects which of the three currently operating nodes 21-23 to remove. This node selection process uses, for example, the following methods: (Method 1) Eliminate the node with the fewest registered states. This minimizes state transitions. (Method 2) Eliminate the node with the highest transfer rate (bps) over the most recent fixed period (e.g., one minute). By redistributing the states of nodes with high loads, load leveling is expected. In this example, the controller 40 selects node 23 as the node to be removed.

[0023] 2, the distribution device 10 arranges the states as follows based on the hash function 11A (the function before the node 23 is removed and before the states are reallocated): States ST1 and ST2 in node 21; States ST3 and ST4 in node 22; States ST5 and ST6 in node 23.

[0024] FIG. 3 is an explanatory diagram showing the second-stage processing accompanying the scaling-in of the number of nodes. The controller 40 calculates the reallocation of each state from the state shown in FIG. 2 based on the hash function 11B (the function after the removal of node 23 and the reallocation of states). As a result, the controller 40 instructs each node to transition each state shown in FIG. 3 as follows (indicated by the dashed arrows in FIG. 3): - Copy states ST5 and ST6 in node 23 to node 21. - Copy state ST2 in node 21 to node 22. In this second stage, the original (existing) state is not deleted, so the same state is stored in multiple nodes. The hash function 11A (the function before reallocation) of the distribution device 10 distributes traffic to the node where the original state is stored, allowing the distributed traffic to execute stateful processing without interruption.

[0025] The second-stage state transition period is a period during which the same state (hereinafter referred to as "duplicate state") exists in duplicate in different nodes at the same time. For example, state ST2 exists in both the copy-source node 21 and the copy-destination node 22, and therefore corresponds to a duplicate state. Here, it is desirable for the controller 40 to perform the following processing to prevent the contents of the duplicate states from becoming inconsistent during the overlapping period: - During the state transition period, changes to the duplicate state are prohibited. - If a change to a duplicate state is necessary during the state transition period, the controller 40 knows the location of the duplicate state, so it changes each duplicate state at once. - For duplicate states not managed by the controller 40, there is no problem even if inconsistencies occur due to content changes. This is because the controller 40 is an exception and does not manage information about adjacent nodes resolved by ARP (Address Resolution Protocol).

[0026] 4 is an explanatory diagram showing the third stage processing accompanying the scale-in of the number of nodes. The controller 40 switches the hash function 11A of the distribution device 10 to the hash function 11B. As a result, the hash function 11B of the distribution device 10 distributes traffic to the node where the copy destination state of FIG. 3 is saved, so that the distributed traffic can be processed statefully.

[0027] The controller 40 then completes the scale-in by deleting unnecessary information before the reallocation as follows: States ST5 and ST6 are deleted from node 23, which is the copy source. As a result, node 23 no longer has any states to manage itself, so it is also deleted (reduced) from the processing targets (destination nodes) of the traffic allocation system 100. State ST2 is deleted from node 21, which is the copy source.

[0028] The above has described an overview of the processing associated with scaling in the number of nodes with reference to Figures 2 to 4. As such, the controller 40 of the traffic distribution system 100 has the following features. When the number of nodes to which traffic is distributed is changed, the controller 40 calculates the destination node for each traffic using a first hash function, which is the hash function 11 before the number of nodes is changed, and a second hash function, which is the hash function 11 after the number of nodes is changed. For traffic whose destination node is changed due to a change in the number of nodes, the controller 40 copies the state used by the traffic before the change to the destination node after the change, and then updates the first hash function used by the distribution device 10 for distribution to the second hash function.

[0029] Below, we will explain the following two patterns as specific examples of how the controller 40 described in Figures 2 to 4 calculates the reallocation of each state. Pattern 1 (Figures 5 to 7): A pattern in which the hash function used in ECMP is used alone. Pattern 2 (Figures 8 to 11): A pattern in which a hash function and a rule table are used together, as proposed in the following literature: Olteanu, Vladimir, et al., "Stateless datacenter load-balancing with beamer," [online], [Retrieved August 10, 2023], Internet <URL: https: / / www.usenix.org / system / files / conference / nsdi18 / nsdi18-olteanu.pdf>, 15th USENIX Symposium on Networked Systems Design and Implementation, USENIX Association, 2018, pp. 125-139.

[0030] 5 is an explanatory diagram showing scale-in in pattern 1. In a state 101 before reallocation, the distribution device 10 distributes traffic T11 for which the calculation result is 0 to node #0, traffic T12 for which the calculation result is 1 to node #1, and traffic T13 for which the calculation result is 2 to node #2 based on the calculation result of the hash function 11C of "hash(key) % 3." Note that hash(key) is the output value of hash() with the key embedded in the traffic packet as input, and "% 3" is the remainder when the output value is divided by 3, indicating that three nodes exist as distribution destinations.

[0031] In the state 102 after reallocation, the distribution device 10 distributes traffic T12, for which the calculation result is 0, to node #0, and traffic T11 and T13, for which the calculation result is 1, to node #1, based on the calculation result of the hash function 11D of "hash(key) % 2." Because the number of destination nodes has been reduced to two, the latter half of the hash function 11D becomes "% 2." In other words, in pattern 1, the distribution device 10 uses the hash function 11 of "hash(key) % N" (N = number of nodes).

[0032] FIG. 6 is a configuration diagram showing the contents of the key tables 111 and 112 for pattern 1. The controller 40 creates a key table 111 showing correspondence information between input values ​​(keys) and output values ​​(destination nodes) as the calculation results of the hash function 11C in FIG. 5, and a key table 112 showing the calculation results of the hash function 11D in FIG. 5. The controller 40 compares the key tables 111 and 112 and sets the transition flag in the key table 112 to ON for records with the same input value (key) but different output values. For example, the second row of the key table 112, "key-2," the node "node #1," and the transition flag "ON" are associated. This association indicates that the state used by traffic with key=2 needs to be migrated to node #1. On the other hand, records without a transition flag "ON," such as key-1, indicate that the state used by that key-1 does not need to be migrated due to scale-in.

[0033] FIG. 7 is a flowchart showing the state reassignment process of pattern 1. The controller 40 calculates a hash function using the number of nodes N before and after the migration (S101). This creates the key tables 111 and 112 shown in FIG. 6. The controller 40 executes a loop that increments the variable n=1, 2, ... of key-n (nth key) (S111 to S116). The controller 40 references the calculation result of the nth key in the key tables 111 and 112 and determines whether there is a difference before and after the scale-in (S112). If the answer is Yes in S112, the controller sets the state migration flag in the key tables 111 and 112 (S113). If the answer is No in S112, the controller proceeds to S114.

[0034] The controller 40 determines whether or not a key that has not yet been compared in S112 exists in the key tables 111, 112 (S114). If the result in S114 is Yes, the controller 40 adds 1 to n (n=n+1) to calculate the next key (S115) and returns the process to S112. After completing the loop up to S116, if there is a node that can transition a state with a transition flag "ON" (Yes in S121), the controller 40 transitions that state (S122). Note that the controller 40 executes the determination in S121 by calculating the number of states that each node can accept transitions from the number of states that each node currently accommodates and the number of states that each node can accommodate.

[0035] FIG. 8 is an explanatory diagram showing scale-in in pattern 2. In a state 201 before reallocation, the distribution device 10 identifies a destination node corresponding to the calculation result (b) of the hash function 11E of "b = hash(key) % B" by referring to the rule table 12A before reallocation based on the calculation result (b). As a result, the distribution device 10 distributes traffic T11 to node #0, traffic T12 to node #1, and traffic T13 to node #2. Note that "B" of the hash function 11E is set in advance to a value (e.g., B = 10 times N) that is sufficiently larger than the maximum number of downstream servers (N) expected during scale-out.

[0036] In the post-reallocation state 202, the distribution device 10 identifies the destination node corresponding to the calculation result (b) by referring to the post-reallocation rule table 12B based on the calculation result (b) of the hash function 11E, which is the same as in the pre-reallocation state 201. As a result, the distribution device 10 distributes the traffic T12 to node #0 and the traffics T11 and T13 to node #1.

[0037] FIG. 9 is a configuration diagram showing the rule tables 12A and 12B in FIG. 8. The rule table 12A before scale-in is a table for allocating b=[0,B-1] to three nodes #0 to #2. The rule table 12A allocates the range of b=[0,9] to node #0, the range of b=[10,19] to node #1, and the range of b=[20,29] to node #2. The rule table 12B after scale-in is a table for allocating b=[0,B-1] to two nodes #0 to #1. There is no change in the allocation destination for the range of b=[0,19] between the rule tables 12A and 12B, but in the rule table 12B, only the allocation destination for the range of b=[20,29] (by group) has been changed from node #2 to node #0. As a result, there is no change in the allocation destination in the range of b=[0,19], so the frequency of state transitions due to scale-in can be reduced.

[0038] The controller 40 prepares a hash function 11E (FIG. 8) that is the same before and after scale-in, and rule tables 12A and 12B (FIG. 9) that are different before and after scale-in, and sets these in the distribution device 10. Here, the rule tables 12A and 12B are tables in which each group of hash values ​​calculated from the flow key (for example, a group where b=0 to 9) is associated with an allocation destination server. Therefore, the capacity (number of records) of the rule tables 12A and 12B can be made smaller than the key table 11x in FIG. 13 in which each flow key is associated with an allocation destination server.

[0039] FIG. 10 is a configuration diagram showing the contents of allocation tables 211 and 212 for pattern 2. In the key tables 111 and 112 of FIG. 6 used in pattern 1, key-n is directly associated with the allocation destination node. On the other hand, in the allocation tables 211 and 212 of FIG. 10 used in pattern 2, the allocation destination node and the state to be used are associated with the calculation result (b) of the hash function 11E instead of key-n. For each calculation result (b=1, 2, . . .), the controller 40 creates the allocation table 211 before scale-in by referencing the rule table 12A, and creates the allocation table 212 after scale-in by referencing the rule table 12B. Then, the controller 40 sets the migration flag in the allocation table 212 to ON for records in the allocation tables 211 and 212 that have the same calculation result (e.g., b=28) but different allocation destination nodes.

[0040] 11 is a flowchart showing the state reassignment process in pattern 2. First, the controller 40 creates allocation tables 211 and 212 as shown in FIG. 10, and identifies states whose transition flags are ON. Then, the controller 40 optimizes the placement of states whose transition flags are ON (S201). In the calculation of S201, the controller 40 calculates an arbitrary method for optimizing the placement of states using the following conditions as constraints: - The number of states that can be registered in each node is not exceeded - The difference in the number of states between each node is reduced - The number of state transitions is reduced

[0041] For example, the controller 40 may perform the following mathematical programming calculations as an optimization method. A constraint condition is set to be satisfied: "The number of states that can be registered in each node must not be exceeded." A linear programming problem is defined with at least one of the following objective functions: "reducing the difference in the number of states between each node" or "reducing the number of state transitions." The linear programming problem is solved using the simplex method.

[0042] Next, the controller 40 updates the rule table 12A (for the number of nodes = 3) of the sorting device 10 to the rule table 12B (for the number of nodes = 2) (S202), and transitions the state based on the calculation result of S201 (S203, similar to S122 in FIG. 7). Note that the optimization method of S201 takes into consideration the constraint condition that "the number of states that can be registered in each node will not be exceeded" in the state placement, and therefore, unlike pattern 1, it is guaranteed that there is a node to which the state can be transitioned (Yes in S121 in FIG. 7).

[0043] 12 is a hardware configuration diagram of each device constituting the traffic distribution system 100. Each device (distribution device 10, each node 21-29, client 30, and controller 40) of the traffic distribution system 100 is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each unit by executing a program (traffic distribution program) loaded into the RAM 902. This program (also called an application, or simply "app") can be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM and distributed.

[0044] [Effect] The present invention is a distribution device 10 used in a traffic distribution system 100 that distributes traffic from the distribution device 10 to each node, characterized by having a processing unit 13 that calculates a hash function 11 from the key of the traffic to be distributed and determines the node to which the traffic should be distributed based on the calculation result, and a memory unit 14 that stores the hash function 11 used to determine the node to which the state used by the traffic should be placed before the processing unit 13 distributes the traffic.

[0045] This allows the distribution device 10 to distribute traffic to nodes that are state allocation destinations, allowing the nodes to perform stateful traffic processing. Furthermore, the distribution device 10 does not need to have a complex mechanism such as a correspondence table that determines the traffic distribution destination node for each traffic key, so it can be realized at low cost.

[0046] The present invention is a traffic distribution system 100 having the distribution device 10 and a controller 40, wherein the controller 40, when the number of nodes to which traffic is distributed is changed, calculates the node to which each traffic is distributed using a first hash function 11 which is the hash function 11 before the number of nodes is changed and a second hash function 11 which is the hash function 11 after the number of nodes is changed, and for traffic whose distribution node is changed due to the change in the number of nodes, copies the state used by the traffic before the change to the node to which the traffic is to be distributed after the change, and then updates the hash function 11 used by the distribution device 10 for distribution.

[0047] As a result, the distribution device 10 allows the nodes to which the hash function 11 assigned before the number of nodes was changed to use the source state, and allows the nodes to which the hash function 11 assigned after the number of nodes was changed to use the destination state. Therefore, even if the hash function 11 is switched, there is no time period during which the state cannot be used for traffic, and stable stateful traffic processing can be performed on the nodes.

[0048] 10 Distribution device (traffic distribution device) 11 Hash function 12A, 12B Rule table 13 Processing unit 14 Storage unit 21-29 Node 30 Client 40 Controller 100 Traffic distribution system 111, 112 Key table 211, 212 Allocation table

Claims

1. A traffic distribution device used in a traffic distribution system that distributes traffic from a traffic distribution device to each node, the traffic distribution device comprising: a processing unit that calculates a hash function from the key of the traffic to be distributed and determines the node to which the traffic should be distributed based on the calculation result; and a memory unit that stores the hash function used to determine the node to which the state used by the traffic should be placed before the processing unit distributes the traffic.

2. A traffic distribution system having the traffic distribution device of claim 1 and a controller, wherein the controller, when the number of nodes to which traffic is distributed is changed, calculates a destination node for each traffic using a first hash function which is the hash function before the number of nodes is changed and a second hash function which is the hash function after the number of nodes is changed, and for traffic whose destination node is changed due to the change in the number of nodes, copies the state used by the traffic before the change to the destination node after the change, and then updates the first hash function used by the traffic distribution device for distribution to the second hash function.

3. A traffic distribution method executed by a traffic distribution device used in a traffic distribution system that distributes traffic from the traffic distribution device to each node, wherein the traffic distribution device calculates a hash function from a key of the traffic to be distributed, and determines a node to which the traffic is to be distributed based on the calculation result, and the hash function is used to determine a node to which a state used by the traffic is to be placed before distributing the traffic.

4. A traffic distribution program for causing a computer to function as the traffic distribution device according to claim 1.

Images