Maintaining quality of service processing for packets using security parameter index values

Abstract

Techniques are described herein for load balancing encrypted traffic based on a security parameter index (SPI) value of a packet header and a set of five-tuple values of the packet header. In addition, techniques are described herein for including quality of service (QoS) type information in a SPI value field of a packet header. The QoS type information can indicate a particular traffic class that processing of the packet is to be subject to. In addition, techniques are described herein for preconfiguring a backend host such that encrypted traffic can be migrated from another backend host to the backend host without causing a temporary service interruption.

Description

Using security parameter index values ​​to maintain the quality of service processing of groups

[0001] Related applications

[0002] This application claims priority to U.S. Utility Model Patent Application No. 17 / 171,604, filed February 9, 2021, and U.S. Provisional Patent Application No. 63 / 124,317, filed December 11, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to improved techniques for load balancing encrypted traffic using Security Parameter Index (SPI) values ​​in packet headers. Background Technology

[0004] Building cloud-delivered Software-as-a-Service (SaaS) products involves creating a distributed system that is delivered to users in the cloud. Typically, traffic is sent to these services according to one or more routing policies, such as Equal-Cost Multi-Path (ECMP) routing. ECMP and other routing policies allow for fixed flows based on “5-tuples” to send packets to specific backend instances. A packet’s 5-tuple typically refers to a set of five distinct values ​​that contain the Transmission Control Protocol / Internet Protocol (TCP / IP) connection. The set of 5-tuple values ​​includes the source IP address, source port number, destination IP address, destination port number, and the specific protocol being used.

[0005] However, because ECMP and other routing policies use 5-tuples, they do not take into account the individual flows of encrypted connections (such as Internet Protocol Security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulated Security Payload (ESP) traffic). Furthermore, encrypted connections (such as IPsec) are difficult to classify traffic for due to their encrypted nature. Once packets are encrypted and encapsulated, implementing any form of Quality of Service (QoS) becomes virtually impossible.

[0006] Furthermore, in network environments where the load balancer is placed before the pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted sessions assigned to that worker node must be migrated to one or more other hosts. This typically results in temporary service interruptions while the new host worker node(s) and the client negotiate new encrypted connections. Attached Figure Description

[0007] The following is a detailed description in conjunction with the accompanying drawings. In the drawings, the leftmost number(s) of the reference numerals identify the drawing in which that reference numeral first appears. The same reference numerals are used in different drawings to denote similar or identical items. The systems depicted in the drawings are not drawn to scale, and the components in the drawings may be depicted as not being drawn to scale with each other.

[0008] Figure 1 illustrates a schematic diagram of an example system architecture for a networked environment that includes a tunnel communication session, which comprises separate control plane and data plane traffic flows.

[0009] Figure 2 illustrates a sample traffic flow of a load balancer node sending traffic to downstream nodes according to one or more routing policies.

[0010] Figure 3 shows a data flow diagram of an example traffic flow between individual nodes and / or devices for establishing a load-balanced communication session using the SPI value of the packet header.

[0011] Figures 4A and 4B show data flow diagrams of example traffic flows between individual nodes and / or devices in a communication session that uses QoS type information in the SPI value used to indicate packet headers.

[0012] Figures 5A-5C together illustrate a schematic diagram of an example data flow associated with performing a cryptographic tunnel migration.

[0013] Figure 6 shows a logic flowchart of an example method for maintaining QoS processing of packets using SPI values.

[0014] Figure 7 shows a logic flowchart of an example method for load balancing traffic based on the SPI value of the packet header.

[0015] Figure 8 shows a logic flowchart of an example method for performing encrypted tunnel migration.

[0016] Figure 9 shows a logic flowchart of another example method for performing encrypted tunnel migration.

[0017] Figure 10 illustrates a schematic diagram of an example computer hardware architecture for implementing network nodes and / or devices (e.g., load balancers, control nodes, data nodes, etc.), which can be used to implement various aspects of the various technologies presented herein. Detailed Implementation Summary of the Invention

[0018] Overview

[0019] Various aspects of the invention are set forth in the independent claims, and preferred features are set forth in the dependent claims. A feature of one aspect may be applied to each aspect alone or in combination with other aspects.

[0020] This disclosure describes systems and methods that improve techniques related to load-balanced encrypted traffic by using a Security Parameter Index (SPI) value in the packet header. By way of example, and not limitation, methods according to the various techniques described in this disclosure may include receiving, from a client device and at a network device in the network, a request to establish an encrypted tunnel over the network so that data plane traffic can flow between the client device and a service via the encrypted tunnel. The method may also include generating an SPI value to be used by the client device for data plane traffic and sending an indication of the SPI value to the client device. Additionally, the method may include receiving a data packet including the SPI value at a load balancer and determining, at least in part based on the SPI value, which server in a set of servers supporting the service to send the data packet to. Accordingly, the load balancer may send the data packet to the server.

[0021] In some cases, the method may additionally or alternatively include determining that the data plane traffic belongs to a specific traffic category within a set of traffic categories. The specific traffic category may be associated with a specific Quality of Service (QoS) performance metric. Therefore, the method may include generating an SPI value to be used by client devices for the data plane traffic. The SPI value may correspond to a specific traffic category. In this way, the load balancer can receive data packets of data plane traffic including the SPI value, and based at least in part on the data packets including the SPI value, the load balancer can transmit data packets over the network such that the data packets are processed according to the specific QoS performance metric.

[0022] In an additional or alternative example, the method may include receiving, at the load balancer and from the client device, first data plane traffic having a first SPI value and a set of 5-tuple values. Based at least in part on the first SPI value and the set of 5-tuple values, the load balancer may send the first data plane traffic to a first node. The method may also include receiving, at the load balancer, an indication that at least a portion of the first data plane traffic is to be sent to a second node. Based at least in part on this indication, the load balancer may prompt the second node to provide one or more interfaces such that at least a portion of the first data plane traffic can be sent to the second node. In this manner, the load balancer may receive, from the client device, second data plane traffic having a second SPI value and the set of 5-tuple values. Based at least in part on the second SPI value and the set of 5-tuple values, the load balancer may determine that the second data plane traffic comprises at least that portion of the first data plane traffic, and in response, send the second data plane traffic to the second node.

[0023] Furthermore, the techniques described herein can be implemented as a method and / or by a system having a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, perform the techniques described herein.

[0024] Example Implementation

[0025] As mentioned above, traffic is typically sent to various services according to one or more routing policies, such as Equal Cost Multipath (ECMP) routing. However, because these routing policies use 5-tuples, they do not take into account the individual flows of encrypted connections (such as Internet Protocol Security (IPsec) connections, which include Internet Key Exchange (IKE) traffic and Encapsulated Security Payload (ESP) traffic). This means that the entropy of these flows may be much less than that achievable with the per-tunnel entropy provided by the Security Association (SA) for IPsec IKE and ESP flows. For example, IPsec IKE and ESP traffic contains an additional identifier called the Security Parameter Index (SPI). The SPI value is used to uniquely identify the established IPsec SA.

[0026] Therefore, one aspect of this disclosure provides techniques for leveraging SPI values ​​to allow load balancing and pinning per-IPsec IKE and ESP flows to a specific backend. By implementing these techniques, several advantages can be achieved, including the ability to terminate the same encrypted tunnel / SA on multiple systems, which allows for scalable capacity. Furthermore, higher performance can be achieved through finer-grained control over where control plane and data plane traffic sessions might land on backend nodes / servers. Additionally, different traffic types can be handled accordingly (e.g., performance levels, allowing customers to have their own backends).

[0027] Furthermore, encrypted connections (such as IPsec) are difficult to classify traffic for due to their encrypted nature. Once packets are encrypted and encapsulated, performing any form of Quality of Service (QoS) becomes virtually impossible. For example, when specifically looking at virtual private network (VPN) products built for cloud delivery of IPsec, you are dealing with ESP in IP packets and / or ESP in UDP packets. Therefore, another aspect of this disclosure includes techniques for encoding and mapping QoS type information into SPI values ​​to enable QoS to be performed on packets even after they have been encrypted and encapsulated. Performing these techniques can allow for the classification of encrypted traffic, thereby allowing for differentiated service (DiffServ-style) resource allocation on data nodes, enabling the allocation of separate SAs on data nodes more suited to that traffic category, and also allowing for the adjustment of traffic shaping rules within data nodes to adjust network throughput in the flows allocated to them.

[0028] Furthermore, in networking environments where the load balancer is placed before the pool of worker nodes responsible for handling encrypted traffic, when a worker node goes offline, the encrypted sessions assigned to that worker node must be migrated to one or more other hosts. This typically results in temporary service interruptions while the new host worker node(s) and the client negotiate new encrypted connections. Therefore, another aspect of this disclosure includes techniques for adding support to backend worker nodes (e.g., data nodes) to signal the load balancer that a worker node is about to be removed from the backend worker node queue. In this way, these techniques can mitigate the impact of planned or unplanned shutdowns by migrating encrypted tunnels out of a worker node when it becomes unhealthy or is set to be replaced by another node. Additionally, these techniques can reduce the impact of rebalancing load across server pools.

[0029] Therefore, improvements to computer-related technologies can be achieved based on the various techniques described in this disclosure. As previously mentioned, the entropy for an encrypted stream can be much smaller than that achievable when per-tunnel entropy is provided. For example, most routing policies (such as ECMP) use a set of five-tuple values ​​for hashing. However, utilizing the packet's SPI value allows for the use of six-tuple logic, thereby better allocating the flow to the headend node. Furthermore, even if the packet is encapsulated, the SPI value can be further used to indicate the QoS type information of the data packet, allowing for the allocation of a separate SA on the data node more suitable for that traffic category. These are just some examples of the many improvements that can be implemented based on the techniques described in this disclosure. These and other improvements will be readily understood and appreciated by those skilled in the art.

[0030] By way of example and not limitation, methods according to the various techniques described in this disclosure may include receiving packets from a client device indicating a request to establish an encrypted tunnel over the network so that data plane traffic can flow between the client device and the service via the encrypted tunnel. In some examples, the request packet may be received at the network by a load balancer or router, and the load balancer or router may forward the request to the network's control node. Additionally, the load balancer or router may forward the request packet to the control node based at least in part on an Equal Cost Multipath (ECMP) routing policy and / or a 5-tuple associated with the request packet. In some examples, the request to establish an encrypted tunnel may include a request to establish an IPsec connection and / or a request to establish an IPsec SA or subSA.

[0031] In some examples, the network can be configured such that it includes separate control nodes and data nodes. In other words, the network can be configured to split the processing of control plane traffic (e.g., IKE traffic) and data plane traffic (e.g., ESP traffic) across different nodes (e.g., control nodes or "IKE" nodes for handling the control plane and data nodes for handling the data plane). This allows the network to scale each node type individually and / or independently. Control nodes and data nodes may include headend servers associated with the service. In some examples, control nodes may operate on a first set of compute resources associated with the network, and data nodes may operate on a second, different set of compute resources associated with the network.

[0032] In some examples, the method may include determining that data plane traffic belongs to a specific traffic category within a set of traffic categories. A specific traffic category may be associated with a specific Quality of Service (QoS) performance metric. In some examples, to allow the control node to create an SPI value that matches the correct traffic category, a classifier may be invoked before the SA is established. This can be done in several different ways. For example, a load balancer or router may invoke the classifier and inject category information as a header before control plane traffic is forwarded to the control node. This can be done by using a field in the IP header that is not currently used (such as the DSCP field) or by creating a new field. Alternatively, the control node may invoke the classifier when it initiates the SA. Either way, category information can be provided to the control node, allowing it to create an SPI value corresponding to the traffic category.

[0033] In some examples, the method may include generating an SPI value to be used by the client device for data plane traffic. The SPI value may include a combination of bits identifying a specific SA. In some cases, multiple SPI values ​​may be generated, and each individual SPI value may identify a corresponding SA. Alternatively, the SPI value may be generated by the control node.

[0034] As described above, in various examples, the SPI value may include QoS type information (e.g., Differentiated Service (DiffServ) type information, Type of Service (ToS), Differentiated Service Code Point (DSCP) type information, and / or Experimental Bit (EXP) type information) indicating the specific traffic class upon which the packet is processed. Therefore, in some examples, generating the SPI value may include generating a first-bit combination representing the specific traffic class to be processed based on, generating a second-bit combination representing a specific SA, and masking or combining the first and second-bit combinations such that the first-bit combination includes a first portion of the SPI value and the second-bit combination includes a second portion of the SPI value. For example, the packet's SPI field is defined as an arbitrary 32-bit value, with the range 0-255 defined as reserved. This leaves values ​​from 256 (0x00000100) to 4294967295 (0xffffffff) as the SPI value. Therefore, in some examples, the first portion of the 32-bit field (e.g., the "front" portion) can be used for QoS mapping, and the first hexadecimal digit can be "reserved" for mapping by offsetting the SPI value by 4 bits. For example, using hexadecimal values ​​0x[0]3ec7b2a to 0x[f]3ec7b2a, hexadecimal numbers [0] to [f] can represent a QoS mapping, and 0x3ec7b2a can represent the actual 4-bit SPI offset consumed. That is, the first part of the SPI value (e.g., hexadecimal numbers [0] to [f]) ​​can represent a specific traffic class QoS mapping, and the second part of the SPI value (e.g., hexadecimal number 3ec7b2a) can identify a specific SA. This results in the use of 15 mapping values. Additionally or alternatively, the established 802.1q Class of Service (CoS) or Multiprotocol Label Switching (MPLS) EXP to DSCP bit mappings can be followed because they have similar bit sizes.

[0035] After generating the SPI, the method may include sending an indication of the SPI value to the client device. In some examples, the control node may perform a Direct Server Return (DSR) to send the indication to the client device. In some examples, the data packet including the SPI value may be received by a load balancer. The data packet may include data plane traffic packets. That is, the protocol associated with the data packet may correspond to a data plane traffic protocol, such as ESP. In some examples, the data packet may include a set of five-tuple values. For example, a set of five-tuple values ​​for a data packet may include a source IP address value, a source port value, a destination IP address value, a destination port value, and the protocol associated with the data packet. As described herein, a set of six-tuple values ​​can be used to refer to both the SPI value and a set of five-tuple values. That is, a set of six-tuple values ​​may include a source IP address value, a source port value, a destination IP address value, a destination port value, the protocol associated with the data packet, and the SPI value. However, different values ​​may be used.

[0036] In some cases, a load balancer may determine which server (e.g., a data node) to send data packets to, among a set of servers or nodes supporting the service. For example, the load balancer may determine to send data packets to a server based at least in part on the SPI value and / or the set of 5-tuple values. In some examples, the load balancer may receive data representing the association between the SPI value and the set of 5-tuple values ​​associated with a client device, and the determination to send data packets to the server may further be based at least in part on that data. That is, the load balancer may be updated with a mapping indicating the association between the SPI value and the 5-tuple values. In some examples, the determination to send data packets to the server may be based at least in part on calculating a hash value representing the SPI value and / or the set of 5-tuple values. The load balancer may use a hash function to calculate the hash. Additionally or alternatively, the load balancer may determine to send data packets to the server based at least in part on one or more routing policies (e.g., ECMP).

[0037] In some examples, the method may include sending data packets to a server. Alternatively, data packets may be sent over the network such that they are processed according to specific QoS performance metrics and / or traffic categories. For example, if the SPI value includes an indication of a specific traffic category and / or QoS performance metric to which data packets should be processed, the load balancer can send packets over the network according to that specific traffic category and / or QoS performance metric.

[0038] In additional or alternative examples, the method may include generating a second SPI value for use by the client device for data plane traffic. The second SPI value may identify a second SA. The second SPI value may be generated by the control node. In some examples, generating the second SPI value may be based at least in part on classifying request packets to determine the traffic category associated with the request packets, as described above. After generating the second SPI, the method may include sending an indication of the second SPI value to the client device. In some examples, the control node may perform a direct server return (DSR) to send the indication of the second SPI value to the client device.

[0039] In some examples, the method may include receiving a second data packet at the load balancer that includes a second SPI value. Additionally, the second data packet may include the set of 5-tuple values, a portion of the set of 5-tuple values, or a new set of 5-tuple values. In some cases, the set of 5-tuple values ​​may indicate that the second data packet was sent by a client device or a different client device. Based at least in part on the second data packet including the second SPI value and / or the set of 5-tuple values, the load balancer may send the second data packet to a second server (e.g., a second data node) in a server group. In some examples, a first portion of the second SPI value may correspond to a second traffic category associated with a second QoS performance metric. Thus, based at least in part on the second data packet including the second SPI value, the load balancer can send the second data packet over the network such that the second data packet is processed according to the second QoS performance metric.

[0040] As described above, one aspect of the technology described herein may also include adding support for backend worker nodes (e.g., data nodes) to signal to the load balancer that a worker node is about to be removed from the backend worker node queue. Therefore, in additional or alternative examples, the method may include receiving first data plane traffic at the load balancer from a client device having a first SPI value and a set of 5-tuple values. The first SPI value may identify a first security association (SA) between the client device and the first node.

[0041] In some examples, the method may include sending first data plane traffic to a first node in a set of nodes. In at least one example, the first node may include a first data node in a set of data nodes. The first node may be associated with a first encrypted tunnel (e.g., IPsec SA). In some examples, sending the first data plane traffic to the first node may be based at least in part on a first SPI value and that set of 5-tuple values. For example, the load balancer may compute a hash value (e.g., a six-tuple) representing the first SPI value and that set of 5-tuple values. Based at least in part on the hash value, the load balancer may send the first data plane traffic to the first node according to an ECMP routing policy.

[0042] In various examples, the method may include receiving information at the load balancer regarding additional data plane traffic received from client devices that will be sent to a second node in the group of nodes. In at least one example, the second node may include a second data node in the group of data nodes. The second node may be associated with a second encrypted tunnel (e.g., a second IPsecSA). In some cases, the indication may include an indication that at least a portion of the first data plane traffic is to be sent to the second node. That is, the indication may inform the load balancer that it needs to adjust where it sends the data plane traffic. For example, the load capacity associated with the first node may have reached or exceeded a threshold load capacity. Additionally or alternatively, the indication may inform the load balancer that the first node is to be removed from the group of nodes (e.g., offline, serviced, etc.).

[0043] In some examples, a controller associated with the network can send instructions to a load balancer or cause instructions to be sent. For example, the controller can receive telemetry data from the group of nodes. Based at least in part on the telemetry data, the controller can determine where the load balancer will adjust where it sends data plane traffic and / or control plane traffic. For example, the telemetry data can indicate the load capacity associated with a corresponding node in the group. Additionally or alternatively, the telemetry data can indicate the status associated with a corresponding node in the group (e.g., whether the node is unhealthy, suspended, or crashed, whether a host is to be rotated, etc.). Therefore, the controller can send instructions to the load balancer and / or, in some examples, send a notification to a first node to prompt the first node to send instructions to the load balancer. In some examples, the controller may include a distributed system that includes a key-value store.

[0044] Based at least in part on this instruction, in some examples, the load balancer and / or controller may prompt the second node to provide one or more resources so that a portion of the first data plane traffic can be sent to the second node. One or more resources may include interfaces, channels, compute resources, etc. In this way, by prompting the second node to provide one or more resources, the second node can "warm up" before data plane traffic is sent to the second node. In examples where a portion of data plane traffic is redirected from the first node to the second node, warming up or pre-configuring the second node can help reduce downtime and / or temporary service interruptions while the second node and client devices negotiate new encrypted connections. In at least one example, prompting the second node to provide one or more resources may include generating and / or sending empty Encapsulated Security Payload (ESP) packets to the second node. Empty ESP packets may include an Internet Protocol (IP) address and a port associated with the client device, and in some cases, other 5-tuple values.

[0045] In some examples, the load balancer and / or controller may send a request to a third node to generate a second SPI value. The load balancer and / or controller may send the request based at least in part on this instruction. In at least one example, the third node includes a first control node (e.g., an IKE node) in a set of control nodes. Furthermore, the request may include a "key update (rekey)" request. That is, the request may be for the third node to create a second SA between the client device and the second node, replacing the first SA between the client device and the first node. Therefore, in some examples, the method may include receiving an instruction regarding the second SPI value at the load balancer. The instruction regarding the second SPI value may include an instruction regarding the association between the second SPI value and the set of 5-tuple values.

[0046] In some examples, the method may include receiving second data plane traffic with a second SPI value and the set of 5-tuple values ​​at the load balancer and from the client device. The second data plane traffic may include some (e.g., a portion) or all of the first data plane traffic previously sent by the load balancer to the first node. Based at least in part on the second data plane traffic with the second SPI value and the set of 5-tuple values, the method may include, in some cases, determining that the second data plane traffic includes some (e.g., a portion) or all of the first data plane traffic. For example, the load balancer may not know the second SPI value, and the load balancer may track all SPI values ​​associated with a particular set of 5-tuple values. Once the load balancer issues a key update request, it may begin monitoring for new / unknown SPIs associated with that set of 5-tuple values ​​and send all data plane traffic with new / unknown SPIs to the second node.

[0047] In some examples, based at least in part on the second SPI value and the set of 5-tuple values, the method may include sending second data plane traffic to the second node. Additionally, in some cases, the method may include removing a first association between the first SPI value and the set of 5-tuple values ​​and / or storing a second association between the second SPI value and the set of 5-tuple values.

[0048] Certain embodiments and implementations of this disclosure will now be described more fully below with reference to the accompanying drawings, in which various aspects are illustrated. However, these aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. For example, while many examples herein are described with respect to ECMP routing, it should be understood that other routing strategies may be used. Furthermore, while many examples are shown as distributed systems, it should be understood that the various processes and methods described may be performed by more or fewer devices. As stated herein, this disclosure includes variations of the embodiments. The same numbers always refer to the same elements.

[0049] Figure 1 illustrates a schematic diagram of an example system architecture 100 for a networking environment 102 including a tunnel communication session comprising separate control plane and data plane traffic flows. Typically, the networking environment 102 may include equipment housed in or located in one or more data centers 104, which may be located in different physical locations. For example, the networking environment 102 may be supported by a network of equipment from a public cloud computing platform, a private / enterprise computing platform, and / or any combination thereof. One or more data centers 104 may be physical facilities or buildings located in a geographic area designated to store the networking equipment as part of the networking environment 102. Data centers 104 may include various network devices, as well as redundant or backup components and infrastructure for power, data communication connectivity, environmental control, and various security devices. In some examples, data centers 104 may include one or more virtual data centers, which are pools or collections of cloud infrastructure resources specifically designed for enterprise needs and / or the needs of cloud-based service providers. Typically, data centers 104 (physical and / or virtual) can provide basic resources such as processors (CPU), memory (RAM), storage devices (disks), and networks (bandwidth). However, in some examples, the devices in the networked environment 102 may not be located in the explicitly defined data center 104, but may be located in other locations or buildings.

[0050] Networking environment 102 can be accessed by client device 106 through one or more networks 108. Networking environment 102 and network 108 may each include one or more networks implemented using any feasible communication technology (e.g., wired and / or wireless modes and / or technologies). Networking environment 102 and network 108 may each include a Personal Area Network (PAN), Local Area Network (LAN), Campus Network (CAN), Metropolitan Area Network (MAN), Extranet, Intranet, Internet, Short-range Wireless Communication Network (e.g., ZigBee, Bluetooth, etc.), Virtual Private Network (VPN), Wide Area Network (WAN) – centralized and / or distributed – and / or any combination, arrangement, and / or aggregation thereof. Networking environment 102 may include devices, virtual resources, or other nodes that relay packets from one network segment to another through nodes in a computer network.

[0051] In some examples, networking environment 102 may provide one or more services 110, host one or more services 110, provide connectivity to one or more services 110, or otherwise support one or more services 110 for client device 106 to connect to and use. Client device 106 may include any type of device configured to communicate over network 108 using various communication protocols (e.g., VPN, SSL, TLS, DTLS, and / or any other protocol). For example, client device 106 may include personal user devices (e.g., desktop computers, laptop computers, telephones, tablets, wearable devices, entertainment devices such as televisions, etc.), network devices (e.g., servers, routers, switches, access points, etc.), and / or any other type of computing device.

[0052] In some examples, the networking environment 102 may include edge routers 112(1) and 112(2) (hereinafter collectively referred to as “edge router 112”), load balancers 114(1)-114(N) (hereinafter collectively referred to as “load balancer 114”) (where N represents any number greater than or equal to one), data nodes 116(1)-116(N), control nodes 118(1)-118(N), firewall nodes 120(1)-120(N), key-value store 122, and controller 124. In various examples, the various systems / devices / nodes of the networking environment 102 may communicate with each other via a management plane and / or message bus associated with the networking environment 102. For example, a common message bus associated with the networking environment 102 may enable a data node to signal to the load balancer that it is about to be removed from the data node queue, the load balancer to adjust where it will send traffic, and so on. Furthermore, the message bus associated with the networking environment 102 may enable any device / system / node of the networking environment 102 to communicate directly with each other.

[0053] In some examples, edge router 112 and load balancer 114 may use ECMP, a policy in which next-hop packet forwarding to a single destination can occur on multiple “best paths” that are tied for first place in routing metric calculations. Furthermore, edge router 112 and load balancer 114 may combine or override ECMP routing with any routing policy, such as Open Shortest Path First (OSPF), Intermediate System to Intermediate System (ISIS), Enhanced Interior Gateway Routing Protocol (EIGRP), Domain Name System (DNS) load balancing, and / or Border Gateway Protocol (BGP). Although shown as separate entities in Figure 1, it is to be understood that in some cases, edge router 112 and load balancer 114 may reside on the same hardware device and / or node.

[0054] In some cases, edge router 112 can balance traffic 126 based on the hash of the network 5-tuple to route packets to load balancer 114. Traffic 126 may include control plane traffic 128 and data plane traffic 130. Furthermore, load balancer 114 can balance traffic 126 based on the hash of the network 6-tuple to route control plane traffic 128 to control node 118 and data plane traffic 130 to data node 116. The network 6-tuple of a packet may include the packet's SPI value, source IP address, source port, destination IP address, destination port, and protocol.

[0055] As shown in the figure, the networking environment 102 may include data nodes 116(1)-116(N) (hereinafter collectively referred to as "data nodes 116") (where N represents any number greater than or equal to one). In some examples, data nodes 116 may process data plane traffic 130 on behalf of the networking environment 102. Data plane traffic 130 may include ESP traffic associated with IPsec connections. In some examples, data node 116(1) of data node 116 may be associated with one or more IPsec security associations. In addition, data node 116 may forward data plane traffic 130 to one or more downstream nodes and / or devices, such as firewall nodes 120(1)-120(N) (hereinafter collectively referred to as "firewall nodes 120") (where N represents any number greater than or equal to one). In some examples, a first data node in data node 116 may be associated with a first traffic category, a second data node in data node 116 may be associated with a second traffic category, and so on. Additionally or alternatively, the first interface of the first data node in data node 116 may be associated with a first traffic category, the second interface of the first data node in data node 116 may be associated with a second traffic category, and so on.

[0056] The networking environment 102 may also include one or more control nodes 118(1)-118(N) (collectively referred to below as "control node 118") (where N represents any number greater than or equal to one). In some examples, control node 118 may process control plane traffic 128 on behalf of the networking environment 102. Control plane traffic 128 may include IKE traffic associated with IPsec connections.

[0057] As shown in the figure, both data node 116 and control node 118 can perform a direct server return (DSR) to send return traffic 132 back to client device 106. That is, data node 116 and control node 118 can send return traffic 132 to client device 106 via edge router 112 (1), bypassing load balancer 114. Alternatively or additionally, data node 116 and control node 118 can bypass edge router 112 (1) and send return traffic 132 directly to client device.

[0058] The network environment 102 may also include a key-value store 122 and a controller 124. The key-value store 122 may include one or more databases accessible to various nodes and devices of the network environment 102. In some examples, the load balancer 114, data node 116, control node 118, and other nodes and / or devices of the network environment 102 may read data from and write data to the key-value store 122. The key-value store 122 may store associations between SPI values ​​and SAs, sets of SPI values ​​and quintuple values, etc. In some examples, the controller 124 may receive telemetry data from data node 116 and / or control node 118 and determine, at least in part, the state associated with each node in data node 116 and / or control node 118 based on the telemetry data. For example, the controller 124 may receive telemetry data indicating the load capacity associated with data node 116(1). The controller 124 can also determine whether the load capacity meets or exceeds the threshold load capacity, and if so, the controller 124 can prompt the data node 116(1) to send a notification to the load balancer 114(1) to request the load balancer 114(1) to adjust the location to which it sends the data plane traffic 130.

[0059] Although depicted as separate hardware components in Figure 1, it should be understood that the edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key-value store 122, and / or controller 124 may be software components that reside at least partially in memory. In this way, one or more processors can execute instructions that cause one or more processors to perform all the operations described herein with respect to the edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key-value store 122, and / or controller 124. In some cases, the edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key-value store 122, and / or controller 124 may be separate hardware and / or software components residing in a separate device or a separate device system. Additionally or alternatively, the edge router 112, load balancer 114, data node 116, control node 118, firewall node 120, key-value store 122, and / or controller 124 may include any type of network device, such as a server, switch, router, hub, bridge, gateway, modem, repeater, access point, etc.

[0060] Figure 2 illustrates a schematic diagram of an example traffic flow 200, in which a load balancer 114(1) sends traffic to downstream nodes according to one or more routing policies. For example, the load balancer 114(1) may receive incoming tunnel traffic 202 from client devices (e.g., one or more client devices 106). In some cases, the incoming tunnel traffic 202 may include control plane traffic 128 and / or data plane traffic 130. Additionally, the incoming tunnel traffic 202 may include an SPI value and a set of 5-tuple values.

[0061] In some examples, when load balancer 114(1) receives incoming tunnel traffic 202, load balancer 114(1) may calculate a hash value representing the SPI value of the incoming tunnel traffic 202 and the value of that 5-tuple. Load balancer 114(1) may then determine, at least in part, a specific node among data nodes 116 or control nodes 118 to which the incoming tunnel traffic 202 will be sent, based on the hash value and using an ECMP routing policy. For example, if the incoming tunnel traffic 202 includes control plane traffic 128 (e.g., IKE traffic), load balancer 114(1) may send control plane traffic 128 to one of the control nodes 118 at least in part based on the hash value. Similarly, if the incoming tunnel traffic 202 includes data plane traffic 130 (e.g., ESP traffic), load balancer 114(1) may send data plane traffic 130 to one of the data nodes 116 at least in part based on the hash value.

[0062] Figure 3 illustrates a data flow diagram of an example traffic flow 300 between individual nodes and / or devices for establishing a load-balanced communication session using SPI values ​​in packet headers. The example traffic flow 300 includes a client 302, a router / load balancer 304, a first header 306, and a second header 308. In this example, the first header 306 and the second header 308 may include data nodes, control nodes, servers, etc. For example, the first header 306 may include a control node, and the second header 308 may include a data node.

[0063] To initiate example traffic flow 300, client 302 sends a connection request packet 310 to router / load balancer 304. Connection request packet 310 may indicate a request to establish an encrypted tunnel so that traffic can flow from client 302 to second headend 308. Connection request packet 310 may include a set of five-tuple values. Upon receiving connection request packet 310, router / load balancer 304 may forward connection request packet 310 to first headend 306. Router / load balancer 304 may determine to send connection request packet 310 to first headend 306 based at least in part on calculating a hash value representing the set of five-tuple values ​​included in connection request packet 310. Additionally or alternatively, router / load balancer 304 may determine to send connection request packet 310 to first headend 306 based at least in part on ECMP routing policies.

[0064] Upon receiving a connection request packet 310, the first headend 306 can establish an IKE session 314 with the client 302. In this way, IKE traffic can flow between the client 302 and the first headend 306. In some cases, establishing an IKE session 314 may include verifying the user 302 associated with the client, for example, by determining the user's identity. Once the IKE session is established, the first headend can send a reply packet 316 to the client 302. The reply packet 316 indicates that the IKE session has been established.

[0065] Client 302 can then send ESP traffic 318 to router / load balancer 304, and router / load balancer 304 can forward the ESP traffic 318 to second headend 308. Upon receiving ESP traffic 318, second headend 308 can generate an SPI value 320 for client 302 to use for sending data plane traffic via the ESP channel. Second headend 308 can further associate the SPI value with a set of 5-tuple values. In this way, second headend 308 can update router / load balancer 304 with a 5-tuple and SPI mapping 322. In some cases, the 5-tuple and SPI mapping 322 may include a hash value. Additionally or alternatively, the 5-tuple and SPI mapping 322 may indicate that future data plane packets, including certain sets of 5-tuple values ​​and certain SPI values, will be sent to second headend 308. Second headend 308 can then send a reply packet 324 back to client 302. Reply packet 324 may indicate that client 302 can begin using the ESP channel or encrypted tunnel to send data plane traffic 326.

[0066] After receiving the reply packet 324, client 302 can begin sending data plane traffic 326 via the ESP channel. When router / load balancer 304 receives the data plane traffic, router / load balancer 304 can calculate a hash value representing the network 5-tuple and an SPI value 328. For example, the packet of data plane traffic 326 may include the SPI value and the network 5-tuple. Based at least in part on the hash calculation, router / load balancer 304 can send data plane traffic 326 to second headend 308. For example, router / load balancer 304 can send data plane traffic 326 to second headend 308 based at least in part on the 5-tuple and SPI mapping 322.

[0067] Figures 4A and 4B illustrate data flow diagrams of example traffic flows 400(1) and 400(2) between individual nodes and / or devices in a communication session that indicate SPI value and / or QoS type information in the SPI value field of the packet header. Example traffic flows 400(1) and 400(2) may include a client 302, a router / load balancer 304, an IKE node 402, and a classifier 404.

[0068] Regarding Figure 4A, client 302 may send a connection request packet 406 to router / load balancer 304. In some examples, connection request packet 406 may include an IKE SA INIT request packet. Connection request packet 406 may indicate a request to establish an encrypted tunnel (e.g., an IPsec connection) for client 302 to send data to and / or receive data from the service. Upon receiving connection request packet 406, router / load balancer 304 may invoke classifier 404 to determine the traffic category associated with connection request packet 406. For example, connection request packet 304 may indicate the type of traffic client 302 wishes to send and / or receive (e.g., voice, video, audio, network, etc.), and classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, connection request packet 406 may include requests to establish multiple connections, each associated with a different traffic category and / or priority. In some examples, invoking classifier 404 may include router / load balancer 304 sending connection request packet 406 to classifier 404.

[0069] In some examples, classifier 404 can operate on data packet 408 to determine the traffic category associated with connection request data packet 406. For example, classifier 404 can determine that connection request packet 406 includes a request to establish one or more of voice traffic channels, video traffic channels, audio traffic channels, network traffic channels, etc. After determining the traffic category, classifier 404 can send a classification packet 410 indicating the traffic category associated with connection request packet 406. Classifier 404 can send classification packet 410 to router / load balancer 304. Router / load balancer 304 can then inject the indication of traffic category classification into the packet header of connection request packet 406. In this way, connection request packet 406 can include an updated connection request packet 414, which includes an indication of traffic category classification information in its packet header.

[0070] Router / load balancer 304 may send updated connection request packets 414 to control nodes such as IKE node 402. IKE node 402 may receive updated connection request packets 414. Based at least in part on traffic category classification information included in the packet header of the updated connection request packet 414, IKE node 402 may generate one or more SPI values ​​416. One or more SPI values ​​416 may indicate, in whole or in part, the traffic category for which data plane traffic is to be processed. That is, a particular SPI value may include a unique combination of bits, and the first bit combination of that unique combination of bits (e.g., the first part of the SPI value) may indicate the traffic category, and the second bit combination of that unique combination of bits (e.g., the second part of the SPI value) may identify the security association between client 302 and one or more hosts associated with the encrypted tunnel connection. In other words, the SPI value field in the data plane packet header may include a first combination indicating the traffic category and a second combination of bits identifying the security association between client 302 and one or more hosts associated with the encrypted tunnel connection.

[0071] After generating one or more SPI values ​​416, the IKE node 402 may send a response packet 418. In some cases, the response packet 418 may include an IKE INIT response packet. Additionally or alternatively, the response packet 418 may include some or all of the one or more SPI values ​​416. In this way, the client 302 may use a first SPI value from the one or more SPI values ​​416 to send first data plane traffic according to a first traffic category associated with a first QoS metric and may use a second SPI value from the one or more SPI values ​​416 to send second data plane traffic according to a second traffic category associated with a second QoS metric.

[0072] Regarding Figure 4B, client 302 may send a connection request packet 406 to router / load balancer 304. In some examples, connection request packet 406 may include an IKE SAINIT request packet instructing IKE node 402 to establish an IPsec secure association. Connection request packet 406 may instruct for establishing an encrypted tunnel (e.g., an IPsec connection) for client 302 to send data to and / or receive data from the service. Upon receiving connection request packet 304, router / load balancer 304 may send connection request packet 406 to IKE node 402. In some examples, router / load balancer 304 may calculate a hash value representing the network 5-tuple included in connection request packet 406 and send connection request packet 406 to IKE node 402 based at least in part on that hash value. For example, router / load balancer 304 may use an ECMP routing policy and determine to send connection request packet to IKE node 402 based at least in part on the hash value.

[0073] To enable IKE node 402 to generate one or more SPI values ​​that match the correct traffic category, IKE node 402 may invoke classifier 404 before establishing a connection for client 302. For example, IKE node 402 may send connection request packet 406 or a portion of connection request packet 406 to the classifier, allowing classifier 404 to determine the traffic category associated with connection request packet 406. For example, connection request packet 406 may indicate the type of traffic that client 302 wishes to send and / or receive (e.g., voice, video, audio, network, etc.), and classifier 404 may be configured to determine what type of traffic that is. Additionally or alternatively, connection request packet 406 may include requests to establish multiple connections, each associated with a different traffic category and / or priority.

[0074] In some examples, classifier 404 can operate on connection request packets 408 to determine the traffic category associated with connection request packets 406. For example, classifier 404 can determine that connection request packet 406 includes a request to establish one or more of voice traffic channels, video traffic channels, audio traffic channels, network traffic channels, etc. After determining the traffic category, classifier 404 can send a classification packet 410 indicating the traffic category associated with connection request packet 406. Classifier 404 can send classification packet 410 to IKE node 402.

[0075] In some examples, IKE node 402 may receive classification packets 410. Based at least in part on the traffic category associated with connection request packets 406, IKE node 402 may generate one or more SPI values ​​416. One or more SPI values ​​416 may indicate, in whole or in part, the traffic category for which data plane traffic is to be processed. That is, a particular SPI value may include a unique combination of bits, and the first bit combination of that unique combination of bits (e.g., a first portion of the SPI value) may indicate the traffic category, and the second bit combination of that unique combination of bits (e.g., a second portion of the SPI value) may identify the security association between client 302 and one or more hosts associated with the encrypted tunnel connection. In other words, the SPI value field in the data plane packet header may include a first combination indicating the traffic category and a second combination of bits identifying the security association between client 302 and one or more hosts associated with the encrypted tunnel connection.

[0076] After generating one or more SPI values ​​416, IKE node 402 may send a response packet 418 to client 302. In some cases, response packet 418 may include an IKE INIT response packet. Additionally or alternatively, response packet 418 may include some or all of the one or more SPI values ​​416. In this way, client 302 may use a first SPI value from one or more SPI values ​​416 to send first data plane traffic according to a first traffic category associated with a first QoS metric and may use a second SPI value from one or more SPI values ​​416 to send second data plane traffic according to a second traffic category associated with a second QoS metric.

[0077] Figures 5A-5C together illustrate a schematic diagram of an example data flow 500 associated with performing a cryptographic tunnel migration. At “1”, load balancer 114(1) may receive traffic 502 from one or more client devices 106 and forward traffic 502 to one or more backend nodes 504(1)-504(N) (hereinafter collectively referred to as “backend nodes 504”) (where N represents any number greater than or equal to one). Traffic 502 may include a first traffic 502(1) to be sent to node 504(1), a second traffic 502(2) to be sent to node 504(2), and an Nth traffic 502(N) to be sent to node 504(N) (where N represents any number greater than or equal to one). Additionally, traffic 502 may include data plane traffic and / or control plane traffic.

[0078] In some examples, load balancer 114(1) can determine that first traffic 502(1), second traffic 502(2), and Nth traffic 502(N) will be sent to nodes 504(1), 504(2), and 504(N), respectively, at least in part, based on an ECMP routing policy. The ECMP routing policy can use six-tuple logic to determine which node in the backend nodes 504 sends the individual packets of traffic 502. The six-tuple logic can include the SPI value of a single packet and a set of five-tuple values ​​(source address, destination address, source port, destination port, and protocol) for a single packet. For example, each packet of first traffic 502(1), second traffic 502(2), and Nth traffic 502(N) can each include a corresponding SPI value and a corresponding set of five-tuple values, and load balancer 114(1) can compute a corresponding hash value for each individual packet, the respective hash value representing the corresponding SPI value and the corresponding set of five-tuple values ​​for each individual packet. In this way, each corresponding hash value of each individual packet of traffic 502 can indicate which node in backend node 504 the individual packet should be sent to.

[0079] In “2”, controller 124 may receive telemetry data 506 associated with backend nodes 504. For example, node 504(1) may send first telemetry data to controller 124, node 504(2) may send second telemetry data to controller 124, and node 504(N) may send Nth telemetry data to controller 124. In some examples, telemetry data 506 may indicate the load capacity associated with each of the backend nodes 504. That is, telemetry data 506 may indicate that node 504(1) operates at 27% capacity, node 504(2) operates at 100% capacity, and node 504(N) operates at 17% capacity. In some examples, the load capacity associated with a backend node may include one or more of the following: tunnel load capacity associated with the backend node, the amount of hardware resources available / used by the backend node, the amount of virtual computing resources available / used by the backend node, etc.

[0080] At “3”, controller 124 may send instruction 508 to load balancer 114(1) to adjust the data flow. That is, to adjust the location (e.g., which node of backend node 504) to which the various portions of traffic 502 are sent by load balancer 114(1). For example, based at least in part on telemetry data 506, controller 124 may determine that the load capacity of node 504(2) exceeds a threshold load capacity. The threshold load capacity may include, for example, a percentage value (e.g., 80%, 85%, 90%, 100%, etc.). Furthermore, the threshold load capacity may be dynamic and change according to the time of day, day of the week, current demand, etc. (e.g., from 80% to 90%). In some cases, controller 124 may send the instruction directly to load balancer 114(1). Additionally or alternatively, controller 124 may send the instruction to node 504(2), as shown in FIG5B.

[0081] At “4”, node 504(2) may send or forward instruction 508 to load balancer 114(1) based at least in part on an instruction received from controller 124. Instruction 508 may be configured to prompt load balancer 114(1) to perform one or more actions to adjust where it sends data streams. Therefore, at “5”, load balancer 114(1) may send instruction 510 to node 504(N) based at least in part on receiving instruction 508 to prompt node 504(N) to prepare one or more interfaces so that a portion of the second traffic 502(2) may be sent / redirected to node 504(N). In at least one example, node 504(N) may include a data node for processing ESP traffic and instruction 510 may include an empty ESP packet containing the source IP address and port associated with a client device of one or more client devices 106. In this way, when node 504(N) receives an empty ESP packet, node 504(N) may begin setting up an interface to prepare for receiving IPsec security associations. Additionally, load balancer 114(1) can send key update requests to the control node responsible for the corresponding IKE session associated with the IPsec security association.

[0082] At “6”, load balancer 114(1) may begin sending additional traffic 512 to node 504(N). Additional traffic 512 may include at least a portion of the second traffic 502(2) previously sent to node 504(2). In this way, the load capacity of node 504(2) may be reduced (e.g., to 75%) and the load capacity of node 504(N) may be increased (e.g., to 42%). In some examples, load balancer 114(1) may begin sending additional traffic 512 to node 504(N) at least in part based on receiving an instruction 508 to adjust the data flow. Furthermore, load balancer 114(1) may determine that additional traffic 512 will be sent to node 504(N) at least in part based on the SPI value included in individual packets of additional traffic 512. For example, in some examples, based at least in part on the load balancer 114(1) sending the key update request, the load balancer 114(1) may not know the SPI value included in the separate packet because the IKE node may have published a new SPI value for use by the client device. Therefore, the load balancer 114(1) can identify a set of 5-tuple values ​​included in the separate packet and determine, at least in part, based on identifying that set of 5-tuple values ​​and at least in part on publishing the key update request, that additional traffic 512 will be sent to node 504(N). Additionally, the load balancer 114(1) can store the association between the new / unknown SPI value and that set of 5-tuple values.

[0083] Figures 6, 7, 8, and 9 illustrate logical flowcharts of various example methods associated with the techniques presented herein for load balancing encrypted traffic based on SPI values. The logical operations described herein with reference to Figures 6, 7, 8, and 9 can be implemented as (1) a series of computer-implemented actions or program modules running on a computing system and / or (2) as machine logic circuits or circuit modules interconnected within the computing system.

[0084] The implementation of the various components described herein depends on the choice of computing system performance and other requirements. Therefore, the logical operations described herein are referred to differently as operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules can be implemented using software, firmware, dedicated digital logic, and any combination thereof. It should also be understood that more or fewer operations than those shown in Figures 6, 7, 8, and 9 and described herein can be performed. These operations can also be performed in parallel or in a different order than that described herein. Some or all of these operations can also be performed by components other than those specifically identified. Although the techniques described in this disclosure are referenced to specific components, in other examples, these techniques can be implemented with fewer components, more components, different components, or any configuration of components.

[0085] Figure 6 illustrates a logical flowchart of an example method 600 for maintaining QoS processing of packets using SPI values. Example method 600 begins with operation 602, which includes receiving a request from a client device and at a network device in the network to establish an encrypted tunnel over the network so that data plane traffic can flow between the client device and the service via the encrypted tunnel. For example, load balancer 114(1) and / or control node 118(1) may receive the request from a first client device among one or more client devices 106. Furthermore, in some examples, the request may include an IKE SA INIT request packet to establish an IPsec SA between the first client device and the first data node 116(1), so that data plane traffic 130 can flow between the client device and the service 110.

[0086] In operation 604, example method 600 includes determining that data plane traffic belongs to a specific traffic category within a set of traffic categories, which is associated with a specific quality of service (QoS) performance metric. In some examples, control node 118(1) may invoke a classifier to determine the specific traffic category. Additionally or alternatively, load balancer 114(1) may invoke a classifier to determine the specific traffic category.

[0087] In operation 606, example method 600 includes generating a Security Parameter Index (SPI) value to be used by a client device for data plane traffic, the SPI value corresponding to a specific traffic class. For example, control node 118(1) may generate an SPI value to be used by a first client device among one or more client devices 106. The SPI value and / or a portion of the SPI value field may correspond to a specific traffic class. That is, in some examples, the SPI value may include a unique combination of bits, and a portion of that unique combination of bits may correspond to a specific traffic class.

[0088] In operation 608, example method 600 includes sending an indication of the SPI value to a client device. For example, control node 118(1) can send the indication of the SPI value directly to a first client device among one or more client devices 106 by performing a Direct Server Return (DSR) to at least bypass load balancer 114. In some examples, the indication may include an IKEINIT response packet indicating that an IPsec SA has been established for the first client device among one or more client devices 106.

[0089] In operation 610, example method 600 includes receiving data packets of data plane traffic, including SPI values, at a load balancing node associated with the network. For example, load balancer 114(1) may receive data packets of data plane traffic 130 from edge router 112(1), or, where load balancing techniques can be performed by edge router 112(1), directly from a first client device. In some cases, edge router 112(1) may execute an ECMP routing policy based on a hash value representing a set of network 5-tuple values ​​included in the data packet and determine which data packet to send to load balancer 114(1).

[0090] In operation 612, example method 600 includes sending data packets over the network based at least in part on data packets including SPI values, such that the data packets are processed according to a specific QoS performance metric. For example, load balancer 114(1) may send data packets to data node 116(1) based on executing an ECMP routing policy that includes calculating a hash value representing the SPI value and the set of network 5-tuple values ​​included in the data packet. Additionally, data node 116(1) may be associated with a traffic class, and load balancer 114(1) may determine to send data packets to data node 116(1) at least in part based on the hash value and / or determining that the SPI value is associated with a traffic class.

[0091] Figure 7 shows a logic flowchart of an example method 700 for load balancing traffic based on the SPI value in the packet header. Example method 700 begins with operation 702, which includes receiving a request from a client device and at a network device in the network to establish an encrypted tunnel over the network so that data plane traffic can flow between the client device and the service via the encrypted tunnel. For example, load balancer 114(1) and / or control node 118(1) may receive the request from a first client device among one or more client devices 106. Furthermore, in some examples, the request may include an IKE SA INIT request packet to establish an IPsec SA between the first client device and the first data node 116(1) so that data plane traffic 130 can flow between the client device and the service 110.

[0092] In operation 704, example method 700 includes generating a Security Parameter Index (SPI) value to be used by a client device for data plane traffic. For example, control node 118(1) may generate an SPI value to be used by a first client device among one or more client devices 118(1). In some examples, the SPI value may identify an IPsec SA between the first client device and one or more data nodes 116 and / or one or more interfaces of data node 116.

[0093] In operation 706, example method 700 includes sending an indication of the SPI value to a client device. For example, control node 118(1) can send the indication of the SPI value directly to a first client device among one or more client devices 106 by performing a Direct Server Return (DSR) to at least bypass load balancer 114. In some examples, the indication may include an IKEINIT response packet indicating that an IPsec SA has been established for the first client device among one or more client devices 106.

[0094] In operation 708, example method 700 includes receiving data packets including SPI values ​​at a load balancer. For example, load balancer 114(1) may receive data packets of data plane traffic 130 from edge router 112(1), or, where load balancing techniques can be performed by edge router 112(1), directly from a first client device. In some cases, edge router 112(1) may execute an ECMP routing policy based on a hash value representing a set of network 5-tuple values ​​included in the data packet and determine which data packet to send to load balancer 114(1).

[0095] In operation 710, example method 700 includes determining, by a load balancer and at least in part based on the SPI value, which server in a group of servers supporting the service will be sent to. In some examples, determining which server in the group of servers will be sent to may also include determining an encrypted tunnel between the load balancer and the server, which will be used to send the data packet to the server. For example, load balancer 114(1) may send the data packet to data node 116(1) based on executing an ECMP routing policy that includes calculating a hash value representing the SPI value and the value of the group of network 5-tuples included in the data packet.

[0096] In operation 712, example method 700 includes sending data packets to a server. For example, load balancer 114(1) may send data packets of data plane traffic 130 to data node 116(1), such that data node 116(1) may forward the data packets to firewall node 120(1), and firewall node 120(1) may then forward the data packets downstream to service 110.

[0097] Figure 8 shows a logic flowchart of an example method 800 for performing encrypted tunnel migration. Example method 800 begins with operation 802, which includes receiving first data plane traffic with a first Security Parameter Index (SPI) value and a set of 5-tuple values ​​at a load balancer and from a client device. For example, load balancer 114(1) may receive the first data plane traffic from edge router 112(1), or directly from a first client device where load balancing techniques can be performed by edge router 112(1). In some cases, edge router 112(1) may execute an ECMP routing policy based on a first hash value representing the set of network 5-tuple values ​​included in the first data plane traffic and determine to send the first data plane traffic to load balancer 114(1).

[0098] In operation 804, example method 800 includes sending first data plane traffic to a first node, associated with a first encrypted tunnel, based at least in part on a first SPI value. For example, load balancer 114(1) may send first data plane traffic 130 to data node 116(1). For example, the first SPI value may identify an IPsec SA between a client device and data node 116(1). In some examples, sending first data plane traffic to the first node may be based at least in part on calculating a second hash value representing a set of network 5-tuple values ​​representing the first SPI value and the first data plane traffic.

[0099] In operation 806, example method 800 includes receiving at the load balancer an indication that additional data plane traffic received from a client device is to be sent to a second node, which is associated with a second encrypted tunnel. For example, the indication could indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, or be serviced. The load balancer may receive the indication from a controller such as controller 124 and / or a node such as control node 118 or data node 116. In some examples, the indication could prompt the load balancer to send a key update request, for example, to one of control nodes 118, to establish a new IPsec SA for the client device. Additionally or alternatively, the indication could prompt the load balancer to send an empty ESP packet to the second data node (e.g., data node 116(N)), the empty ESP packet including the client device's IP address and port, so that the second data node can begin setting up its interface to receive additional data plane traffic.

[0100] In operation 808, example method 800 includes receiving second data plane traffic with a second SPI value and the set of network 5-tuple values ​​at a load balancer and from a client device. For example, load balancer 114(1) may receive the second data plane traffic from edge router 112(1), or directly from a first client device where load balancing techniques can be performed by edge router 112(1). In some cases, edge router 112(1) may execute an ECMP routing policy based on calculating a third hash value, which represents the set of network 5-tuple values ​​included in the second data plane traffic. Because the data packets include the set of network 5-tuple values, the third hash value may be equal to the first hash value, and edge router 112(1) may forward the second data plane traffic to load balancer 114(1).

[0101] In operation 810, example method 800 includes sending second data plane traffic to a second node based at least in part on second data plane traffic having the set of 5-tuple values. For example, load balancer 114(1) may send second data plane traffic to data node 116(N). In some examples, load balancer 114(1) may send second data plane traffic 130 to data node 116(N) based at least in part on the fact that the second SPI value is not identified. Alternatively, load balancer 114(1) may send second data plane traffic 130 to data node 116(N) based at least in part on the fact that the second data plane traffic 130 includes the set of network 5-tuple values. For example, load balancer 114(1) may not have an association stored between the second SPI value and the set of network 5-tuple values. However, because load balancer 114(1) may have issued a key update request, load balancer 114(1) may associate the second SPI value with the set of network 5-tuple values. In other words, because the load balancer 114(1) issues a key update request, when the load balancer 114(1) receives data plane traffic that includes a set of known network quintuple values ​​and a new / unknown SPI value, the load balancer 114(1) can associate the new / unknown SPI value with a set of known network quintuple values ​​and thus send the data plane traffic to the data node 116(N).

[0102] Figure 9 shows a logic flowchart of another example method 900 for performing encrypted tunnel migration. Example method 900 begins with operation 902, which includes receiving first data plane traffic with a first security parameter index (SPI) value and a set of 5-tuple values ​​at a load balancer and from a client device.

[0103] In operation 904, example method 900 includes sending first data plane traffic to a first node based at least in part on a first SPI value and the set of 5-tuple values. For example, load balancer 114(1) may send first data plane traffic 130 to data node 116(1). For example, the first SPI value may identify the IPsec SA between the client device and data node 116(1). In some examples, sending the first data plane traffic to the first node may be based at least in part on calculating a hash value representing the first SPI value and the set of network 5-tuple values ​​of the first data plane traffic.

[0104] In operation 906, example method 900 includes receiving an indication at the load balancer that at least a portion of the first data plane traffic is to be sent to a second node. For example, the indication may indicate that the first node is operating at maximum load capacity or that the first node is about to lose connection, leave, be serviced, etc. The load balancer may receive the indication from a controller such as controller 124 and / or a node such as control node 118 or data node 116. In some examples, the indication may prompt the load balancer to send a key update request, for example, to one of control nodes 118, to establish a new IPsec SA for a client device. Additionally or alternatively, the indication may prompt the load balancer to send an empty ESP packet to the second data node (e.g., data node 116(N)), the empty ESP packet including the IP address and port of the client device, so that the second data node can begin setting up its interface to receive that portion of the first data plane traffic.

[0105] In operation 908, example method 900 includes, at least in part, prompting a second node to provide one or more interfaces based on the instruction, such that at least that portion of the first data plane traffic can be sent to the second node. For example, load balancer 114(1) may send an empty ESP packet to data node 116(N). The empty ESP data packet may include an IP address and / or port associated with a client device, such that data node 116(N) may begin providing one or more interfaces for that portion of the first data plane traffic.

[0106] In operation 910, example method 900 includes receiving second data plane traffic with a second SPI value and the set of five-tuple values ​​at a load balancer and from a client device. The second SPI value may include a new or unknown SPI value. That is, the load balancer may not have already associated the second SPI value with the set of five-tuple values. In operation 912, example method 900 includes determining, at least in part, based on the second SPI value and the set of five-tuple values, that the second data plane traffic includes at least that portion of the first data plane traffic. For example, load balancer 114(1) may track all SPI values ​​associated with the set of five-tuple values. In this way, if load balancer 114(1) issues a key update request, load balancer 114(1) may begin monitoring for new / unknown SPI values ​​associated with the set of five-tuple values ​​to determine the second data plane traffic that includes that portion of the first data plane traffic, which includes the set of five-tuple values ​​and the second (new / unknown) SPI value.

[0107] In operation 914, example method 900 includes sending second data plane traffic to a second node. For example, load balancer 114(1) may send second data plane traffic to data node 116(N). In some cases, sending second data plane traffic to a second node may be based at least in part on determining that the second data plane traffic includes a portion of the first data plane traffic. Additionally or alternatively, sending second data plane traffic to a second node may be based at least in part on an ECMP routing policy and calculating a hash value representing the second SPI value and the set of quintuple values.

[0108] Figure 10 illustrates a schematic diagram of an example computer hardware architecture for implementing network nodes and / or devices (e.g., load balancers, control nodes, data nodes, etc.) that can be used to implement various aspects of the technologies presented herein. The computer architecture shown in Figure 10 illustrates conventional server computers, network devices, workstations, desktop computers, laptop computers, tablets, network devices, e-readers, smartphones, and / or other computing devices, and can be used to execute any of the software components presented herein. Computer 1000 may include networking devices such as servers, switches, routers, hubs, bridges, gateways, modems, repeaters, access points, etc.

[0109] Computer 1000 includes a baseboard 1002 or “motherboard”, which is a printed circuit board that can be connected to a number of components or devices via a system bus or other electrical communication path. In one illustrative configuration, one or more central processing units (“CPU”) 1004 operate in conjunction with chipset 1006. CPU 1004 may be a standard programmable processor that performs the arithmetic and logic operations necessary to perform the operation of computer 1000.

[0110] The CPU 1004 performs operations by transitioning from one discrete physical state to the next discrete physical state (by manipulating switching elements that distinguish and change these states). Switching elements typically include electronic circuitry, such as flip-flops, that holds one of two binary states, and electronic circuitry, such as logic gates, that provides an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adders / subtractors, arithmetic logic units, floating-point units, and more.

[0111] Chipset 1006 provides an interface between CPU 1004 and the remaining components and devices on substrate 1002. Chipset 1006 can provide an interface to RAM 1008, which serves as main memory in computer 1000. Chipset 1006 can further provide an interface to a computer-readable storage medium, such as read-only memory (“ROM”) 1010 or non-volatile RAM (“NVRAM”), for storing basic routines that facilitate booting computer 1000 and transferring information between various components and devices. Depending on the configuration described herein, ROM 1010 or NVRAM can also store other software components necessary for the operation of computer 1000.

[0112] Computer 1000 can operate in a networked environment using logical connections to remote computing devices and computer systems via networks (e.g., network 108 and / or network 1024). Chipset 1006 may include functionality for providing network connectivity via NIC 1012, such as a Gigabit Ethernet adapter. NIC 1012 enables computer 1000 to connect to other computing devices via a network. It should be understood that multiple NICs 1012 may be present in computer 1000, connecting the computer to other types of networks and remote computer systems. In some examples, NIC 1012 may be configured to perform at least some of the techniques described herein and may include components for performing the techniques described herein.

[0113] Computer 1000 can be connected to storage device 1018, which provides non-volatile storage for the computer. Storage device 1018 can store operating system 1020, programs 1022, and data, which have been described in more detail herein. Storage device 1018 can be connected to computer 1000 via storage controller 1014 connected to chipset 1006. Storage device 1018 can consist of one or more physical storage units. Storage controller 1014 can be connected to the physical storage units via Serial Attached SCSI (“SAS”) interface, Serial Advanced Technology Attached (“SATA”) interface, Fibre Channel (“FC”) interface, or other types of interfaces for physical connection and data transfer between the computer and the physical storage units.

[0114] Computer 1000 can store data on storage device 1018 by changing the physical state of physical storage units to reflect the information being stored. In different embodiments of this specification, the specific changes in physical state can depend on various factors. Examples of such factors may include, but are not limited to, the technology used to implement the physical storage units, whether storage device 1018 is characterized as main memory or auxiliary memory, etc.

[0115] For example, computer 1000 can store information in storage device 1018 by issuing instructions via storage controller 1014 to change the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location within an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification; the foregoing examples are provided merely for ease of description. Computer 1000 can further read information from storage device 1018 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0116] In addition to the aforementioned high-capacity storage device 1018, computer 1000 can access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that computer-readable storage media are any available medium that provides non-transitory storage of data and can be accessed by computer 1000. In some examples, operations performed by system architecture 100 and / or any components included therein can be supported by one or more devices similar to computer 1000. In other words, some or all of the operations performed by system architecture 100 and / or any components included therein can be performed by one or more computer devices 1000 operating in a cloud-based configuration.

[0117] By way of example and not limitation, computer-readable storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory or other solid-state storage technologies, optical disc ROM (“CD-ROM”), digital versatile optical disc (“DVD”), high-definition DVD (“HD-DVD”), Blu-ray or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0118] As described above, storage device 1018 may store operating system 1020 for controlling the operation of computer 1000. According to one embodiment, the operating system includes a LINUX operating system. According to another embodiment, the operating system includes a Microsoft Corporation from Redmond, Washington. SERVER operating system. According to a further embodiment, the operating system may include one of the UNIX operating systems or variants thereof. It should be understood that other operating systems may also be used. Storage device 1018 may store other systems, applications, and data used by computer 1000.

[0119] In one embodiment, storage device 1018 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into computer 1000, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions transform computer 1000 by specifying how CPU 1004 transitions between states. According to one embodiment, computer 1000 can access a computer-readable storage medium storing computer-executable instructions that, when executed by computer 1000, perform the various processes described above with respect to Figures 1 through 9. Computer 1000 may also include a computer-readable storage medium storing instructions for performing the operations of any other computer implementations described herein.

[0120] Computer 1000 may also include one or more input / output controllers 1016 for receiving and processing input from multiple input devices, such as keyboards, mice, touchpads, touchscreens, electronic styluses, or other types of input devices. Similarly, input / output controllers 1016 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, or other types of output device. It should be understood that computer 1000 may not include all the components shown in FIG. 10, may include other components not explicitly shown in FIG. 10, or may utilize an architecture entirely different from that shown in FIG. 10.

[0121] As described herein, computer 1000 may include one or more of a data node, control node, firewall node, edge router, and / or key-value storage device. Computer 1000 may include one or more hardware processors 1004 (processors) configured to execute one or more stored instructions. Processor 1004 may include one or more cores. Furthermore, computer 1000 may include one or more network interfaces (e.g., NIC 1012) configured to provide communication between computer 1000 and other devices via a network (e.g., networks 108 and 1024). Network interfaces may include devices configured to couple to a personal area network (PAN), wired and wireless local area network (LAN), wired and wireless wide area network (WAN), etc. For example, network interfaces may include interfaces for Ethernet, Wi-Fi, etc. TM Compatible devices.

[0122] Program 1022 may include any type of program or process to perform the techniques described in this disclosure to load balance encrypted traffic based on the SPI value of the packet header, and to use the SPI value to indicate QoS and migrate encrypted connections to different hosts.

[0123] In summary, this paper describes techniques for load balancing encrypted traffic based on the Security Parameter Index (SPI) value and the set of 5-tuple values ​​in the packet header. Furthermore, this paper describes techniques for including Quality of Service (QoS) type information in the SPI value field of the packet header. QoS type information can indicate the specific traffic category on which data packets are processed. Additionally, this paper describes techniques for pre-configuring backend hosts so that encrypted traffic can be migrated from one backend host to another without causing temporary service interruptions.

[0124] While the invention has been described with respect to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. For example, although many examples are described with respect to the IPsec protocol, it should be understood that the described techniques are applicable to other protocols. Since other modifications and variations to suit specific operational requirements and environments will be apparent to those skilled in the art, the invention is not to be considered limited to the examples chosen for disclosure purposes, and covers all changes and modifications that do not constitute a departure from the true spirit and scope of the invention.

[0125] Although this application describes embodiments with specific structural features and / or methodological actions, it should be understood that the claims are not necessarily limited to the specific features or actions described. Rather, the specific features and actions are merely illustrative embodiments that fall within the scope of the claims of this application.

Claims

1. A method comprising: Receive a request from the client device and at the network device of the network to establish an encrypted tunnel through the network so that data plane traffic can flow between the client device and the service via the encrypted tunnel; At the network device, the data plane traffic is determined to belong to a first traffic category in a set of traffic categories, the first traffic category being associated with a first quality of service (QoS) performance metric; a first security parameter index (SPI) value is generated at the network device for use by the client device for the data plane traffic, the first SPI value corresponding to the first traffic category; an indication regarding the first SPI value is sent from the network device to the client device; a first data packet including the first SPI value of the data plane traffic is received at a load balancing node associated with the network; the first data packet is sent from the load balancing node through the network, at least in part, based on the first data packet including the first SPI value, such that the first data packet is processed according to the first QoS performance metric; a second data packet is received at the load balancing node, the second data packet including a second SPI value corresponding to a second traffic category, the second traffic category being associated with a second QoS performance metric; And the second data packet is sent from the load balancing node through the network based at least in part on the second data packet including the second SPI value, such that the second data packet is processed according to the second QoS performance metric.

2. The method as described in claim 1, wherein, Generating the first SPI value includes: generating a first bit combination representing the first QoS performance metric of the first data packet to be processed, at least in part based on the first traffic category; generating a second bit combination representing a security association; and masking the first bit combination and the second bit combination such that the first bit combination includes a first portion of the first SPI value and the second bit combination includes a second portion of the first SPI value.

3. The method as described in claim 2, wherein, The first combination is represented by a first hexadecimal digit, and the second combination is represented by a plurality of hexadecimal digits.

4. The method of claim 1, wherein, The first part of the first SPI value is a first identifier corresponding to the first traffic category, and the second part of the first SPI value is a second identifier corresponding to the security association of the network.

5. The method of claim 1, wherein sending the first data packet over the network comprises sending the first data packet over the network using an equal cost multipath (ECMP) routing algorithm based at least in part on the first SPI value and the 5-tuple of the first data packet.

6. The method of claim 1, wherein generating the first SPI value includes generating a plurality of SPI values ​​to be used by the client device for the data plane traffic, each of the plurality of SPI values ​​corresponding to a corresponding traffic category, each corresponding traffic category being associated with a corresponding QoS performance metric.

7. A system comprising: One or more processors; And a non-transitory computer-readable medium storing one or more instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: receiving from a client device and at a network device of the network a request to establish an encrypted tunnel through the network to allow data plane traffic to flow between the client device and a service via the encrypted tunnel; determining at the network device that the data plane traffic belongs to a first traffic category in a set of traffic categories, the first traffic category being associated with a first quality of service (QoS) performance metric; generating at the network device a first security parameter index (SPI) value to be used by the client device for the data plane traffic, the first SPI value corresponding to the first traffic category; and sending information about the first SPI value from the network device to the client device. The instructions are as follows: receiving a first data packet, including the first SPI value, from the client device and at a load balancing node associated with the network; transmitting the first data packet from the load balancing node via the network, at least in part based on the first data packet including the first SPI value, such that the first data packet is processed according to the first QoS performance metric; receiving a second data packet at the load balancing node, the second data packet including a second SPI value corresponding to a second traffic category associated with a second QoS performance metric; and transmitting the second data packet from the load balancing node via the network, at least in part based on the second data packet including the second SPI value, such that the second data packet is processed according to the second QoS performance metric.

8. The system of claim 7, wherein, Generating the first SPI value includes: generating a first bit combination representing the first QoS performance metric of the first data packet to be processed, at least in part based on the first traffic category; generating a second bit combination representing a security association; and masking the first bit combination and the second bit combination such that the first bit combination includes a first portion of the first SPI value and the second bit combination includes a second portion of the first SPI value.

9. The system of claim 8, wherein, The first combination is represented by a first hexadecimal digit, and the second combination is represented by a plurality of hexadecimal digits.

10. The system as claimed in any one of claims 7 to 9, wherein, The first part of the first SPI value is a first identifier corresponding to the first traffic category, and the second part of the first SPI value is a second identifier corresponding to the security association of the network.

11. The system of claim 7, wherein sending the first data packet over the network comprises sending the first data packet over the network using an equal cost multipath (ECMP) routing algorithm based at least in part on the first SPI value and the 5-tuple of the first data packet.

12. The system of claim 7, wherein generating the first SPI value includes generating a plurality of SPI values ​​to be used by the client device for the data plane traffic, each of the plurality of SPI values ​​corresponding to a corresponding traffic category, each corresponding traffic category being associated with a corresponding QoS performance metric.

13. A non-transitory computer-readable medium storing instructions that, when executed by one or more computing devices, cause the computing devices to perform operations, the operations including: Receive a request from the client device and at the network device of the network to establish an encrypted tunnel through the network so that data plane traffic can flow between the client device and the service via the encrypted tunnel; At the network device, it is determined that the data plane traffic belongs to a first traffic category in a set of traffic categories, the first traffic category being associated with a first quality of service (QoS) performance metric; at the network device, a first security parameter index (SPI) value is generated to be used by the client device for the data plane traffic, wherein a first portion of the first SPI value corresponds to the first traffic category; and an indication regarding the first SPI value is sent from the network device to the client device. A first data packet, including the first SPI value, is received from the client device and at a load balancing node associated with the network as part of the data plane traffic; the first data packet is transmitted from the load balancing node through the network, at least in part, based on the first data packet including the first SPI value, such that the first data packet is processed according to the first QoS performance metric; a second data packet, including a second SPI value corresponding to a second traffic category associated with a second QoS performance metric, is received at the load balancing node; And the second data packet is sent from the load balancing node through the network based at least in part on the second data packet including the second SPI value, such that the second data packet is processed according to the second QoS performance metric.

14. The non-transitory computer-readable medium of claim 13, wherein generating the first SPI value comprises: At least in part, a first-order combination of the first QoS performance metric representing the first data packet to be processed is generated based on the first traffic category; Generate a second combination representing the security association; And mask the first bit combination and the second bit combination such that the first bit combination includes a first portion of the first SPI value and the second bit combination includes a second portion of the first SPI value.

15. The non-transitory computer-readable medium of claim 14, wherein the first bit combination is represented by a first hexadecimal digit, and the second bit combination is represented by a plurality of hexadecimal digits.

16. The non-transitory computer-readable medium as claimed in any one of claims 13 to 15, wherein, The second part of the first SPI value is a second identifier corresponding to the security association of the network.

17. The non-transitory computer-readable medium of claim 13, wherein generating the first SPI value includes generating a plurality of SPI values ​​to be used by the client device for the data plane traffic, each of the plurality of SPI values ​​corresponding to a corresponding traffic category, each corresponding traffic category being associated with a corresponding QoS performance metric.

18. An apparatus comprising: A means for receiving, from a client device and at a network device in a network, a request to establish an encrypted tunnel over the network so that data plane traffic flows between the client device and the service via the encrypted tunnel; A means for determining at the network device that the data plane traffic belongs to a first traffic category in a set of traffic categories, the first traffic category being associated with a first quality of service (QoS) performance metric; Means for generating at the network device a first security parameter index (SPI) value for use by the client device for the data plane traffic, the first SPI value corresponding to the first traffic category; A means for sending an indication of the first SPI value from the network device to the client device; A means for receiving a first data packet, including the first SPI value, of the data plane traffic at a load balancing node associated with the network; A means for performing the following operations: transmitting the first data packet from the load balancing node via the network based at least in part on the first data packet including the first SPI value, such that the first data packet is processed according to the first QoS performance metric; and means for receiving a second data packet at the load balancing node, the second data packet including a second SPI value corresponding to a second traffic category associated with a second QoS performance metric. And means for performing the following operations: transmitting the second data packet from the load balancing node via the network, at least in part based on the second data packet including the second SPI value, such that the second data packet is processed according to the second QoS performance metric.

19. The apparatus of claim 18, further comprising means for carrying out the method of any one of claims 2 to 6.

20. A computer program product or computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method as claimed in any one of claims 1 to 6.

Images