Computer-implemented method and computer system (Version tagging in packets using boundary NAT)

Abstract

To provide a computer-implemented method and system for maintaining flow affinity regardless of version or state changes in a network receiving data.SOLUTION: A process of packet processing includes: receiving, at a receiving device, a packet from a node in a first network; determining a version identifier for the packet; encoding the version identifier into the packet; and transmitting the packet containing the encoded version identifier to a load balancing device in a second network. The version identifier may be encoded into a destination port field of the packet. The packet is received at the load balancing device. The load balancing device extracts the version identifier and executes a hash of source address information. The version and hash are used to select a back-end device in the second network. The packet is transmitted to the selected back-end device.SELECTED DRAWING: Figure 7B

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to transmitting data between networks, and more particularly to maintaining flow affinity despite changes in the version or state of the network receiving the data. [Background technology]

[0002] Maintaining flow affinity ensures that related data packets, or "data flows," are forwarded to the same destination. A data flow, for example, could be communication between a client in a particular network and a backend server in a different network. Maintaining data flow affinity across a network is a desirable network attribute in network forwarding functions (e.g., load balancing systems) that support multiple destinations and / or multiple routes. For example, data packets from the same Transmission Control Protocol (TCP) connection require affinity to avoid connection loss, data packets from the same source IP address require affinity to enable Transport Layer Security (TLS) handshakes, and data packets from the same client computing device require affinity to support application-specific optimizations and session state. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure aims to provide a technique that can maintain flow affinity regardless of changes in the version or state of the network that receives the data. [Means for solving the problem]

[0004] In an embodiment of the present invention, the connection tracking functionality of a connection tracking device is utilized to enable affinity preservation decisions by a forwarding device. Relevant information, such as the configuration version at the start of each connection, may be passed from the connection tracking device to the forwarding device. The connection tracking device may be located anywhere on the path between the source (e.g., client) and the forwarding device. The connection tracking device may be on the perimeter of the source network or on the perimeter of the forwarding device's network. The connection tracking device and forwarding device may be devices that also perform network address translation (NAT) or load balancing, or any other network devices where the connection tracking device performs connection tracking and the forwarding device makes affinity-based decisions.

[0005] The communication between the connection tracking device and the forwarding device to pass the version information can be performed in any connection-non-disruptive manner. In one embodiment, this involves encoding a version identifier into the destination port of each packet. This has the advantage of not changing the packet size and is a standard Network Address / Port Translation (NAPT) operation that can be easily reversed by the forwarding device. It is therefore transparent to the client and backend server. In another embodiment, an overlay tunnel, such as a virtual private network (VPN), can be created between the connection tracking device and the forwarding device. In this embodiment, the version information is encoded into the overlay rather than into individual packets. A tunnel can then be created for a specific version, and the version identifier can be attached to the tunnel label in the header.

[0006] Various embodiments relate to a computer-implemented method. In one embodiment, the method may include, at a connection tracking device, receiving a packet from a node in a first network, determining a version identifier of the packet, encoding the version identifier in the packet, and sending the packet including the encoded version identifier to a forwarding device. These operations may be performed by the connection tracking device. In one embodiment, the forwarding device selects a backend device in a second network based on a client connection affinity and sends the packet to the selected backend device. The version identifier may be encoded in a destination port field of the first packet. The connection tracking device may be a perimeter network address translation (NAT) device. The forwarding device may be a load balancing device. The connection tracking device may indicate its own timestamp, and the load balancing device may track the correlation between the timestamp and the version. This may not be perfectly synchronized, but this is acceptable if the timestamp updates much faster than the version. The load balancing device only needs a way to repeat the decision made for the first packet of a connection for all other packets in that connection. The connection tracking device can ensure that the same timestamp is used for all packets in a connection. The timestamp needs to be updated fast enough so that when the load balancer version changes, new connections will get a different mapping, and fast enough so that fewer packets are mapped using the old version.

[0007] In one embodiment, the version identifier corresponds to a set of backend devices currently operational in the second network. In various embodiments, the current version is time-based, and determining the version identifier may include determining a period of time during which the current version identifier has been in use, and updating the version identifier if the determined period of time exceeds a period specified for updating the version identifier.

[0008] In various embodiments, the current version is based on a packet count, and determining the version identifier can include determining the number of packets received at the connection tracker that use a particular version identifier, and updating the version identifier if the determined number of packets exceeds a specified packet count for updating the version identifier. Similar to the timestamp-version correlation described above, the load balancer simply needs a way to repeat the determination made for the first packet of a connection for all other packets in that connection. The connection tracker can ensure that the same version is used for all packets in a connection. The version needs to be updated fast enough so that when the load balancer version is changed, new connections will have different mappings. If the update rate is fast enough, fewer packets will be mapped using the old version.

[0009] In various embodiments, determining the version identifier includes receiving, by a connection tracking device, a control message from the other node, and updating the version identifier in response to the control message.

[0010] In various embodiments, determining the version identifier includes receiving, by the connection tracking device, a return packet that is sent to the connection tracking device in response to the packet including the encoded version identifier being sent to the forwarding device, the return packet having the version identifier encoded in it, and updating the current version based on the version identifier encoded in the return packet.

[0011] In various embodiments, the version identifier corresponds to a hash function used by a load balancer device in the second network or a policy adopted by the load balancer device.

[0012] In another embodiment, a computer-implemented method includes receiving a packet from a network device in a first network by a network device in a second network. The forwarding device and at least one back-end device may be included in the second network. The method may include extracting, by the forwarding device, a version identifier of the packet from the packet. The method may further include extracting, by the forwarding device, a source address and a source port from the packet and determining a hash value using a specific hash function based on the extracted source address and source port. The method may further include using the hash value to select, by the forwarding device, a network address of a back-end device corresponding to a remainder when the integer hash value is divided by the number of back-end devices, encoding, by the forwarding device, the selected network address as a destination address in the packet, and transmitting, by the forwarding device, the packet including the selected network address to the back-end device having the selected network address.

[0013] In some embodiments, the particular hash function used to determine the hash value is selected from two or more hash functions based on the version identifier of the packet.

[0014] In some embodiments, the particular set of backend devices currently operational in the second network is selected from two or more sets of backend devices based on a version identifier included in the packet.

[0015] Other embodiments relate to systems, apparatus, or computer program products for performing the methods disclosed herein.

[0016] This Summary presents in a simplified form some concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a cloud computing environment in accordance with an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating abstraction model layers according to an embodiment of the present invention. [Figure 3] FIG. 3 is a block diagram illustrating a computer device that can be used as the network device of FIG. 4, according to an embodiment of the present invention. [Figure 4] FIG. 4 is a functional block diagram illustrating an example computing environment in accordance with an embodiment of the present invention. [Figure 5] FIG. 5 is a functional block diagram illustrating data flow between networks according to an embodiment of the present invention. [Figure 6A] FIG. 6A is a block diagram illustrating an example of packet flow between networks. [Figure 6B] FIG. 6B is a block diagram illustrating packet flow between networks in accordance with various embodiments of the present invention. [Figure 7A]FIG. 7A is a flowchart illustrating the process of packet processing by a network gateway (referred to herein as a "boundary NAT" or "connection tracking" device) according to various embodiments of the present invention. [Figure 7B] FIG. 7B is a flowchart illustrating a process for packet processing by a load balancer (also referred to herein as a "forwarding" device) according to various embodiments of the present invention. [Figure 8] FIG. 8 is a block diagram illustrating packet routing at a load balancer in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] In distributed utility computing environments, such as clouds and other networks, it is desirable for resources providing computing services to be able to flexibly scale up and down in response to the service loads imposed by client computing devices. When a client's service load increases, the network is expected to adapt by allocating more resources and increasing service capacity. This prevents network outages due to increased service loads on client computing devices. When a client's service load decreases, the network is expected to adapt by releasing unnecessary resources in a timely manner, reducing overall operational costs and allocating resources elsewhere to improve overall resource utilization. Resource flexibility is typically achieved through a scalable pool of backend network service nodes (e.g., backend network servers). Backend network service nodes can be added or removed to align the allocated resource capacity with the total service load imposed by clients. A load balancer can be added as an additional entity to a distributed computing environment to receive client requests and multiplex these requests among the backend network service nodes in the pool.

[0019] A load balancer is typically configured to balance client requests for computing services and forward them across a network to back-end network service nodes. A load balancing algorithm that determines the routing path for forwarding network client data packets must consider multiple dynamic parameters. These parameters include, for example, the pool size of back-end network service nodes, resource utilization, resource capacity, and the characteristics of network client requests. To prevent the load balancer from becoming a single point of failure or a service bottleneck, the load balancer may be implemented as an elastic service using a pool of load balancer nodes. The pool of load balancer nodes can also be dynamic, in that load balancers can be added or removed from the pool in response to load balancer failures or changes in network load. In this case, data packets from the same data flow (i.e., from the same network client computing device) can be load balanced by multiple different load balancer nodes. An elastic pool of load balancers typically attempts to maintain affinity between clients and load balancers.

[0020] When balancing network load, it is important and often necessary to route related data packets ("data flows") from the same network client computing device to the same backend network service node. Routing data flows from a given client computing device to the same backend network service node is referred to herein as "client connection affinity." Maintaining affinity in large-scale network environments or large-scale cloud environments can pose significant scalability challenges and lead to network bottlenecks. These challenges are further complicated in dynamic networks where network components may be added, modified, removed, or become unavailable. Load balancer forwarding decisions must be modified while maintaining affinity. For example, when a new backend network service node (e.g., a backend network server) is added, new data flows may be forwarded to the new backend network service node. However, existing data flows should continue to be routed to the existing backend network service node, if possible. As another example, in a network environment that utilizes a pool of load balancer nodes, multiple load balancer nodes may be dynamically added or removed, and the resulting per-flow forwarding decisions must be consistent across all load balancer nodes.

[0021] Connection tracking can be used to maintain affinity in large, dynamic networks by tracking active connections. Connection tracking stores the forwarding decision for each data flow throughout its lifetime. For example, the TCP 5-tuple can be tracked over the lifetime of each TCP flow, and a forwarding decision entry can be recorded in a forwarding table as a forwarding rule. The term "5-tuple" refers to a set of five distinct values ​​required to initiate a connection over the Internet Protocol (IP), including the source IP address / source port number, the destination IP address / destination port number, and the network protocol in use. Note that an "IP address" refers to an address that can be used by protocols such as the Internet Protocol (IP) to identify a specific client at the network layer, and a "port" refers to a port number that can be assigned to a given service at the transport layer; for example, the Telnet protocol uses port number "23." Note that embodiments of the present invention do not require a specific protocol; any protocol, such as TCP or User Datagram Protocol (UDP), may be used.

[0022] Connection tracking may be implemented in load balancers. With connection tracking, each newly arrived IP packet is first checked to see if it is part of an existing data flow by looking up its 5-tuple. If so, the existing forwarding rules are applied. However, connection tracking in this manner is not scalable, as both the system memory requirements and the lookup time for forwarding decision entries increase with the number of data flows. Another drawback of connection tracking is the need to synchronize the connection tracking state across all load balancer nodes. While it may be possible to capture minimal state, such as the "version" of the load balancer configuration, some mechanism is still needed to consistently map traffic flows to specific versions.

[0023] Another method for maintaining affinity is hash-based routing, where the load balancer calculates a hash of the flow attributes based on the number of available backend servers. With hash-based routing, affinity is maintained as long as the number of backend servers remains constant. If a backend server is added or removed, affinity is lost.

[0024] Hash-based routing has been improved by using "consistent hashing," a distributed hashing scheme that works independently of the number of backend servers. In this method, the load balancer calculates a consistent hash of the flow attributes and uses the hash result to make routing decisions. Consistent hashing ensures that for most flows, the hash remains constant even as service instances are added or removed. The drawbacks of consistent hashing are that it requires high computational power, consuming a huge amount of memory, and that it works for most flows, but not all flows.

[0025] Other techniques for maintaining affinity, such as the use of cookies, are known, but these techniques also have various drawbacks. For example, using cookies requires clients to cooperate in maintaining affinity by inserting cookies into messages, which exposes routing decisions and therefore the IP addresses of backend servers to the client. This can have adverse effects on network security. Another disadvantage is the performance overhead incurred by the load balancer, as it must parse and manipulate protocol messages to ensure that clients insert cookies in a protocol-appropriate manner.

[0026] Therefore, there is a need for a technology that enables fast and reliable transitioning between different versions of multi-node services implemented in utility computing systems. Version, in its broadest sense, includes changes such as configuration changes, node hardware or software upgrades, and adding or removing nodes (flexible changes to service pool sizes).

[0027] Advantages of the novel embodiments described herein include using a network device on the perimeter of a first network that includes multiple clients, such as a private network. Using a network device on the perimeter of the first network can reduce the need for connection tracking by a load balancer in a second network, such as a network that hosts multiple backend servers. Another advantage is that it can reduce the processing and memory requirements of the load balancer in the second network. A further advantage is that the novel embodiments described herein are transparent to the clients and backend devices.

[0028] According to embodiments of the present communication system and method, this capability can be provided by offloading the connection tracking function to another network device. Many network environments have perimeter gateway devices, such as routers and switches, to separate different attributes between separate network domains (e.g., private network addresses versus public network addresses). A common function of perimeter gateway devices is Network Address Translation (NAT), which replaces the internal IP address with the external IP address (or vice versa) in each packet. NATs require connection tracking so that the same address translation occurs for all packets of the same connection and so that return packets can be translated back. In other words, NATs inherently maintain flow affinity. Because these perimeter gateway devices typically perform NAT, they are also referred to herein as "perimeter NATs." In addition to IP address translation, perimeter NATs may also translate or modify port numbers. Perimeter NATs maintain "versions" in a load-balanced distributed network and can insert the current version into fields such as the destination port field in a given packet. The load balancer can then be configured to record the current version when its state changes, and can look at the assigned field in the packet to determine the version that was in use at the start of the connection (i.e., the version associated with the first packet in the data flow), and use this information along with other fields in the packet's 5-tuple to determine which backend server to route the packet to.

[0029] Although this disclosure includes detailed descriptions of cloud computing, implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the invention can be practiced in conjunction with any other type of computing environment now known or later developed.

[0030] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five characteristics, at least three service models, and at least four implementation models.

[0031] The characteristics are as follows:

[0032] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time or network storage, automatically as needed, without the need for human interaction with the service provider.

[0033] Broad network access: Computing power is available over the network and can be accessed through standard mechanisms, facilitating use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, PDAs).

[0034] Resource Pooling: Computing resources from a provider are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated based on demand. Consumers generally have no control or knowledge of the exact location of the resources they are provided with, so there is a sense of location independence. However, consumers may be able to identify their location at a higher level of abstraction (e.g., country, state, data center).

[0035] Rapid Elasticity: Computing capacity can be provisioned quickly and elastically, sometimes automatically, to instantly scale out and quickly released to instantly scale in. To the consumer, the computing power available to provision often appears unlimited, and can be purchased at any time and in any quantity.

[0036] Metered Services: Cloud systems leverage measurement capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts) to automatically control and optimize resource usage. Resource usage can be monitored, controlled, and reported to provide transparency to both providers and consumers of utilized services.

[0037] The service model is as follows:

[0038] Software as a Service (SaaS): The functionality offered to the consumer is the availability of a provider's applications running on a cloud infrastructure. The applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., webmail). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for limited user-specific application configuration settings.

[0039] Platform as a Service (PaaS): The capability offered to consumers is to deploy applications they create or acquire onto a cloud infrastructure using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of their hosting environment.

[0040] Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processors, storage, networking, and other basic computing resources on which the consumer can deploy and run any software, including operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, and deployed applications, and in some cases partial control over some network components (e.g., host firewalls).

[0041] The deployment model is as follows:

[0042] Private Cloud: This cloud infrastructure is dedicated to a specific organization and can be managed by that organization or a third party, and can exist on-premise or off-premise.

[0043] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance). This cloud infrastructure can be managed by those organizations or a third party and can exist on-premises or off-premises.

[0044] Public cloud: This cloud infrastructure is available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0045] Hybrid cloud: This cloud infrastructure combines two or more cloud models (private, community, or public), each of which retains its inherent nuances but is bound by standards or specific technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0046] A cloud computing environment is a service-oriented environment that emphasizes statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0047] FIG. 1 illustrates an exemplary cloud computing environment 50. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers (e.g., PDAs or cell phones 54A, desktop computers 54B, laptop computers 54C, or automobile computer systems 54N, or combinations thereof) can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped (not shown) in one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof. This enables the cloud computing environment 50 to provide infrastructure, platform, or software as a service, or combinations thereof, for which cloud consumers are not required to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N illustrated in FIG. 1 are exemplary only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of electronic device via any type of network or network-addressable connection (e.g., using a web browser), or both.

[0048] Next, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 1) is shown in FIG. 2. It should be understood in advance that the components, layers, and functions shown in FIG. 2 are merely exemplary, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0049] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, reduced instruction set computer (RISC) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and network components 66, such as load balancers. In some embodiments, software components include network application server software 67 and database software 68.

[0050] The virtualization layer 70 provides an abstraction layer from which virtual entities such as virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be provided.

[0051] By way of example, the management layer 80 may provide the following functionality: Resource provisioning 81 enables dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 enables cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. By way of example, these resources may include application software licenses. Security enables identification and verification of cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 enables allocation and management of cloud computing resources so that requested service levels are met. Service level agreement (SLA) planning and fulfillment 85 enables advance arrangement and procurement of anticipated future cloud computing resources required in accordance with SLAs.

[0052] The workload layer 90 provides examples of functionality available in a cloud computing environment. Examples of workloads and functionality that can be provided from this layer include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and other applications 96, such as programs residing in memory 308 of FIG. 3.

[0053] FIG. 3 is a block diagram illustrating a computing device 300 capable of implementing processes related to embodiments described herein. The computing device 300 is typically a programmed general-purpose computing system, such as an embedded processor, a system-on-a-chip, a personal computer, a workstation, a server system, and a minicomputer or mainframe computer. Similarly, the computing device 300 may be implemented in a communications device, such as a network switch or router. The computing device 300 may include one or more processors (CPUs) 302A-B, input / output circuitry 304, a network adapter 306, and memory 308. The CPUs 302A-B execute program instructions to perform the functions of the present communications systems and methods. In the embodiment illustrated in FIG. 3, the computing device 300 is implemented as a single multiprocessor computing system, with the multiple processors 302A-B sharing system resources, such as the memory 308, the input / output circuitry 304, and the network adapter 306. However, the present communications systems and methods also include embodiments in which the computing device 300 is implemented as multiple networked computing systems. These computing systems may be single-processor computing systems, multiprocessor computing systems, or a combination thereof.

[0054] Input / output circuitry 304 provides the ability to input data to and output data from computing device 300. For example, input / output circuitry may include input devices such as a keyboard, mouse, touchpad, trackball, scanner, analog-to-digital converter, etc., output devices such as a video adapter, monitor, printer, and modem, etc. Network adapter 306 interfaces computing device 300 with network 310. The network may be any public or private LAN or WAN, such as the Internet.

[0055] Memory 308 stores program instructions executed by and data used and processed by CPUs 302A-B to perform the functions of computer device 300. Memory 308 may include, for example, electronic memory devices such as RAM, ROM, programmable ROM (PROM), electrically erasable programmable ROM (EEPROM), and flash memory, as well as electromechanical memory devices such as magnetic disk drives, tape drives, and optical disk drives. These may use an Integrated Drive Electronics (IDE) interface or a variant or extension thereof (such as Enhanced IDE (EIDE) or Ultra Direct Memory Access (UDMA)), a Small Computer System Interface (SCSI) standard interface or a variant or extension thereof (such as Fast SCSI, Wide SCSI, or Fast and Wide SCSI), Serial ATA (SATA) or a variant or extension thereof, or a Fibre Channel Arbitrated Loop (FC-AL) interface.

[0056] The contents of memory 308 may vary depending on the functions that computing device 300 is programmed to perform. In the example shown in Figure 3, memory contents representing routines and data for performing embodiments of the processes described herein are shown. However, these routines and their associated memory contents need not be contained in a single system or device, but may be distributed across multiple systems or devices based on well-known technological considerations. The present communication systems and methods may include any such configuration.

[0057] In the example shown in FIG. 3 , in the case where the computing device 300 implements a boundary NAT 430, the memory 308 may include a boundary NAT program 324. Also, in the case where the computing device 300 implements a load balancer node 450, the memory 308 may include a load balancing program 326. In a further example, in the case where the computing device 300 implements a client 410, the memory 308 may include a client program 320. Also, in the case where the computing device 300 implements a back-end node 470, the memory 308 may include a back-end program 322. In these and other examples, an operating system 328 may be included. For example, the boundary NAT program 324 may include routines and data that provide the functionality to manipulate packets by replacing destination IP addresses and ports or other packet fields. The load balancing program 326 may include routines and data that provide the functionality to spread data traffic to other devices, such as the back-end node 470. The operating system 328 may provide the overall functionality of the system.

[0058] As shown in Figure 3, the communication system and method may be implemented in one or more systems that provide multiprocessor, multitasking, multiprocess, or multithreaded computing, or a combination thereof, as well as in systems that only provide single-processor or single-threaded computing. Multiprocessor computing involves the use of multiple processors to perform computing. Multitasking computing involves the use of multiple operating system tasks. A task is an operating system concept that refers to the combination of an executing program and bookkeeping information used by the operating system. When a program is executed, the operating system creates a new task for that program. A task is like a program's envelope, identifying the program with a task number and attaching other bookkeeping information. Many operating systems are called multitasking operating systems because they can execute many tasks simultaneously. Multitasking refers to the operating system's ability to run multiple executable programs simultaneously. Each executable program runs in its own address space; that is, these executable programs have no way to share their own memory. This has advantages, as neither program can harm the execution of other programs running on the system. However, programs have no way to exchange information other than through the operating system (or by reading files stored in the file system). Multiprocess computing is similar to multitasking computing, and the terms "task" and "process" are often used interchangeably, although some operating systems distinguish between the two.

[0059] FIG. 4 is a functional block diagram illustrating a system 400 according to an embodiment of the present invention. The system 400 may include multiple clients (e.g., clients 410A, 410B, and 410C), multiple border NATs (e.g., border NAT 430), multiple load balancers (e.g., load balancer 450), and multiple backends (e.g., backend nodes 470A and 470B), all interconnected via network 310. In the example illustrated in FIG. 4, the client 410 may belong to a particular network, while the backend node 470 may belong to a different network. To distinguish between them, the two networks may be referred to as a “source” network and a “destination” network. Furthermore, while the example of FIG. 4 illustrates two networks, other embodiments may include three or more networks. Notably, the border NAT 430 and the load balancer 450 may reside in either the source network or the destination network, although this is not required. That is, border NAT 430 need not reside on the same network as client 410, and load balancer 450 need not reside on the same network as backend node 470. The connection tracking device, e.g., border NAT 430, may reside anywhere on the path between the source (client 410) and the backend device. In various embodiments, the connection tracking device may be on the border of the source network or on the border of the network that includes the backend device.

[0060] In various embodiments, network 310 may represent, for example, an intranet, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof. Network 310 may include, for example, wired, wireless, or fiber optic connections. In general, network 310 may be any combination of connections and protocols that support communication between clients 410A-C, boundary NAT 430, load balancer 450, and backend nodes 470A-B in accordance with embodiments of the present invention.

[0061] Clients 410A-C may represent networked computing devices capable of hosting client applications. For example, clients 410A-C and boundary NAT 430 may be included in a private network. Clients 410A-C may include client program 320. In an exemplary embodiment, client 410 may be a desktop computer, a notebook, a laptop computer, a tablet computer, a handheld device, a smartphone, a thin client, or any other electronic device or computing system capable of sending and receiving data to and from other computing devices, such as load balancer 450 or backend 470, over network 310 through boundary NAT 430. As illustrated in the example of FIG. 5, client 410 may be implemented using multiple devices (referred to as client 410A, client 410B, and client 410C, respectively).

[0062] In various embodiments, client program 320 may be a program, or a subroutine included in a program, that enables a user of client 410 to request and utilize the services of backend nodes 470A-B through boundary NAT 430, network 310, and load balancer 450. Client program 320 may include components used to send data to applications residing on boundary NAT 430. In an exemplary embodiment, client program 320 may send TCP data packets to boundary NAT 430 as part of a data flow. Data packets sent by clients 410A-C to boundary NAT 430 include a destination address. The destination address may be an internal address.

[0063] Border NAT 430 may represent a network component or device capable of hosting an address translation program. Furthermore, border NAT 430 may represent a network component or device capable of tracking connections, and thus may be referred to herein as a "connection tracking" device. Border NAT program 324 may perform network address translation (NAT) and connection tracking functions. Border NAT 430 may allow data to flow from one network to another, e.g., from a network including multiple clients 410A-C to network 310. Border NAT 430 separates attributes between domains, e.g., between an internal network (e.g., an internal network including clients 410A-C) and an external network. Border NAT 430 tracks packet connections in a data flow between a particular network client and a particular backend network service node. Border NAT 430 may receive data packets from clients 410A-C, load balancer 450, backend nodes 470A-B, or other sources. Each packet received by border NAT 430 includes a destination address. Border NAT 430 may replace a destination address with a different address. For example, in a data packet sent by client 410A-C to border NAT 430, border NAT 430 may replace an internal address with an external address. In addition to modifying the network address, border NAT 430 may also modify the port number. A packet sent by client 410A-C to border NAT 430 may be one of multiple related data packets that form a data flow from client 410A-C.

[0064] Load balancer 450 may represent a network component or device capable of making packet routing decisions that maintain client connection affinity. Load balancer 450 may host load balancing program 326. In an exemplary embodiment, load balancer 450 may be a desktop computer, a networked computer appliance, a server blade, a dedicated computer server, or any other electronic device or computer system capable of hosting load balancing program 326. As illustrated in the example of FIG. 5, load balancer 450 may be implemented using multiple devices (referred to as load balancer 450A, load balancer 450B, etc.) (distributed load balancing). Load balancer 450 is an example of a packet forwarding device.

[0065] In various embodiments, the load balancing program 326 may be a program or subroutine within a program that performs the function of balancing and forwarding network client requests to backend network service nodes over the network. The load balancing program 326 may make packet forwarding decisions according to an algorithm designed to maintain client connection affinity and balance workload. The load balancing program 326 may maintain client connection affinity to the extent feasible. In an exemplary embodiment, the load balancing program 326 may receive data packets from the client program 320 over the network 310 and forward the received data packets to the backend program 322. The received data packet may be one of multiple related data packets forming a data flow from the clients 410A-C.

[0066] Backend nodes 470A-B may represent computing platforms capable of hosting one or more resources utilized by other computing devices, such as clients 410A-C. Backend nodes 470A-B may include backend program 322. In an exemplary embodiment, backend nodes 470A-B may be desktop computers, networked computing appliances, server blades, dedicated computer servers, or any other electronic device or computing system capable of sending and receiving data to and from other computing devices, such as clients 410A-C, boundary NAT 430, and load balancer 450, over network 310. Backend nodes 470A-B may include a cluster of servers running the same software to collectively process and monitor requests for other computing devices, such as clients 410A-C. Backend nodes 470A-B may be implemented using multiple devices (referred to as backend node 470A, backend node 470B, backend node 470C, and backend node 470D).

[0067] In various embodiments, backend program 322 can be a program, or a subroutine included in a program, that can facilitate processing of data requests received from client program 320 via load balancing program 326. In an exemplary embodiment, backend program 322 can receive data packets from load balancing program 326. The received data packets can originate from client program 320 and represent a portion of the data flow from clients 410A-C. Backend program 322 can process the received data packets.

[0068] Figure 5 is a functional block diagram illustrating an example networking environment 500. Networking environment 500 may include multiple clients 410A-F, as described above with reference to Figure 4, multiple border NATs 430A-B, as described above with reference to Figure 4, multiple load balancers 450A-N, as described above with reference to Figure 4, and multiple backend nodes 470A-N, as described above with reference to Figure 4. Networking environment 500 also illustrates an example of data flow from clients 410A-F to border NAT 430A for private network 420 or border NAT 430B for private network 440. Private network 420 may include clients 410A-C and border NAT 430A, and private network 440 may include clients 410D-F and border NAT 430B. Data flows are forwarded from border NATs 430A-B to load balancers 450A-N, and from load balancers 450A-N to backend nodes 470A-N, in accordance with various embodiments of the present invention. The data flow forwarding described herein can be implemented by system 400 of FIG.

[0069] Each load balancer node 450A-N can spread traffic to other devices, such as backend nodes 470A-N. These other devices may generally include backend web servers or other servers, storage, or computing devices. The number of load balancers may vary. In this example, there are initially two load balancers 450A-B, but the number of load balancers may change to N, resulting in load balancers 450A-N. Similarly, the number of backend nodes may vary. In this example, there are initially four backend nodes 470A-D, but the number of backend nodes may change to N, resulting in backend pool nodes 470A-N.

[0070] In one embodiment, load balancer 450A may forward data (flow 502) sent from client 410A through border NAT 430A to backend node 470A. Furthermore, flow 504 sent from client 410C and forwarded by border NAT 430A may be forwarded to backend 470D by both load balancer 450A and load balancer 450B. Furthermore, in one embodiment, load balancer 450N may forward data (flow 506) sent from client 410D through border NAT 430B to backend 470B. Furthermore, flow 508 sent from client 410F and forwarded by border NAT 430B may be forwarded to backend 470C by both load balancer 450A and load balancer 450B.

[0071] 5, network switches, routers, and other devices may be present in networking environment 500. For example, a network switch may be present between load balancer nodes 450A-B and backend nodes 470A-D to achieve the desired network connectivity.

[0072] FIG. 6A is a functional block diagram of a known TCP / IP networking process, shown from the perspective of a network packet. A client 610 may reside on a private network within a home or on a virtual private network (VPN) connection to a corporate network, and wishes to communicate with one or more backends 640 of web services over the public Internet. The example shown in FIG. 6A corresponds to a virtual private endpoint (VPE) or similar service, which can form a VPN tunnel connection between the source and destination by adding an appropriate header to the network packet. The network packet 612 sent by the client 610 to the border NAT 620 includes the client's IP address as the source address, the selected source port, the IP address of the border NAT 620's private network interface as the destination address, and the destination port assigned to the selected Internet protocol (e.g., port number 80 for the Hypertext Transfer Protocol (HTTP)). In this example, the border NAT 620 receives the packet 612 and performs network address translation (NAT) processing to allow the data flow to continue on another network. However, the present invention does not require the connection tracking device to perform NAT processing. In this example, packet 612 is converted into modified packet 622 by NAT processing, but as described above, in this case, the boundary NAT must perform connection tracking on all packets it forwards. In another example, packet 612 is converted into modified packet 622 by a process separate from any NAT processing. This modified packet 622 includes the IP address of boundary NAT 620 as its source IP address, a newly selected source port, the IP address of load balancer 630 as its destination IP address, and the same destination port as packet 612. Packet 622 is forwarded to load balancer 630, which determines a backend 640 that has the service requested by packet 622.In this example, load balancer 630 modifies packet 622 by changing the destination IP address to the address of backend 640, resulting in packet 632. Load balancer 630 forwards packet 632 to backend 640. Backend 640 receives packet 632 and can respond directly to client 610 via boundary NAT 620. In this case, backend 640 uses its own address and port as the source information and the boundary NAT information as the destination address and port.

[0073] FIG. 6B is a functional block diagram of one embodiment of the present invention, also shown from the perspective of a network packet. Packets 652, 662, and 672 may include data (not shown) and header information including a source network address (“SRC”) and a destination network address (“DST”). Packets 652, 662, and 672 may include source and destination port number fields. Packets 652, 662, and 672 may contain the same data, but may each include different source and destination network address information and different data in the source and destination port number fields. For example, packet 652 includes a destination network address of “10.240.10.10” and a port number of “80.” The network addresses of client 410, boundary NAT 430, load balancer 450, and backend 470 are shown in FIG. 6B. For example, the address of load balancer 450 is “161.26.0.30.” It can be seen that the destination network address of packet 652 is the same as the network address of boundary NAT 430. Client 410 can forward packet 652 to boundary NAT 430, as described above with reference to FIG.

[0074] In some embodiments, the border NAT 430 may perform NAT in response to receiving packet 652. Additionally, according to various embodiments, the border NAT 430 may modify the destination port to a version identifier. In this example, the version identifier "81" is encoded in packet 662. The version identifier "81" replaces the destination port number "80" in packet 652. The border NAT 430 forwards the modified packet 662 to the load balancer 450, which determines the version from packet 662. The load balancer 450 executes an appropriate hash function depending on the version, as described below. The load balancer 450 then selects a destination backend 470 using the extracted version identifier and hash function, as described below. The load balancer 450 then restores the destination port to "80," the port number before modification by the border NAT 430. Load balancer 450 forwards packet 672 to the selected backend 470. Backend 470 receives packet 672 and can respond directly to client 410 via boundary NAT 430. In this case, backend 470 uses its own address and port as the source information and the boundary NAT information as the destination address and port.

[0075] 7A is a flowchart illustrating a process 700 for tagging network packets with a version number. Process 700 may be performed by a connection tracking network device, such as a border NAT, a gateway, a virtual private endpoint (VPE), a virtual private network (VPN) ingress server, a software-defined network (SDN) egress device, or an SDN node-based NAT.

[0076] At step 702, a connection tracking device receives a data packet from a client computing device (e.g., client 410). The packet may be received over a TCP connection. In one embodiment, the connection tracking device is configured to perform NAT. In one embodiment, the source address and port in the packet identify the device in the internal network (e.g., private network) to which they belong. The device can use NAT to change the private source address and port to an address in a public address space (e.g., another network).

[0077] In step 704, a version identifier for the packet may be determined. In one embodiment, a connection tracking device, e.g., border NAT 430, maintains the version and its corresponding version identifier in memory. How the version identifier is determined in step 704 depends on whether the received packet is the first or subsequent packet in a sequence of packets for which connection affinity is maintained. If the received packet is the first packet in the sequence, the version identifier is established or set. If the received packet is a subsequent packet in the sequence, the version identifier is determined by identifying the packet as belonging to an existing (tracked) connection and fetching the version identifier for that sequence from memory. According to various embodiments, the version information may change. In some embodiments, the current version is updated based on a time or quantity condition. For example, the current version may be incremented when a certain number of packets have been received or transmitted by the network device, e.g., every 10,000 packets. As another example, the current version may be incremented after a certain period of time has elapsed, e.g., every 5 seconds. In other embodiments, the current version may be updated when a message indicating a change in the number of operational devices in the network is received. Networks are dynamic. For example, the number of gateways, load balancers, endpoints, or other network devices may change. In some embodiments, each particular network configuration may correspond to a version. In other embodiments, a policy change, rather than a change in backend capacity, may correspond to a version. Thus, the version identifier may change in response to receiving a message. The message may be a control message from the SDN. The message may be from the load balancer 450, such as an Internet Control Message Protocol (ICMP) message or other out-of-band message.In some embodiments, the current state or version fetch may be via a DNS lookup of the service or using a fake SYN. ​​In yet another example, load balancer 450 may encode the current version as the source port in the return packet.

[0078] At step 706, a version identifier may be encoded into the packet. The current version number may be inserted into any appropriate field of the received network packet. In some embodiments, the version identifier is encoded into the destination port field. At this stage, the version identifier encoded into the packet is saved in the boundary NAT's memory so that it can be used by subsequent packets in the packet sequence, i.e., in the same data flow.

[0079] At step 708, the packet containing the encoded version identifier is sent to a transport network device (eg, a load balancer device).

[0080] 7B is a flowchart illustrating a process 720 for packet processing by a load balancer according to various embodiments of the present invention. Process 720 may be performed by any suitable network device (e.g., load balancer 450) having forwarding functionality. At step 722, the forwarding device receives a data packet. For example, the packet may be received from boundary NAT 430. The packet may be received via a TCP connection. The receiving device and at least one backend device may be included in the same network.

[0081] At step 724, a version identifier may be extracted from the received packet. In one embodiment, the version identifier may be encoded in the destination port field. In other embodiments, the version identifier may be encoded in any other suitable portion of the packet. Additionally, the source IP address / port number, the destination IP address / port number, and the network protocol in use may be extracted from the packet.

[0082] In step 726, a destination (i.e., a backend server) to which the packet is routed is selected. For example, a specific hash function may be used to determine a hash value using the extracted source address and source port number as inputs to the hash function. At the same time, a table of backend server addresses may be selected based on the version identifier extracted from the packet. This table stores a list of available backends for a specific version. The hash value is divided by a specific number of backend devices currently operational in the backend device network (determined by the row number in the selected table). The remainder of the division corresponds to the row number in the selection table to be used. The network address of the backend device in this table row is selected as the destination for the packet. A more specific example of this technique is described below in FIG. 8. When the load balancer 450 experiences a state change, it can check the version in a specific field for each flow it is processing and update its forwarding rules accordingly.

[0083] In step 728, the selected network address is encoded as a destination address in the first packet. Additionally, a source port of the originating node may be encoded as a source port in the first packet, replacing the version identifier in the packet with the source port of the originating node. The first packet containing the selected network address is transmitted to a backend device having the selected network address.

[0084] 8 is a block diagram illustrating packet routing by a load balancer (or forwarding device) according to various embodiments. Load balancer 450 receives packet 862 from NAT device 430. In one embodiment, client 410 and NAT device 430 may be part of a first network. In another embodiment, NAT device 430, load balancer 450, and backends 470A, 470B may be part of a second network.

[0085] The load balancer 450 extracts the version identifier of the packet 862 from the packet 862. In this example, the load balancer 450 extracts the version identifier "81" from the destination port field of the packet 862.

[0086] Furthermore, load balancer 450 extracts the source address and source port from packet 862. In this example, load balancer 450 extracts "165.20.10.10" as the source address and "35017" as the source port. Load balancer 450 uses the extracted source address and source port as input to determine a hash value using a specific hash function selected based on the version identifier. This specific hash function may be a currently known hash function or a hash function developed in the future.

[0087] The hash function may be either a hash function that produces a remainder or a hash function that does not produce a remainder. For both of these examples, in FIG. 8, it is assumed that the load balancer 450 determines the hash value to be “71.” The load balancer 450 determines the remainder by dividing the hash value “71” by a specific number of operational backend devices (determined from the latest version identifier at the time the connection was initiated). While only two backend devices are shown in FIG. 8 (for simplicity), any number of backend devices may be operational, even if not shown. In the first example using a hash function that produces a remainder and with a version identifier of “81,” it is assumed that three backend devices are currently operational. Dividing 71 by 3 leaves a remainder of “2” (71=23×3+2). The load balancer 450 uses this remainder “2” as an index to a row in table 804. In this example, the remainder “2” points to row number 2. Line number 2 contains the backend server address "10.131.10.3".

[0088] If the hash function used produces a remainder, the load balancer 450 selects the network address of the backend device from the table row corresponding to the remainder for the relevant version. If the hash function does not produce a remainder, the load balancer 450 may select a row from the table using the result produced by the hash function. Continuing with the example where the hash function produced hash value "71," the load balancer 450 selects backend device 470A having address "10.131.10.3" from row number "2" for version identifier "81."

[0089] In a second example shown in FIG. 8 with version identifier "82," assume that four backend devices are currently operational. Further assume that the load balancer 450 extracts the version identifier "82" from packet 862. When the hash value "71" is divided by 4, the remainder is "3" (71 = 17 × 4 + 3). The load balancer 450 uses this remainder "3" as an index to a row in table 804. In the second example, the remainder "3" points to row number 3, which contains the backend server address "10.131.10.4." In the second example, the load balancer 450 selects the backend device 470B from row number 3 that has the address "10.131.10.4" for the version identifier "82."

[0090] Load balancer 450 encodes the selected network address as the destination address in the packet and sends the packet containing the selected network address to the backend device having that address. Continuing with the above example, if the selected address is "10.131.10.3," packet 872 is sent to backend device 470A. If the selected address is "10.131.10.4," packet 874 is sent to backend device 470B.

[0091] The load balancer 450 may encode the source port number of the originating node (client 410) as the source port in packets 872 and 874. The NAT device replaces the source port number of the originating node included in packet 862 with a version identifier. By encoding the source port number in packets 872 and 874, the packets are restored to the format expected by backend devices 470A and 470B. The backend devices 470A and 470B can use the source port number of the originating node in packet 862 when processing the packet, for example, when generating a response.

[0092] In some embodiments, the particular hash function used to determine the hash value is selected from two or more hash functions based on the version identifier of the first packet, and in some embodiments, consistent hashing may be used.

[0093] In some embodiments, the particular number of backend devices currently operational in the second network is selected from two or more numbers of backend devices based on the version identifier of the first packet.

[0094] When multiple load balancers 450 are employed to perform the load balancing function, each instance of the load balancer must be aware of the set of back-end devices corresponding to each version. Thus, according to various embodiments, the set of back-end devices corresponding to a particular version is distributed to each instance of the load balancer.

[0095] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0096] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0097] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computer / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computer / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computer / processing device for storage.

[0098] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0099] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer-readable program instructions.

[0100] The computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, form means for performing the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams. The computer-readable program instructions may also be stored on a computer-readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer-readable storage medium having instructions stored thereon constitutes an article of manufacture including instructions for performing aspects of the functions / acts identified in one or more blocks of the flowcharts and / or block diagrams.

[0101] Computer-readable program instructions may also be loaded into a computer, other programmable device, or other device and a series of operational steps executed on the computer, other programmable device, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable device, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0102] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing a specific logical function. In some implementations, the functions depicted in the blocks may be performed in an order different from that depicted in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, may be executed simultaneously or substantially simultaneously, may be executed in a partially or fully overlapping manner, or may even be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs a specific function or operation, or by a combination of dedicated hardware and computer instructions.

[0103] While various embodiments of the present invention have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements to commercially recognized technologies of the embodiments, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method comprising: receiving, at a connection tracking device disposed between a first network and a second network, a packet from a node in the first network, the packet being part of a data flow; determining, by the connection tracking device, a version identifier, the version identifier being information identifying a version of a network configuration of the second network at the time the data flow is initiated; encoding, by the connection tracking device, the version identifier in the packet; sending, by the connection tracking device, the packet including the encoded version identifier to a forwarding device, wherein the forwarding device selects a backend device based on the encoded version identifier; transmitting, by the forwarding device, the packet to the back-end device, the back-end device being in the second network; 11. A computer-implemented method comprising:

2. encoding the version identifier in the packet includes encoding the version identifier in a destination port field of the packet. The computer-implemented method of claim 1 .

3. encoding the version identifier in the packet includes encoding the version identifier in a field in a packet header. The computer-implemented method of claim 1 .

4. the connection tracking device is a boundary network address translation (NAT) device; The computer-implemented method of claim 1 .

5. the version identifier corresponds to a set of back-end devices in the second network; The computer-implemented method of claim 1 .

6. The step of determining the version identifier, comprising: determining, by the connection tracking device, a period during which the current version has been in use; updating the version identifier if the determined period exceeds a period designated for updating the version identifier. The computer-implemented method of claim 1 .

7. The step of determining the version identifier, comprising: determining, by the connection tracking device, a quantity of packets received at the connection tracking device that use a particular version identifier; updating the version identifier if the determined number of packets exceeds a specific number of packets designated for updating the version identifier. The computer-implemented method of claim 1 .

8. The method of claim 7, wherein determining the version identifier comprises: receiving, by the connection tracking device, a control message; updating the version identifier in response to the control message. The computer-implemented method of claim 1 .

9. receiving a return packet at the connection tracking device, the return packet being sent to the connection tracking device in response to the packet including the encoded version identifier being sent to the forwarding device, the return packet having the version identifier encoded therein; updating the version identifier in response to the return packet. The computer-implemented method of claim 1 .

10. The version identifier corresponds to a hash function used by the forwarding device or a policy adopted by the forwarding device. The computer-implemented method of claim 1 .

11. one or more processors; a memory coupled to at least one of the processors; and computer program instructions stored in said memory and operable to be executed by at least one of said processors, said computer program instructions comprising: receiving, at the computer system disposed between a first network and a second network, a packet from a node in the first network, the packet being part of a data flow; determining, by the computer system, a version identifier, the version identifier being information identifying a version of a network configuration of the second network at the time the data flow is initiated; encoding, by the computer system, the version identifier within the packet; transmitting, by the computer system, the packet including the encoded version identifier to a load balancer device in the second network; A computer system that performs operations including:

12. encoding the version identifier in the packet includes encoding the version identifier in a destination port field of the packet.

12. The computer system of claim 11.

13. the computer system is a boundary network address translation (NAT) device; 12. The computer system of claim 11.

14. the version identifier corresponds to a set of backend devices currently operable in the second network; 12. The computer system of claim 11.

15. The step of determining the version identifier, comprising: determining, by the computer system, the length of time that the current version has been in use; If the determined period exceeds a period set for updating the version identifier, updating the version identifier.

12. The computer system of claim 11.

16. The step of determining the version identifier, comprising: determining, by the computer system, a number of packets received at the computer system that use the current version identifier; updating the version identifier if the determined number of packets exceeds a specific number of packets set for updating the version identifier.

12. The computer system of claim 11.

17. The step of determining the version identifier, comprising: receiving, by the computer system, a control message from a node in the second network, the second network including at least one back-end device; updating the version identifier in response to the control message.

12. The computer system of claim 11.

18. A computer-implemented method comprising: receiving, at a connection tracking device, a packet from a node in a first network; determining, by the connection tracking device, a version identifier of the packet; encoding, by the connection tracking device, the version identifier in the packet; sending, by the connection tracking device, the packet including the encoded version identifier to a forwarding device, wherein the forwarding device selects a backend device based on the affinity of the client connection; transmitting, by the forwarding device, the packet to the back-end device, the back-end device being in a second network; Including, The current version is based on a quantity of packets, and determining the version identifier includes: determining a quantity of packets received at the connection tracking device that use a particular version identifier; updating the version identifier if the determined number of packets exceeds a specific number of packets designated for updating the version identifier. Computer-implemented methods.

19. One or more processors; a memory coupled to at least one of the processors; and computer program instructions stored in said memory and operable to be executed by at least one of said processors, said computer program instructions comprising: receiving, at the computer system, a first packet from a node in a network; determining, by the computer system, a version identifier for the packet; encoding, by the computer system, the version identifier within the packet; transmitting, by the computer system, the packet including the encoded version identifier to a load balancer device in a second network; Performing an action including The version is based on a quantity of packets, and determining the version identifier comprises: determining a number of packets received at the computer system that use the current version identifier; updating the version identifier if the determined number of packets exceeds a specific number of packets set for updating the version identifier. Computer system.

Images