Facilitating distributed snat services

Abstract

Some embodiments of the invention provide novel methods for facilitating distributed SNAT (dSNAT) middlebox service operations of a first network at a host computer in the first network and at a gateway device between the first network and a second network on which middlebox service operations are performed. The novel methods enable dSNAT of stateful middlebox services to be provided at multiple host computers, thus avoiding the bottleneck problem associated with providing stateful middlebox services at gateways, and also significantly reduce the need for redirection of packets received at the wrong host by using the capabilities of off-the-shelf gateway devices to perform IPv6 encapsulation of IPv4 packets and assign locally unique IPv6 addresses to each host executing a dSNAT middlebox service instance used by the performing gateway device.

Description

Background Technology

[0001] Many networks rely on Source Network Address Translation (SNAT) to translate addresses used within the network's address space into globally unique addresses for communication with external networks. Because SNAT is a stateful service, some networks provide it in a centralized location (e.g., an SNAT server). However, providing centralized SNAT can lead to bottlenecks because all traffic using SNAT must traverse the centralized SNAT provider. To address this bottleneck, some networks use distributed SNAT architectures. However, distributed SNAT architectures have their own challenges. For example, because each instance of distributed SNAT performs SNAT operations using the same IP address, in some cases, traffic traversing the provider's gateway device will be forwarded to a randomly selected SNAT instance. For a system with "N" distributed SNAT instances, this will result in (N-1) out of N packets being directed to a distributed SNAT instance that does not store state information about the packet.

[0002] Similarly, stateful load balancing operations on a set of workload compute nodes running on multiple host computers available on a shared Virtual Internet Protocol (VIP) suffer from the same problem. Therefore, for middleware services such as SNAT and stateful load balancing, a solution is needed that addresses both this bottleneck and the misdirection problem. Summary of the Invention

[0003] Some embodiments of the present invention provide novel methods for facilitating distributed middlebox service operations (e.g., distributed SNAT (dSNAT) or distributed load balancing (dLB) middlebox services) on a first network where middlebox service operations are performed on host computers in the first network and at a gateway device between the first and second networks. These novel methods implement distributed middlebox services (e.g., dSNAT or dLB) that provide stateful middlebox services at multiple host computers, thus avoiding the bottleneck problems associated with providing stateful middlebox services at the gateway, and also significantly reducing the need to redirect packets received at incorrect hosts by utilizing the capabilities of off-the-shelf gateway devices to perform IPv6 encapsulation on IPv4 packets and assign locally unique IPv6 addresses to each host performing the distributed middlebox service instance.

[0004] These methods configure the gateway device to receive packets destined for the distributed middlebox service and identify the IP version 6 (IPv6) address based on the port and IPv4 address specified in the destination Internet Protocol version 4 (IPv4) header of the received packet. This identification is then used to forward the received packet to the specific host computer on which the middlebox service instance associated with the destination IPv4 address is running. In some cases, the destination IPv4 address is the IPv4 VIP address associated with the dLB middlebox service, or an IPv4 address used by dSNAT as the source address for packets destined for the external network from the first network. The gateway device then encapsulates the packet with an IPv6 header using the identified IPv6 address and forwards the encapsulated packet based on the gateway device's IPv6 routing table.

[0005] The host computer is configured to receive encapsulated packets destined for an identified IPv6 address from a gateway device and remove the encapsulation to provide the internal IPv4 packets to a middlebox service instance running on the host computer based on the IPv4 address in the header of the internal IPv4 packet. The middlebox service instance performs address substitution operations (e.g., a lookup in a connection tracker that associates the translated IP address and port with the original IP address and port, or replaces the VIP destination address and port with the workload IP address and port) to replace the IP address and port number in the IPv4 header with the IPv4 address and port used by the source machine in the first network. In some embodiments, in the case of dSNAT, the middlebox service instance running on the host computer is assigned a range of port numbers for performing middlebox service operations on packets destined for an external network from the first network; or, in the case of dLB, the middlebox service instance is assigned a range of port numbers for receiving packets.

[0006] In some embodiments, the host computer is configured to advertise the availability of an IPv6 address prefix based on the IPv4 source address of packets used by the middlebox service instance as the source address for packets destined for an external network from a first network, and the range of port numbers assigned to the middlebox service instance. In some embodiments, the advertised IPv6 address prefix begins with 16 bits (e.g., FC00) that are not assigned as globally unique in IPv6, followed by the IPv4 address used by the middlebox service, and then a set of bits from the 16-bit port address that are common to the range of port numbers assigned to the middlebox service instance executing on the host computer (e.g., the first 6 bits are common to the range of 1024 assigned port numbers). In some embodiments, packets destined for an external network and processed by the middlebox service instance executing on the host computer are sent from the host without being encapsulated in an IPv6 header using the advertised IPv6 address as the source IP address. While the IPv6 address associated with the middlebox service instance is not used to encapsulate packets, in some embodiments, other encapsulation may be used to reach the gateway device.

[0007] In some embodiments, the announcement is made by a Border Gateway Protocol (BGP) instance (e.g., a managed forwarding element) running on a host computer. In some embodiments, the announcement is made to a route reflector (e.g., a routing server) that announces the availability of an IPv6 address prefix at the host computer to other network elements, including gateway devices or groups of gateway devices. In some embodiments, the announcement includes instructions to the gateway device to identify an IPv6 address based on the port number and IPv4 address of a packet received at the gateway device and to encapsulate the IPv4 packet with the identified IPv6 address. In other embodiments, the announced IPv6 address is based on the existing capabilities of the hardware gateway device used for processing IPv6 encapsulation of IPv4 packets.

[0008] In some embodiments, a cluster of controller computers in a first network (i.e., a group of one or more controller computers) provides configuration information to network elements to facilitate distributed middlebox service operations of the first network. In some embodiments, this configuration information includes IPv6 routing table entries and a set of middlebox service records provided to gateway devices. The middlebox service records map the combination of IPv4 and destination port numbers used by a particular middlebox service operation to an IPv6 destination address. In some embodiments, the middlebox service records are policy-based routing (PBR) rules that define an algorithm for generating IPv6 addresses from IPv4 destination addresses and port numbers included in the IPv4 header, and specify the IPv4 destination address to which the algorithm should be applied. In some embodiments, each IPv6 routing table entry identifies an IPv6 address prefix associated with a particular host computer in a group of multiple host computers (each of which performs an instance of the middlebox service) and the next hop toward that particular host computer.

[0009] The controller computer cluster configures middlebox service instances to use specific IPv4 addresses when performing middlebox service operations and assigns non-overlapping port number ranges to each middlebox service instance for a specific middlebox service. In some embodiments, the IPv4 address and port number range are provided to the host computer (e.g., the host computer's MFE) to identify the IPv6 address prefix corresponding to the IPv4 address and the assigned port number range and to announce the availability of the identified IPv6 address at the host computer. In some embodiments, the size of the port number range assigned to the middlebox service instance is fixed by the administrator based on the expected maximum number of middlebox service instances (e.g., for a maximum of 64 middlebox service instances, 64 different port number ranges are created, each containing 1024 ports, and each port number range is assigned to one middlebox service instance at startup). In other embodiments, the size of the port number range is dynamic and can change based on the number of active middlebox service instances. The size of the port number range may also differ between middlebox service instances. For example, a larger range of port numbers is assigned to a first middlebox service instance running on a host computer that runs ...

[0010] The foregoing overview is intended as a brief introduction to some embodiments of the invention. It is not intended to be an introduction or overview of all the inventive subjects disclosed in this document. The following detailed description and the accompanying drawings, with reference to the detailed description, will further describe the embodiments described in the summary and other embodiments. Therefore, a full examination of the summary, detailed description, drawings, and claims is necessary to understand all the embodiments described in this document. Furthermore, the claimed subject matter is not limited to the illustrative details in the summary, detailed description, and drawings, but is defined by the appended claims, as the claimed subject matter may be embodied in other specific forms without departing from the spirit of the subject matter. Attached Figure Description

[0011] The novel features of the invention are set forth in the appended claims. However, for illustrative purposes, several embodiments of the invention are illustrated in the following drawings.

[0012] Figure 1 An exemplary network is illustrated, in which a novel method for facilitating the provision of distributed SNAT middlebox services to a first network is executed.

[0013] Figure 2 The concept illustrates the process performed by a gateway device to process received packets destined for a specific middlebox service instance.

[0014] Figure 3A The illustration shows, for example Figure 1 The described gateway device performs packet processing on the first packet in a specific packet stream received from an external network.

[0015] Figure 3B The illustration shows, for example Figure 1 The packet processing described in the diagram at the gateway device for subsequent packets in a specific packet stream, for that specific packet stream, Figure 3A The packet depicted is the first packet received from the external network.

[0016] Figure 4 The illustration shows, for example Figure 1 The gateway device described herein performs packet processing for subsequent packets in a specific packet stream destined for an LB VIP, for which packets were previously received from an external network.

[0017] Figure 5 The concept illustrates the process executed at the host computer for processing received IPv6 packets destined for an instance of an intermediate box service executed on the host computer.

[0018] Figure 6The illustration shows packets being sent from an external machine to a guest machine on the internal network.

[0019] Figure 7 The concept illustrates the process performed by a NAT instance, which processes the first packet in a packet stream destined for a destination machine in an external network.

[0020] Figure 8 The illustration shows the first packet in the packet stream being sent from the GM and processed by a NAT instance running on the same host computer.

[0021] Figure 9 The concept illustrates the process of generating configuration data for different network elements that provide and facilitate middlebox services.

[0022] Figure 10 The illustration shows a cluster of computer controllers in a data center sending different types of configuration data to different network components.

[0023] Figure 11 The concept illustrates the process executed by a gateway device to facilitate the provision of middlebox services based on received configuration data.

[0024] Figure 12 The concept illustrates the process of configuring a host computer to execute a distributed middlebox service instance and announcing the IPv6 address prefix associated with the middlebox service instance executed on the host computer.

[0025] Figure 13 The illustration shows three different advertised exemplary IPv6 address prefixes and corresponding exemplary destination IPv6 addresses. These IPv6 address prefixes are used in different embodiments to advertise the availability of the service at the host computer. The corresponding exemplary destination IPv6 addresses are generated by a gateway device for use in the IPv6 encapsulation header to forward packets to a specific service instance running on the host computer that made the advertisement.

[0026] Figure 14 The concept illustrates a set of data exchanges between network elements for migrating computing nodes.

[0027] Figure 15 A computer system is conceptually illustrated therein for implementing some embodiments of the present invention. Detailed Implementation

[0028] In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the described embodiments and that the invention can be practiced without discussing some specific details and examples.

[0029] Some embodiments of the present invention provide novel methods for facilitating distributed middlebox service operations (e.g., distributed SNAT (dSNAT) or distributed load balancing (dLB) middlebox services) on a first network at a host computer in the first network on which middlebox service operations are performed and at a gateway device between the first and second networks. These novel methods implement distributed middlebox services (e.g., dSNAT or dLB) that provide stateful middlebox services at multiple host computers, thus avoiding the bottleneck problems associated with providing stateful middlebox services at the gateway, and also significantly reducing the need to redirect packets received at incorrect hosts by using the capabilities of readily available gateway devices to perform IPv6 encapsulation on IPv4 packets and assign locally unique IPv6 addresses to each host performing the distributed middlebox service instance. The following discussion focuses on dSNAT and dLB middlebox services; however, those skilled in the art will understand that the methods discussed can be applied to any distributed, stateful middlebox service using the same IP address at multiple hosts.

[0030] This method configures the gateway device to receive packets whose source IPv4 address is used by the dLB service's IPv4 VIP or the dSNAT service as the destination IPv4 address for packets destined for an external network from a first network. It identifies the IPv6 address based on the port specified in the IPv4 header of the received packet and the IPv4 destination address. This IPv6 address is used to forward the received packet to the host computer on which the dSNAT operation is performed. The gateway device then encapsulates the packet with an IPv6 header using the identified IPv6 address and forwards the encapsulated packet based on the gateway device's IPv6 routing table.

[0031] As used in this document, a packet refers to a set of bits in a specific format that is transmitted across a network. In some embodiments, a packet is referred to as a data message. Those skilled in the art will recognize that, as used in this document, the term "packet" refers to a set of bits in various formats that are transmitted across a network. The formatting of these bits can be specified by standardized or non-standardized protocols. Examples of packets that follow standardized protocols include Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Furthermore, as used in this document, references to layers L2, L3, L4, and L7 (or layers 2, 3, 4, and 7) refer to the second data link layer, third network layer, fourth transport layer, and seventh application layer of the OSI (Open Systems Interconnection) layer model, respectively.

[0032] Furthermore, in this example, the term "managed forwarding element (MFE)" refers to software or hardware forwarding elements configured by a cluster of controller computers (i.e., a group of one or more controller computers providing configuration data to network elements) to implement a logical network comprising a set of logical forwarding elements (LFEs). In some embodiments, each LFE is a distributed forwarding element implemented by configuring multiple MFEs on multiple host computers. For this purpose, in some embodiments, each MFE or module associated with an MFE is configured to encapsulate the LFE's data message using an overlay network header containing a Virtual Network Identifier (VNI) associated with the overlay network. Accordingly, in the following discussion, the LFE is referred to as an overlay network construct spanning multiple host computers.

[0033] In some embodiments, the LFE also spans a configured hardware forwarding element (e.g., a top-of-rack switch). In some embodiments, the group of LFEs includes logical switches implemented by configuring multiple software switches or associated modules on multiple host computers. In other embodiments, the LFE can be other types of forwarding elements (e.g., logical routers), or any combination of forwarding elements (e.g., logical switches and / or logical routers) that form a logical network or a part thereof. Numerous examples of LFEs, logical switches, logical routers, and logical networks exist, including those provided by VMware's NSX networking and services virtualization platform.

[0034] Figure 1 An exemplary network 100 is illustrated, in which a novel method for facilitating the provision of distributed SNAT operations for a first network is executed. Figure 1The diagram illustrates a network 100 including an external network 101 that connects a group of machines 102 outside a data center 105 to guest machines (GMs) 126 running on multiple host computers 120 within the data center 105 (e.g., a public cloud data center or a group of data centers). A gateway device 110 operates between the internal network and the external network 101 and is configured as follows regarding... Figure 11 As discussed, this facilitates the provision of distributed middlebox services for communication between external machine 102 and internal computing nodes (e.g., GM 126). Gateway device 110 includes a cache 111 and a pre-routing rule set 112. The cache 111 stores information about forwarding decisions made for previously received packets. In some embodiments, the pre-routing rule set includes policy-based routing rules based on IP (IPv4 or IPv6) addresses. After a lookup is performed in the cache 111 and in the pre-routing rule set 112, the packet can be routed using an IPv4 routing table or an IPv6 routing table, or it can be sent to an IPv6 encapsulator to be encapsulated in an IPv6 header and then routed based on an IPv6 routing table.

[0035] System 100 also includes a group of controller computers 140 that provide configuration information to the group of host computers 120 to implement a set of logical forwarding elements (e.g., using MFE 121), an IPv6 processing module 122 that processes received IPv6 packets, a routing machine (RM) 123 that acts as a BGP instance to interact with a group of routing servers 130, a distributed middlebox service instance 124 (e.g., an SNAT instance 124 providing dSNAT middlebox service in the illustrated embodiment), and a set of compute nodes (e.g., GM 126). Host computers 120 also include a set of NAT records 125, which in some embodiments are caches recording the association between internal IP address / port pairs and the ports selected by the SNAT instance 124 as the external ports for each IP address / port pair. In some embodiments, these cached records also store the IP addresses of external machines along with the ports selected by the SNAT instance 124 to allow the SNAT instance 124 to use the same port for multiple data message flows established with different external machines using different IP addresses.

[0036] System 100 also includes a group of routing servers 130 (also referred to as route reflectors), each receiving routing information from multiple network elements and providing that routing information to other network elements to simplify the exchange of routing information. For example, instead of using a full mesh connecting each BGP instance (e.g., RM 123), each BGP instance interacts with routing server 130 to reduce the number of announcements made by each BGP instance. Data center 105 also includes an intervening fabric 150, which provides physical connectivity between the network elements shown.

[0037] Figure 1 The functions of various components will be about Figures 2-4 Let's discuss this in more detail. Figure 2 A conceptual illustration depicts a process 200 performed by a gateway device (e.g., gateway device 110) to process received packets destined for a specific middlebox service instance (e.g., SNAT instance 124a). In some embodiments, the gateway device is an off-the-shelf gateway device with the ability to encapsulate IPv4 packets within IPv6 packets and generate IPv6 headers for the IPv6 packets using a set of rules or records specifying the IPv6 headers to use based on IPv4 header values. In other embodiments, the gateway device is a fully programmable forwarding element programmed to perform the operations of process 200. Those skilled in the art will understand that process 200 can be performed for packets destined for an IPv4 address associated with any of the following: a dSNAT middlebox service, a dLB middlebox service using a set of load-balanced workload compute nodes, or any distributed middlebox service using the same IP address across multiple middlebox service instances running on multiple host computers.

[0038] Process 200 begins by receiving (at 210) an IPv4 packet destined for a specific middlebox service instance (e.g., SNAT instance 124a) running on a particular host computer (e.g., host computer 120a). This IPv4 packet has a destination address associated with the middlebox service (i.e., all middlebox service instances). The IPv4 header identifies the source machine in the external network by its source IP address and identifies the destination IP address and port, which are the source IP addresses and ports used by the SNAT instance that processes packets for communication sessions (also referred to as packet flows or connections) between the internal compute nodes served by this middlebox service instance and the external machines.

[0039] The process then determines (at 220) that IPv4 packets need to be encapsulated in IPv6 packets. In some embodiments, this determination of the first packet in a specific packet stream received from an external machine is based on a middlebox service record. In some embodiments, the middlebox service record is an IPv4 routing table entry for an IPv4 destination address that indicates that the packet needs to be encapsulated in an IPv6 packet. In some embodiments, the IPv4 routing entry also indicates (1) a specific IPv6 destination address for encapsulating the IPv4 packet based on IPv4 header values ​​(e.g., IPv4 destination address and destination port (for SNAT) or source port (for LB)), or (2) an algorithm for generating the IPv6 destination address for use in the IPv6 encapsulation header. In some embodiments, the middlebox service record is a policy-based routing (PBR) rule that specifies the encapsulation of all packets destined for the IPv4 address associated with the middlebox service. Like the IPv4 routing entry, the PBR rule may specify the IPv6 destination address or a method for generating the IPv6 destination address. In some embodiments, the PBR rule is included in the pre-routing rule 112. In some embodiments, after a first packet is received from a specific packet flow from an external machine, the determination of subsequent packets for that specific packet flow is based on a cache record that includes the determination made for the first packet of that specific flow and is stored in cache 111.

[0040] After determining (at 220) that the IPv4 packet needs to be encapsulated within an IPv6 packet, process 200 identifies (at 230) the IPv6 destination address to be used in the IPv6 encapsulation header based on the IPv4 header value (e.g., the destination IPv4 address and destination port (for dSNAT) or source port (for dLB)). In some embodiments, this identification is based on the IPv6 destination address specified for the IPv4 destination address and port pair in the intermediate box service record. In other embodiments, this identification is based on an algorithm or other programming method used to generate the IPv6 destination address from the IPv4 header. In some embodiments, the algorithm takes an IPv6 prefix for a locally unique address (e.g., FC00:: / 8) and appends the IPv4 destination address and destination (or source) port, followed by zeros (zeros are used for simplicity, but in some embodiments, any trailing bit group will work).

[0041] After identifying the IPv6 destination address (at 230), the IPv4 packet is encapsulated with an IPv6 header (at 240) that uses the identified IPv6 address as the destination address. In some embodiments, this encapsulation is performed by an IPv6 encapsulation module (e.g., IPv6 encapsulator 114). In some embodiments, the destination port is an IPv4 destination port, but in other embodiments, it can be any port. In some embodiments, the source IPv6 address and port are the IPv6 address and port associated with the gateway device. Process 200 then performs a lookup in the IPv6 routing table (e.g., IPv6 routing table 115) (at 250) to identify the next hop for the encapsulated data message. As will be discussed in more detail below, for each of a set of IPv6 address prefixes associated with a set of host computers performing the middlebox service instance, the gateway device is configured with a set of IPv6 routes (i.e., routing table entries) that indicate the next hop for each IPv6 address prefix. In some embodiments, the IPv6 address prefix is ​​provided by the controller computer (e.g., as a static route). In other embodiments, the host computer (e.g., a BGP instance running on the host computer) advertises the IPv6 address prefix as available at the host computer. As will be discussed in more detail below, each intermediate box service instance is assigned a port number range, which can be used to generate a unique IPv6 address prefix based on the algorithm described above for generating IPv6 destination addresses.

[0042] Finally, process 200 forwards the IPv6 packet (at 260) to the identified next hop toward the intermediate box service instance, and the process ends. In some embodiments, the packet is forwarded through an intermediate network structure (e.g., intermediate architecture 150). In some embodiments, the forwarding elements in this intermediate architecture learn the next hop for the destination IPv6 address using standard protocols (such as any or all of BGP, IGP, ARP) or any other routing learning mechanism known to those skilled in the art.

[0043] Figure 3A , Figure 3B and Figure 4 The illustration shows an example of packet processing for the first and subsequent packets in a specific stream used for different distributed middlebox services. Figure 3A The illustration shows, for example Figure 1The depicted gateway device 110 performs packet processing on the first packet 320a from a specific packet stream received from external network 101. As shown, packet 320a is received from machine 102 in external network 101 with a NAT IP address and port N (i.e., the port selected as the source port by SNAT) as the destination. The packet is first compared with cache entries in cache 111 (which does not yet include cache record 111a). Since no cache record is found, a lookup is performed in pre-routing rule set 112, and in the illustrated embodiment, pre-routing rule 112a is identified as being applied to the packet based on the destination IP (i.e., the NAT IP). Based on the identification that pre-routing rule 112a is applied to the packet, cache record 111a is created in cache 111. In the illustrated embodiment, cache record 111a identifies the lookup result in the pre-routing rules; however, those skilled in the art will understand that in other embodiments, the cache record identifies the specific IPv6 address or IPv6 encapsulation address and next hop used to encapsulate the packet in the stream based on the rule.

[0044] Based on pre-routing rule 112a identified based on the destination IPv4 address (i.e., the IPv4 address (NAT IP) associated with the dSNAT service), the packet is passed to IPv6 packet encapsulator 114 for encapsulation, instead of being passed to IPv4 routing table 113 for forwarding based on IPv4 header values. In the illustrated embodiment, as part of passing the packet, pre-routing rule 112a identifies to IPv6 encapsulator 114 encapsulation rules stored by the IPv6 encapsulator for performing the encapsulation. In other embodiments, passing the packet to IPv6 encapsulator 114 includes sending rules (e.g., algorithms) for generating encapsulation header values. IPv6 encapsulator 114 then encapsulates the packet with an IPv6 header based on the identified rule (i.e., rule 114a).

[0045] In the illustrated embodiment, the destination IPv6 address is generated using an FC00 prefix for locally unique addresses, followed by the destination IP (i.e., the NAT IP), then the destination port (port N), and finally 0. Other prefixes or padding bits are used in other embodiments, and those skilled in the art will recognize that this is simply one of many possible algorithms chosen for simplicity in generating IPv6 destination addresses. In some embodiments, the IPv6 destination port, source IPv6 address, and IPv6 source port are the original destination port, the source IPv6 address associated with gateway device 110, and a randomly selected IPv6 source port. However, those skilled in the art will understand that other destination ports and source IPv6 addresses and ports are used in other embodiments without affecting the routing of packets to specific hosts and dSNAT instances.

[0046] After the packet is encapsulated in an IPv6 header including the generated IPv6 address, it is passed to the IPv6 forwarding operation represented by IPv6 routing table 115. A lookup in IPv6 routing table 115 identifies a matching routing table entry 115a, which includes the next-hop (i.e., next-hop 3) interface for forwarding the packet to its destination. In some embodiments, the IPv6 routing table entries for NAT IP (and LB IP) are dynamic routes learned based on advertisements from BGP instances on the host computer, while in other embodiments, the routing table entries are static routes received from controller computers in a group of controller computers (e.g., controller computer cluster 140). The IPv6 packet 330a is then forwarded to the destination host based on the IPv6 destination address (i.e., FC00:NAT IP:PortN::).

[0047] Figure 3B The illustration shows, for example Figure 1 The depicted gateway device 110 performs packet processing on subsequent packets 320b within a specific packet flow (for which packet 320a is the first packet received from external network 101). As shown, packet 320b is received from machine 102 in external network 101, with a NAT IP address and port N (i.e., the port selected as the source port by SNAT) as its destination. The packet is first compared with a cache entry in cache 111 (which contains cache record 111a based on the first packet 320a received from external network 101). Cache record 111a indicates that packet 320b should be encapsulated in an IPv6 header according to a specific encapsulation rule (i.e., "IPv6 Encapsulation Rule 1"). In some embodiments, the packet and rule identifier are passed to IPv6 encapsulator 114, bypassing pre-routing rule 112. However, those skilled in the art will understand that in other embodiments, the cache record identifies the specific IPv6 address or IPv6 encapsulation address and next hop used to encapsulate the packet in the flow based on this rule. IPv6 encapsulator 114 then encapsulates the packet with an IPv6 header based on the identified rule (i.e., rule 114a).

[0048] In the illustrated embodiment, the destination IPv6 address is generated using an FC00 prefix for locally unique addresses, followed by the destination IP (i.e., the NAT IP), then the destination port (port N), and finally 0. Other prefixes or padding bits are used in other embodiments, and those skilled in the art will recognize that this is simply one of many possible algorithms chosen for simplicity in generating IPv6 destination addresses. In some embodiments, the IPv6 destination port, source IPv6 address, and IPv6 source port are the original destination port, the source IPv6 address associated with gateway device 110, and the IPv6 source port randomly selected for that particular flow. However, those skilled in the art will understand that other destination ports and source IPv6 addresses and ports are used in other embodiments without affecting the routing of packets to a particular host and dSNAT instance.

[0049] After the packet is encapsulated in an IPv6 header including the generated IPv6 address, it is passed to the IPv6 forwarding operation represented by IPv6 routing table 115. A lookup in IPv6 routing table 115 identifies a matching routing table entry 115a that includes the next-hop (i.e., next-hop 3) interface for forwarding the packet to its destination. In some embodiments, if cache record 111a identifies the next-hop interface to be used after encapsulation, the lookup in IPv6 routing table 115 is not performed. In some embodiments, the IPv6 routing table entries for NAT IP (and LB IP) are dynamic routes learned based on advertisements from BGP instances on the host machine, while in other embodiments, the routing table entries are static routes received from controller computers in a group of controller computers (e.g., controller computer cluster 140). The IPv6 packet 330b is then forwarded to the destination host based on the IPv6 destination address (i.e., FC00:NAT IP:PortN::).

[0050] Figure 4 The illustration shows, for example Figure 1The gateway device 110 depicted performs packet processing for a subsequent packet 420 destined for an LB VIP in a specific packet flow (for which packets were previously received from external network 101). As shown, packet 420 is received from machine 102 in external network 101 with the LB VIP address and port B (i.e., the port associated with the VIP) as its destination. The packet is first compared with a cache entry in cache 111 (for the specific packet flow destined for the LB VIP, cache 111 contains cache record 111b based on the first packet 320a received from external network 101). Cache record 111b indicates that packet 420 should be encapsulated in an IPv6 header according to a specific encapsulation rule (i.e., "IPv6 Encapsulation Rule 2"). In some embodiments, the packet and rule identifier are passed to IPv6 encapsulator 114, bypassing pre-routing rule 112. However, those skilled in the art will understand that in other embodiments, the cache record identifies the specific IPv6 address or IPv6 encapsulation address and next hop used to encapsulate the packet in the flow based on the rule. The IPv6 encapsulator 114 then encapsulates the packet with an IPv6 header based on the identified rule (i.e., rule 114b).

[0051] In the illustrated embodiment, the destination IPv6 address is generated using an FC00 prefix for a locally unique address, followed by the destination IP (i.e., the NAT IP), then the source port (port Z), and finally 0. In some embodiments, the source port is used instead of the destination port because the source port is randomly selected from possible port values ​​when a flow (e.g., a session or connection) is initiated and remains constant throughout the packet's lifetime. This allows individual flows to be distributed across multiple LB instances based on different ranges of source ports assigned to each LB instance, while ensuring that packets for a particular flow are received by the same LB instance that maintains state information for providing LB services to that flow. On the other hand, if a destination port is used, in some cases, the LB VIP may be associated with multiple servers listening on one or more specific ports, making it impossible for the destination port to identify a specific LB instance. In other embodiments (where the destination port can be used to distinguish different hosts), the rules will be the same as those for the NAT IP in Figure 3. Other prefixes or padding bits are used in other embodiments, and those skilled in the art will recognize that this is simply one of many possible algorithms chosen for simplicity in generating IPv6 destination addresses. In some embodiments, the IPv6 destination port, source IPv6 address, and IPv6 source port are the original destination port, the source IPv6 address associated with gateway device 110, and the IPv6 source port randomly selected for that particular flow. However, those skilled in the art will understand that other destination ports and source IPv6 addresses and ports are used in other embodiments without affecting the routing of packets to a particular host and dBB instance.

[0052] In some embodiments, even some stateful distributed services are advertised as available on each host computer using the same network address. For example, in some embodiments, a stateful distributed load balancing service that distributes requests received from clients in an external network across host computers providing distributed load balancing services relies on a set of gateways in the Availability Zone (AZ) to consistently send the same flow to the same host computer providing distributed load balancing, based on equal cost multipathing (ECMP) operations performed at the gateway device in the Availability Zone (AZ). To implement this ECMP operation, in some embodiments, the routing machine on each host computer executing the distributed load balancer instance advertises the same VIP address as available, and the gateway device in the AZ records multiple advertised next-hop addresses associated with that VIP as possible next hops. For a received data message addressed to that VIP, the gateway device in the AZ uses the ECMP operation to select a specific next hop. In such embodiments, when the number of host computers providing distributed load balancing services changes, an acceptable number of redirection operations may be required, making the effort to ensure that a different host computer can be deterministically identified for each flow (or data message) unprofitable.

[0053] After the packet is encapsulated in an IPv6 header including the generated IPv6 address, it is passed to the IPv6 forwarding operation represented by IPv6 routing table 115. A lookup in IPv6 routing table 115 identifies the next-hop (i.e., next-hop 3) interface for forwarding the packet to its destination. In some embodiments, if cache record 111b identifies the next-hop interface to be used after encapsulation, the lookup in IPv6 routing table 115 is not performed. In some embodiments, the IPv6 routing table entries for LB IP (and NAT IP) are dynamic routes learned based on advertisements from BGP instances on the host machine, while in other embodiments, the routing table entries are static routes received from the controller computers of a set of controller computers 140. The IPv6 packet 430 is then forwarded to the destination host based on the IPv6 destination address (i.e., FC00:NAT IP:PortZ::).

[0054] After the gateway device has encapsulated and forwarded the packet, the packet will reach the host computer on which the destination distributed middlebox instance runs. The host computer is configured to receive the encapsulated packet destined for the identified IPv6 address from the gateway device and remove the encapsulation to provide the internal IPv4 packet to the middlebox service instance running on the host computer based on the IPv4 address in the header of the internal IPv4 packet. Figure 5A process 500 executed at a host computer is conceptually illustrated for processing received IPv6 packets destined for a middlebox service instance executed on the host computer. In some embodiments, process 500 is executed by the host computer on which the middlebox service instance executes. In some embodiments, the host computer uses an MFE (e.g., MFE 121a), a software routing element (e.g., routing machine 123a), an IPv6 processing module (e.g., IPv6 processing module 122a), a distributed middlebox instance (e.g., SNAT 124a), and a storage device (e.g., NAT record 125a) executing on the host computer to execute process 500. In other embodiments, some elements are combined (e.g., the MFE implements (or) the software routing element and contains IPv6 packet processing instructions, while the distributed middlebox service instance includes records for performing middlebox services). Process 500 begins by receiving (at 510) an IPv6 packet destined for an IPv6 address associated with an instance of middlebox service running on the host computer.

[0055] Procedure 500 then removes (at 520) the IPv6 encapsulation header and performs a lookup in the routing table to identify the next hop for the internal IPv4 packet. In some embodiments, the received packet is passed to an IPv6 processing module (e.g., 122) that is programmed with IPv6 routing rules and tables. In some embodiments, these routing rules include rules that identify IPv6 packets destined for the middlebox service instance as requiring removal of the IPv6 encapsulation header and are passed to an IPv4 routing table lookup (e.g., performed by the MFE or routing module). In some embodiments, this rule identifies the next hop and the IPv6 encapsulation should be removed before forwarding the packet.

[0056] In other embodiments, after removing the IPv6 encapsulation, a lookup is performed in the IPv4 routing table to identify the next hop toward the middlebox service instance. In some embodiments, the IPv6 processing module is the IPv6 stack of a dual-stack routing element (MFE or routing module), rather than a separate module. In some embodiments, the lookup is performed in a first virtual routing and forwarding (VRF) context of a first logical network or tenant associated with the middlebox service instance. In some embodiments, a virtual tunnel end point (VTEP) receives encapsulated packets based on the IPv6 address of the encapsulated packets. The VTEP then decapsulates the packets and identifies the next hop based on the IPv4 address and virtual network identifier associated with the received IPv6 packets. The packets are then forwarded (at 530) toward the middlebox service instance with an IPv4 header, using the IPv4 header value from the packets received at the gateway device and through the identified next-hop interface.

[0057] A distributed middlebox service instance receives packets and identifies (at 540) the destination internal IPv4 header value to forward the data to the correct destination. In some embodiments, identifying the destination internal IPv4 header value includes performing a lookup in a middlebox service record storage device (e.g., a cache) that maintains an association between the port used to replace the source port of the outgoing packet and the source IPv4 address and port. In some embodiments, the port used to replace the source port of the outgoing packet belongs to a range of port numbers assigned to the middlebox service instance running on the host computer (e.g., for the middlebox service instance to replace the source port of the outgoing packet or to be redirected to the source port of the middlebox service instance). In some embodiments, the destination port is used to perform a lookup (e.g., a query) in the middlebox service record storage device to identify the internal IPv4 address and port for replacing the current (external) destination IPv4 address and port, thereby enabling the forwarding of packets to the correct destination machine (e.g., virtual machine, container, pod, etc.).

[0058] For a distributed load balancer instance, identifying the destination internal IPv4 address for the first packet of a specific flow involves performing a load balancing operation to select a destination machine (on the same host computer or on a different host computer). In some embodiments, the load balancing operation preferentially selects a local compute node to provide the service associated with the VIP to reduce redirection; however, based on a load balancing algorithm or method, the load balancing operation may also select any compute node operating on any host computer and select at least one compute node on at least one other host computer for at least one flow. In some embodiments, after the destination machine is selected, the distributed load balancer instance creates a record in the middlebox service record storage to identify the destination IPv4 header value for subsequent packets in that specific flow. For subsequent packets in a flow destined for that LB VIP, the lookup in the middlebox service record storage is based on a set of at least one other IPv4 header value (e.g., source IP, source port, source IP / port, etc.).

[0059] After the internal IPv4 address and port have been identified, the distributed middlebox service replaces (at 550) the external IPv4 address and port with the identified internal IPv4 address and port. The packet is then forwarded (at 560) to the destination machine based on the internal IPv4 address and port. In some embodiments, the packet is forwarded (at 560) by connecting the destination machine to a logical switch of the distributed middlebox service instance. In some embodiments, for LB instances, the identified IPv4 destination connected to the logical switch is a compute node running on another host computer that also implements that logical switch (i.e., within the span of that logical switch). The process ends after the packet is forwarded to the destination machine.

[0060] Figure 6 The illustration shows a packet sent from external machine 102 to guest machine (GM) 126 in the internal network. The raw data message is sent along with an IP header 650a, which specifies the destination IP (DIP) address associated with the dSNAT middlebox service and the destination port (DPort) within the range assigned to a specific dSNAT middlebox service instance 124a. The source IP address (SIP, Ext IP1) and port (SPort, Port Y) are associated with external machine 102. At gateway device 110, the IPv4 packet sent by external machine 102 (i.e., internal packet 670) is based on the above information. Figures 2-3B The discussion process is encapsulated with an IPv6 header 660. The resulting packets have a destination IPv6 address (e.g., FC00:NAT IP:PortN::) associated with host computer 120a and NAT instance 124a, and are sent to host computer 120 via intermediate architecture 150.

[0061] The IPv6-encapsulated packet is then received at the managed forwarding element (MFE) 121a and passed to the IPv6 processing module 122a for IPv6 processing. In other embodiments, IPv6 processing is performed in the IPv6 stack of a routing machine 123a or a dual-stack MFE. In some embodiments, IPv6 processing includes removing the IPv6 encapsulation and returning the IPv4 packet to the MFE 121a. In other embodiments, IPv6 processing includes identifying the NAT instance 124a based on the IPv6 header value and removing the encapsulation. In embodiments where the inner packet 670 is returned to the MFE 121a, the MFE 121a identifies the NAT instance 124a as the destination of the inner packet 670 based on the IPv4 header.

[0062] The NAT 124a instance receives raw (or internal) IPv4 packets and performs a lookup in the NAT record storage device 127a. NAT record 127a includes record 680 that associates a port number used to replace the source port with the source port being replaced and the source IPv4 address being replaced. In some embodiments, the port number used to replace the source port is also associated with the IP address of an external machine, allowing the same port number to be reused for connections to different external machines. In some embodiments, the lookup is based on the destination port of the IPv4 packet. The result of the lookup in NAT record 127a is then provided to the NAT instance 124a for replacing the destination IPv4 address and port number. The IPv4 packet with the translated address (i.e., with IPv4 header 650b) is then forwarded to the destination machine (e.g., GM 126a).

[0063] Figure 7 A conceptual illustration shows a process 700 performed by a NAT instance (e.g., NAT instance 124a) that processes the first packet in a packet stream destined for a destination machine 102 in an external network 101. Figure 7 The discussion will refer to Figure 8 Components to provide Figure 7 Examples of operations. Figure 8The illustration shows the first packet in a packet stream being sent from a GM and processed by a NAT instance executing on the same host computer. Process 700 begins by receiving (at 710) a packet destined for a machine on an external network. The received packet has a source IPv4 address associated with the source machine on the internal network and the source port selected by the source machine. For example, packet 831 received at NAT instance 124a from GM 126a is destined for an external IP (i.e., Ext IP1) and a destination port (i.e., Port X), and has the source IP address (i.e., GM IP) and source port (i.e., Port Y) of GM 126a. In some embodiments, the source port of packet 831 is randomly selected from the entire range of source ports (0-65535).

[0064] The process selects a source port number (e.g., port N) from the range of available port numbers allocated to the NAT instance (at 720). In some embodiments, these available port numbers are port numbers within the allocated port number range that have not yet been selected for the currently active connection. In some embodiments, the port number range is allocated by the controller computer cluster. In some embodiments, the port number range is a range of port numbers that share a first set of common bits that are not shared by port numbers allocated to other NAT instances on other host computers. For example, each port number within a range of 1024 port numbers, 0-1023 or 2048-3071, shares a different set of 6 common initial bits in its 16-bit port address. Larger or smaller allocated ranges will have fewer or more common bits, respectively.

[0065] After selecting the port number (at 720), process 700 creates a connection trace record (at 730) in the connection tracker (e.g., NAT record 127), which associates the selected port number with the source IP address and source port of the packet for which the port number was selected. Figure 8 In this process, NAT record 841 associates the selected port (Port N) with the source IP address (GM IP) and source port (Port Y) of the received packet 831. In some embodiments, the port number used to replace the source port is also associated with the IP address of the external machine, allowing the same port number to be reused for connections to different external machines. As described above, connection tracking records are used to translate the NAT IP and port number of packets received from the external network back to the original connection's internal IP and port number associated with the destination port number of the received packet.

[0066] After creating (at 730) a connection trace, process 700 (at 740) replaces the source IP address and source port number with a specific IP address and a selected port number from a set of IP addresses assigned to the distributed NAT service. For simplicity, the examples throughout this specification assume that the distributed NAT service is assigned a single external IP address, i.e., the NAT IP. Those skilled in the art will understand that the methods discussed are similarly applicable to multiple IP addresses associated with a distributed NAT service (or any other distributed middleware service using a set of external IP addresses). For example, the source IP address and port of packet 831 are replaced with the NAT IP and the selected port number to produce serviced packet 832, which is then forwarded (at 750) to its destination and process 700 ends. In some embodiments, forwarding the serviced packet to its destination includes forwarding the serviced packet to an MFE (e.g., MFE 121a) for forwarding to an external destination.

[0067] In some embodiments, packets destined for an external network, processed by an intermediate box service instance running on a host computer, are sent from the host computer without being encapsulated in an IPv6 header using the advertised IPv6 address as the source IP address. While the IPv6 address prefix associated with the intermediate box service instance is not used to encapsulate the packet, in some embodiments, other encapsulation may be used to reach a gateway device or external destination machine. In some embodiments, packets processed by the intermediate box service instance are encapsulated in an IPv6 header using the IPv6 source address associated with the intermediate box service instance. Figure 8 The illustration shows a servicing packet 832 encapsulated in an overlay encapsulation header 870 for transmission over the overlay network to which GM 126a and NAT instance 124a belong (e.g., using an overlay network identifier such as a Virtual Network Identifier (VNI)). In some embodiments, an MFE or Virtual Tunneling Endpoint (VTEP) performs encapsulation of the servicing packet 832 to produce an encapsulated packet 833.

[0068] In some embodiments, a cluster of controller computers in the first network (i.e., a group of one or more controller computers) provides configuration information to network elements to facilitate the operation of the middlebox service of the first network. Figure 9 A conceptual illustration depicts a process 900 for generating configuration data for different network elements that provide or facilitate the provision of middlebox services. In some embodiments, process 900 is executed by a controller computer or a cluster of controller computers. In some embodiments, process 900 is executed each time a new middlebox service instance is spin-up.

[0069] Process 900 begins by identifying (at 910) middlebox service instances in the first network. In some embodiments, identifying a middlebox service instance includes identifying: (1) an active middlebox service instance and (2) a middlebox service instance that has been requested to be activated (e.g., enabled). In some embodiments, identifying a middlebox service instance includes identifying the number of endpoint machines (e.g., workload VMs, containers, etc.) supported (i.e., providing middlebox services to) each middlebox service instance. In some embodiments, in addition to identifying the number of endpoint machines, identifying a middlebox service instance also includes identifying any one or both of: (1) the total number of connections being processed by the distributed middlebox service (i.e., the sum of all distributed middlebox instances) and (2) the number of connections being processed by each middlebox service instance.

[0070] After identifying (at 910) the middlebox service instance, process 900 determines (at 920) the number or size of port ranges that will be available for allocation. In some embodiments, the number or size of port ranges is determined based on input from a user (e.g., an administrator) from a first network or a logical network within the first network. The user input may be based on the maximum amount of resources the user expects the middlebox service instance to consume while providing the middlebox service. In some embodiments, the user input specifies any or all of the following: (1) the maximum number of middlebox service instances that can be instantiated, (2) the maximum number of ports that can be allocated to a single middlebox service instance, or (3) a strategy for determining the number of ports allocated to a particular middlebox service instance. In some embodiments, these strategies are based on any or all of the following: (1) the number of active middlebox service instances, (2) the number of compute nodes for which each active middlebox service instance provides distributed middlebox services, (3) the number of connections being processed by the distributed middlebox service, and (4) the number of connections being processed by each middlebox service instance.

[0071] For example, a strategy could specify dividing the entire range of possible port numbers into a power of two, where the power of two is at least twice (or as large as) the number of middlebox service instances, and the port number range is adjusted based on this strategy as the number of middlebox service instances increases or decreases (e.g., from 4 to 5 middlebox service instances resulting in each of the 8 port ranges being divided into two smaller port number ranges, or from 17 to 16 middlebox service instances resulting in 64 port number ranges being merged into 32 port number ranges). In some embodiments, the strategy specifies that each middlebox service instance is assigned a non-adjacent port number range (e.g., 0-8191 assigned to the first middlebox service instance, 16384-2475 assigned to the second middlebox service instance, and so on). Such a strategy allows the number of host computers to be increased or decreased without frequently reallocating port number ranges.

[0072] In some embodiments, the policy may specify that: (1) when the percentage of port numbers allocated to a particular middleware service instance used by that middleware service instance is higher than a threshold percentage (e.g., 0.8 or 0.9), an adjacent available range will be allocated, workload compute nodes will be migrated from the host computer on which the middleware service instance is running, or a new middleware service instance will be started (e.g., activated on another host computer); (2) when the percentage of port numbers allocated to a particular middleware service instance used by that middleware service instance is lower than a threshold percentage (e.g., 0.3 or 0.2), the allocated port number range will be reduced, or additional terminal machines will be migrated to the host computer on which the middleware service instance is running (e.g., from the host computer running a middleware service instance that uses a larger percentage of its allocated port numbers); and (3) when the total number of connections being processed by a middleware service instance is lower than a certain threshold percentage of capacity (which is based on the number of middleware service instances and the allocated port range), the middleware service instance will be deactivated, or a smaller port range will be allocated to each active middleware service instance. Other strategies may specify port ranges based on the number of workload compute nodes providing middlebox services from the middlebox service instance (e.g., 256 port numbers are allocated for 0-10 workload compute nodes; 512 port numbers are allocated for 11-20 workload compute nodes; and so on). Those skilled in the art will understand that these strategies are merely examples of possible strategies, and different strategies are used in different embodiments depending on user requirements.

[0073] After identifying (at 910) the middlebox service instance and determining (at 920) the port number range, process 900 selects (at 930) at least one port range for allocation to each middlebox service instance. As described above, in some embodiments, the initial port range allocation assigns non-adjacent, non-overlapping port ranges to each middlebox service instance. In some embodiments, subsequent allocations assign at least one additional port number range to specific middlebox service instances that use a number of port numbers higher than a threshold proportion of the allocated port numbers. In some embodiments, other subsequent allocations remove a portion of the port number range from the initial allocation to specific middlebox service instances that use fewer than a threshold number of port numbers in the initially allocated port number range.

[0074] In some embodiments, the size of the port number range assigned to a middlebox service instance is fixed by the administrator based on the expected maximum number of middlebox service instances (e.g., for a maximum of 64 middlebox service instances, 64 different port number ranges are created, each containing 1024 ports, and each range is assigned to one middlebox service instance at startup). In other embodiments, the size of the port number range is dynamic and can change based on the number of active middlebox service instances, active connections, or workload compute nodes using the middlebox service. The size of the port number ranges may also differ between middlebox service instances. For example, a larger port number range may be assigned to a first middlebox service instance running on a host computer running a larger number of workload compute nodes using the middlebox service, compared to a second middlebox service instance running on a host computer running a smaller number of workload compute nodes using the middlebox service, and this range can change as the number of workload compute nodes changes.

[0075] After selecting (at 930) the range of port numbers to be used for each middlebox service instance, process 900 generates (at 940) configuration data for implementing the desired middlebox service instance. In some embodiments, the configuration data includes multiple sets of configuration data for different network elements (e.g., host computers, gateway devices) and for different purposes. Figure 10 The illustration shows a computer controller cluster 1040 in data center 1005 sending different types of configuration data to different network elements. Figure 10 The components as a whole and Figure 1 The components discussed are largely the same, with the addition of a local controller 1028 that receives configuration data from the controller cluster 1040. Figure 10A set of configuration data 1029 for each host computer 1020 (received at the local controller 1028) is illustrated. In some embodiments, the configuration data 1029 includes configuration information for: (1) configuring a middlebox service instance to provide middlebox services, (2) configuring other network elements (e.g., GM1026 and MFE 1021) running on the host computer to communicate with the middlebox service instance (e.g., 1024), and (3) configuring an MFE or BGP instance running on the host computer to advertise the IPv6 address associated with the middlebox service instance running on the host computer. In some embodiments, the local controller 1028 receives the configuration data and identifies the configuration data for each module running on the host computer 1020, such as regarding Figure 12 As explained. In some embodiments, the controller computer cluster 1040 also sends configuration data 1019 to the gateway device group for configuring the gateway devices to perform IPv4 to IPv6 encapsulation, and in some embodiments, for configuring the gateway devices with IPv6 routing table entries.

[0076] Configuration data (e.g., configuration data 1029) includes configuration data for configuring at least one middlebox service instance running on at least one host computer to provide middlebox services using an allocated port number range. In some embodiments, configuration data for initializing a new middlebox service instance on the host computer includes an IPv4 address associated with the middlebox service used when performing middlebox service operations (e.g., replacing the source IP address of packets from a first network to an external network) and an allocated port number range. In some embodiments, additional configuration information (e.g., logical overlay network elements to which the middlebox instance is connected) is sent to the host computer to configure other elements of the host to communicate with the new middlebox service instance, as will be understood by those skilled in the art.

[0077] In some embodiments, the additional configuration data sent to the host computer includes configuration data sent to the host computer to configure the host computer (or an MFE or BGP instance running on the host computer) to identify and advertise the IPv6 address prefix associated with the middlebox service instance, such as regarding Figure 12 As described. As regarding Figure 12 In some embodiments, the configuration data discussed also includes information about the host computer’s internal addressing of the middlebox service instance and the configuration of the machine running on the host computer to use the middlebox service instance for specific packets (e.g., packets destined for external networks).

[0078] In some embodiments, the generated configuration data includes configuration data generated for provision to the gateway device (e.g., configuration data 1019). In some embodiments, the gateway device is a partially or fully programmable gateway device that can be programmed by a controller computer cluster to perform IPv4-to-IPv6 conversion and encapsulation according to PBR rules specified based on the IPv4 address and destination port in the IPv4 header. In other embodiments, the gateway device is an off-the-shelf gateway device (e.g., a dual-stack router) that can be easily programmed to configure the gateway device to implement IPv4-to-IPv6 encapsulation.

[0079] For both programmable gateway devices and off-the-shelf gateway devices, the configuration data includes what will be referred to as a set of middlebox service records and IPv6 routing table entries. In some embodiments, the middlebox service record maps a combination of an IPv4 address and a destination port number used by a particular middlebox service operation to an IPv6 destination address. In some embodiments, the middlebox service record is provided as a lookup table and instructions for using the lookup table to route data messages using the IPv4 address used by the distributed middlebox service. In some embodiments, the middlebox service record is a PBR rule (or a similar rule or policy) that defines an algorithm for generating IPv6 addresses from IPv4 destination addresses and port numbers. In some embodiments, the PBR rule specifies the IPv4 destination address to which the algorithm should be applied, while in other embodiments, both the IPv4 address and the port number are specified. In some embodiments, the middlebox service record is an instruction for configuring the off-the-shelf gateway device to perform IPv6 encapsulation on IPv4 packets destined for IPv4 used by a particular middlebox service operation, according to a specified algorithm. In some embodiments, this instruction is based on functionality provided by the off-the-shelf gateway device (e.g., exposed APIs).

[0080] In some embodiments, each IPv6 routing table entry identifies an IPv6 address prefix associated with a particular host computer in a set of multiple host computers (each host computer performing an instance of the middlebox service) and the next-hop interface for reaching that particular host computer. The IPv6 address prefix specified for a particular host computer in the IPv6 routing entry is based on the IPv4 address associated with the distributed middlebox service and the port number range assigned to the distributed middlebox service instance performing on the host computer. If multiple non-adjacent port ranges are assigned to a particular host computer, the set of IPv6 routing table entries includes multiple entries for that particular host computer.

[0081] After the configuration data is generated (at 940), the configuration data generated for each network element is forwarded (at 950) to the appropriate network element for use as per the configuration data for each network element. Figure 12 and Figure 11 The network element is configured as described. In some embodiments, configuration data is received at the host computer by a local controller (e.g., local controller 1028) or a local controller agent that communicates with the controller computer cluster using control plane messages. The local controller then provides the configuration data or configures the element on the host computer to implement the middlebox service (e.g., instantiating a middlebox service instance, configuring the GM to use the middlebox service instance, and configuring the MFE to advertise the IPv6 address prefix associated with the middlebox service instance, etc.). The configuration data generated for the gateway device is forwarded to the gateway device to configure the gateway device to identify the specific host computer associated with a particular received packet (e.g., by using the provided IPv6 routing table entry). The process ends after forwarding (at 950) the configuration data. Those skilled in the art will understand that in some embodiments, process 900 is performed for each of the multiple distributed middlebox service instances using the same IPv4 address as the source address for outgoing packets.

[0082] In some embodiments, the controller computer cluster periodically or on a schedule basis monitors the load on middlebox service instances and aggregates. In some embodiments, this monitoring is based on a program executed on the same host computer as the middlebox service instance. In some embodiments, this program monitors a set of metrics associated with the middlebox service instance (e.g., latency, number of connections processed, packets per second, number of terminal machines using the middlebox service instance, etc.). In some embodiments, operations 910 and 920 are performed whenever a new middlebox service instance or workload machine is requested to be initialized. In some embodiments, operations 910 and 920 are also performed periodically or on a schedule set by an administrator to determine whether monitoring information indicates that there is a need to reallocate port number ranges or any change in the size of port number ranges. If such a change occurs, operations 930-950 are performed to update the port range allocation and provide updated configuration data to network elements.

[0083] Figure 11 A conceptual illustration depicts process 1100, executed by a gateway device to facilitate the provision of middlebox services based on received configuration data. Process 1100 begins by receiving (at 1110) configuration data for facilitating the provision of middlebox services within the internal network. (See also: Regarding...) Figure 9 and Figure 12In some embodiments discussed, the configuration data is received from a controller computer cluster (e.g., configuration data 1019) or as an announcement of the availability of IPv6 address prefixes (e.g., announcement 1039). In some embodiments, the controller cluster provides a portion of the configuration data while the announcement includes different portions of that configuration data. For example, in some embodiments, the controller computer provides information such as... Figure 9 The description of the intermediate box service records related configuration data, while the IPv6 routing table entry configuration data is received from the host computer through the advertisement of IPv6 address prefixes available at the host computer, such as regarding Figure 12 As described. In an embodiment where the host computer provides configuration information for routing table entries via an announcement, when a new middlebox service instance comes online, the gateway device receives additional configuration information in the form of an additional announcement with the IPv6 address prefix of the new middlebox service instance.

[0084] Based on the received configuration data (at 1110), the gateway device creates at least one IPv6 routing table entry (at 1120) for the received IPv6 address prefix. In some embodiments, the routing table entry is a static entry provided by the controller computer cluster, which can be updated by the controller computer cluster when the configuration data changes. In other embodiments, the routing table entry is a dynamic routing table entry created based on BGP or other routing learning protocols known to those skilled in the art.

[0085] Process 1100 also creates (at 1130) an intermediate box service record based on the configuration data. (See also: Regarding...) Figure 9 and 12The middlebox service record discussed can be any type of record or rule that identifies packets destined for an IPv4 address associated with the middlebox service as requiring IPv6 encapsulation and enables gateway devices to identify the correct IPv6 destination address for reaching the middlebox service instance associated with the packet. For example, in some embodiments, the middlebox service record is any of the following: (1) a PBR rule that identifies packets destined for an IPv4 address associated with the middlebox service as requiring IPv6 encapsulation and specifies a method for producing an IPv6 destination; (2) a set of records for a lookup table that identifies an IPv6 destination address for a set of combinations of IPv4 addresses and destination ports; or (3) API instructions for an API exposed by an off-the-shelf gateway device for implementing programmatic encapsulation of IPv4 packets destined for IPv4 addresses associated with the middleware service into IPv6 packets. Those skilled in the art will understand that the creation of routing table entries (at 1120) and the creation of intermediate box service records (at 1130) can be performed simultaneously or in the reverse order described in procedure 1100. The process ends after the routing table entries and intermediate box service records have been created. However, those skilled in the art will understand that procedure 1100 or a portion thereof is performed each time new configuration data is received at the gateway device. For example, each time a host computer advertises a new IPv6 address prefix, it will cause a new IPv6 routing table entry to be created in the gateway device.

[0086] In some embodiments, the host computer is configured to advertise the availability of an IPv6 address prefix based on the IPv4 source address of packets used by the middlebox service instance as a source address for packets destined for an external network from a first network and the range of port numbers assigned to the middlebox service instance.

[0087] Figure 12 A conceptual illustration of process 1200 for configuring a host computer to execute a distributed middlebox service instance and advertising the IPv6 address prefix associated with the middlebox service instance executed on the host computer. In some embodiments, process 1200 is executed by a host computer (e.g., host computer 1020) that executes a local controller (e.g., local controller 1028) and a BGP instance (e.g., MFE 1021 or routing machine 1023).

[0088] Process 1200 begins by receiving (at 1210) configuration information relating to an intermediate box service instance running on a host computer. In some embodiments, the configuration information relating to the intermediate box service instance includes (1) an IPv4 address, the source address of packets sent from within the first network to machines in an external network by the intermediate box service, and (2) a range of port addresses assigned to the intermediate box service instance running on the host computer. In other embodiments, the configuration information also includes internal information for routing packets to the intermediate box service instance, such as IP and MAC addresses or next-hop interfaces for forwarding packets to the intermediate box service instance. In some embodiments, the IPv6 address associated with the intermediate box service instance is also included in the configuration information. In some embodiments, the configuration information is received from a controller computer in a cluster of controller computers configuring the elements of the first network.

[0089] After receiving the configuration information (at 1210), process 1200 identifies (at 1220) configuration data for configuring different components (e.g., machines, MFEs, filters, containers, pods, etc.) executing on the host computer. For example, the middlebox service instance requires configuration data including the external IP address associated with the middlebox service, the port range assigned to the middlebox service instance, and, in some embodiments, the IP and Media Access Control (MAC) addresses associated with the middlebox service instance. In some embodiments, the BGP instance (e.g., MFE 1021 or routing machine 1023) needs to know the IP address associated with the middlebox service and the port range assigned to the middlebox service instance. In other embodiments, an IPv6 address prefix is ​​provided to the BGP instance for advertising. Other network elements need to be configured with information for interacting with the middlebox service instance, such as policies for identifying packets requiring the middlebox service, the MAC address associated with the middlebox service instance, and other information that is clear to those skilled in the art.

[0090] Following the configuration data for each component, process 1200 configures (at 1230) the middlebox service instance using the IP address and port range associated with it. If the middlebox service instance is being configured for the first time (i.e., being instantiated), the configuration data includes additional information such as the IP and MAC addresses associated with the middlebox service instance. In some embodiments, when the middlebox service instance is instantiated, the configuration data only includes the IP address associated with the middlebox service, and subsequent updates only include the updated port range.

[0091] After configuring (at 1230) the middlebox service instance, process 1200 provides (at 1240) the BGP instance running on the host computer with an IPv6 address prefix identified as associated with the middlebox service instance based on the configuration information. In some embodiments, the identified IPv6 address prefix is ​​based on the IPv4 addresses used by the middlebox service included in the configuration information and the range of port numbers allocated to the middlebox service instance. In some embodiments, the allocated range of port numbers is a range of port numbers sharing a common set of leftmost bits. For example, an allocated range of 1024 port numbers is numbered using the common 6 bits of the leftmost port numbers, while an allocated range of 512 port numbers is numbered using the common 7 bits of the leftmost port numbers. In such embodiments, the IPv6 address prefix associated with the middlebox service instance is then identified as an IPv6 address prefix where the rightmost valid bit is a set of common bits in the port numbers allocated to the middlebox service instance. In some embodiments, the advertised IPv6 address is based on the existing capabilities of the hardware gateway device for IPv6 encapsulation of IPv4 packets.

[0092] Figure 13 The illustration shows three different exemplary advertised IPv6 address prefixes 1331-1333 and corresponding exemplary addresses 1341-1343. The IPv6 address prefixes 1331-1333 are used in different embodiments to advertise the availability of services at a host computer, while the exemplary addresses 1341-1343 are generated by a gateway device for use in the IPv6 encapsulation header to forward packets to a specific service instance performed on the host computer that made the advertisement. Figure 13 A set of identified IPv6 address prefixes 1331-1333 is shown, located in the FC00:: / 8 address block based on the configuration of the middlebox service instance. In some embodiments, other IPv6 prefixes are used for other address blocks, such as the FD00:: / 8 address block. The illustrated IPv6 includes the IPv4 address 1310 used by the middlebox service and the common bits of the port number range 1315 (i.e., the first 6 bits in the illustrated example). The illustrated example is for a middlebox service using the IPv4 address 1310 (or C6A8:2001 in hexadecimal) and the port range 1315 (0-1023 with 6 common "0" bits). The example also illustrates a user-configured prefix 1320 (e.g., F462:5D1C:A451:2BD6), which in some embodiments is used to distinguish packets received by different tenants or logical networks implemented in the same data center. In some embodiments, the user-configured 64-bit prefix is ​​randomly generated, making it unlikely that the public prefixes generated for different tenants will be the same.

[0093] The exemplary IPv6 prefix 1331 (i.e., FCC6:A820:0100:: / 46) is generated using the first 8 bits of the FC00:: / 8 prefix, followed by 32 bits of the serving IPv4 address 1310, and finally a port range prefix common to all port numbers in port range 1315. Similarly, the IPv6 prefix 1332 (i.e., FD00:C6A8:2001:: / 54) is generated using the first 16 bits of the FD00:: / 8 prefix, followed by 32 bits of the serving IPv4 address 1310, and finally a port range prefix common to all port numbers in port range 1315. Alternatively, the IPv6 prefix 1333 (i.e., FC00:<user-configured 64-bit prefix>:C6A8:2001:: / 118) is generated using the first 16 bits of the FC00:: / 8 prefix, followed by the user-configured 64-bit prefix 1320, then the 32 bits of the serving IPv4 address 1310, and finally the port range prefix common to all port numbers in port range 1315.

[0094] Figure 13 The diagram illustrates exemplary received packets with IPv6 destination addresses 1341-1343, each packet having a header value 1302 corresponding to the advertised IPv6 prefixes 1331-1333. The exemplary IPv6 destination address 1341 (i.e., FCC6:A820:0100:0000:01::) is generated by the gateway device using the first 8 bits of the FC00:: / 8 prefix, followed by 32 bits of the destination IPv4 address 1350, and finally the destination port number 1355. Similarly, the IPv6 destination address 1342 (i.e., FD00:C6A8:2001:0000:0001::) is generated using the first 16 bits of the FD00:: / 8 prefix, followed by 32 bits of the destination IPv4 address 1350, and finally the destination port number 1355. Alternatively, the IPv6 destination address 1343 (i.e., FC00:F462:5D1C:A451:2BD6:C6A8:2001::01:0000:0000) is generated using the first 16 bits of the FC00:: / 8 prefix, followed by the 64-bit prefix 1320 configured by the user, then the 32 bits of the destination IPv4 address 1350, and finally the destination port number 1355.

[0095] The BGP instance then advertises the provided IPv6 address prefix associated with the middlebox service instance to the gateway device as available at the host computer. In some embodiments, this advertisement includes instructions to the gateway device to identify the IPv6 address based on the IPv4 address and port number of the packet received at the gateway device and to encapsulate the IPv4 packet with the identified IPv6 address. In other embodiments, the advertised IPv6 address (1) is based on the existing capabilities of the hardware gateway device for processing IPv6 encapsulation of IPv4 packets, or (2) is based on a central controller cluster that is configured to perform IPv6 encapsulation based on the IPv4 address and port number. In these other embodiments, no specific configuration instructions from the BGP instance are required. In some embodiments, the advertisement uses the Border Gateway Protocol (BGP). In other embodiments, other proprietary or non-proprietary (IS-IS, OSPF, FRR, etc.) route advertisement protocols or methods are used to notify other routers of the availability of a particular network address at the host computer. In some embodiments, the advertisement is made to a routing server (or routing reflector) that identifies the advertised IPv6 address as available at the host computer to the gateway device and other forwarding elements of the internal network.

[0096] In some embodiments, the routing server also uses BGP advertisements to identify the advertised IPv6 addresses as available at the host computer, simplifying the exchange of routing information in the network by minimizing the number of peering operations between different network elements necessary to propagate routing information in a full-mesh configuration. However, those skilled in the art will understand that other methods can be used to advertise the availability of IP addresses. Those skilled in the art will understand that the description of the BGP instance and the BGP protocol is merely an example or protocol for advertising routing information and is used herein only as an example.

[0097] Process 1200 then continues to configure other components (e.g., machines, MFEs, filters, containers, pods, etc.) executed on the host computer based on the received configuration data (at 1250). In some embodiments, these other components are configured to direct traffic to the middlebox service instance based on policies included in the configuration data (which identify traffic destined for an external network). Those skilled in the art will understand that the order in which components are configured as presented above is not the only possible order, and in other embodiments, components are configured in parallel (e.g., simultaneously) or in an order different from that presented above.

[0098] Figure 14The concept illustrates a set of data exchanges between network elements to migrate VMs (e.g., guest virtual machines (GVMs)) that communicate with external machines using the dSNAT middlebox service. Figure 14 The diagram illustrates a controller computer 1440 that initiates the VM migration, a source host computer 1420a on which the VM initially runs, a destination host computer 1420b on which the VM is migrated, and a routing server 1430 that is used by the host computers as a central BGP server to propagate routing information (e.g., IP address prefixes available at each host computer). Data exchange begins (at 1410) when the controller computer initiates the VM migration by sending configuration data identifying that the VM is being migrated to each of the source and destination host computers 1420a and 1420b.

[0099] The source host computer then identifies (at 1411) a set of ports used by active connections of the migrated VM for a set of middleware services to notify the destination host computer 1420b of the migrated ports. The set of ports identified to the destination host computer 1420b includes NAT records that associate the identified ports with the internal IPv4 addresses and ports used by the migrated VM. In some embodiments, the identified ports are placed in a list of unused ports at the source host computer 1420a until the migrated VM releases them (e.g., when an active connection using the identified ports ends). In some embodiments, the source host computer 1420a also adds these port numbers to a redirection table to redirect any packets received for that port number to the destination host computer 1420b. In some embodiments where port numbers are reused for different source and destination IP address pairs, the redirection directive specifies specific communication destined for the migrated VM, rather than redirecting all packets using a specific port number.

[0100] Based on the identified port number, in some embodiments, the destination host computer advertises (at 1412) a set of IPv6 addresses for the specific migrated port number (e.g., an IP address prefix of FC00:IPv4 address:port:: / 64), which will be the longest matching prefix at the gateway device and replaces the existing prefix initially advertised by the source host computer 1420a (e.g., the / 54 prefix) without requiring further advertising from the source host computer 1420a. Those skilled in the art will understand that in some embodiments where the same port number is reused for different source and destination IP address pairs, the host computer to which the VM is migrated will not advertise and will instead rely on redirection at the host computer from which the VM is migrated. As each migrated port is "released" (i.e., an active connection using that port number ends), the destination host computer 1420b notifies the source host computer 1420a (at 1413) that the port has been released and can be used by the source host computer 1420a for new connections. In some embodiments, the source host computer then removes the released port from the list of unused ports.

[0101] In some embodiments, the destination host computer also advertises (at 1413) that a specific IPv6 address prefix (e.g., the / 64 prefix) is no longer available at the destination host computer 1420b. In some embodiments, the source host computer 1420a additionally advertises (at 1414) that a specific IPv6 address prefix (e.g., the / 64 prefix) is available at the source host computer 1420a. Those skilled in the art will understand that in embodiments where the same port number is reused for different source and destination IP address pairs, the described subsequent advertisements (at 1413 and 1414) are unnecessary because the host computer to which the VM is migrated does not advertise the IPv6 address. In some embodiments, after the destination host computer 1420b withdraws the IPv6 / 64 address, the source host computer 1420a omits the announcement of the IPv6 / 64 address prefix associated with the released port (at 1414) and relies on the IPv6 address prefix covering that specific IPv6 address (e.g., the / 54 prefix associated with the allocated port number range) to direct packets to the source host computer 1420a. In some embodiments, the destination host computer 1420b sends a message and announces the unavailability associated with each released port as the end of the connection using that port. Those skilled in the art will understand that VM migration is used as an example, but the above set of data exchanges can be used to migrate any similar compute nodes (e.g., containers, pods, etc.).

[0102] Many of the aforementioned features and applications are implemented as a software process that specifies a set of instructions recorded on a computer-readable storage medium (also known as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard disk drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or via wired connections.

[0103] In this specification, the term "software" means including firmware residing in read-only memory or an application stored in magnetic storage, which can be read into memory for processor processing. Furthermore, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while retaining distinct software inventions. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of separate programs that collectively implement the software inventions described herein is within the scope of this invention. In some embodiments, when a software program is installed to run on one or more electronic systems, one or more specific machine implementations define the operations for executing and running the software program.

[0104] Figure 15 A computer system 1500, which implements some embodiments of the present invention, is conceptually illustrated. The computer system 1500 can be used to implement any of the aforementioned host computers, controllers, and managers. Therefore, it can be used to perform any of the aforementioned processes. This computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 1500 includes a bus 1505, one or more processing units 1510, system memory 1525, read-only memory 1530, permanent storage device 1535, input device 1540, and output device 1545.

[0105] Bus 1505 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 1500. For example, bus 1505 communicatively connects one or more processing units 1510 to read-only memory 1530, system memory 1525, and permanent storage device 1535.

[0106] From these various memory units, one or more processing units 1510 retrieve instructions to be executed and data to be processed to perform the process of the present invention. In different embodiments, the processing units (one or more) may be a single processor or a multi-core processor. Read-only memory (ROM) 1530 stores static data and instructions required by the processing units (one or more) 1510 and other modules of the computer system. On the other hand, permanent storage device 1535 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 1500 is turned off. Some embodiments of the present invention use mass storage devices (such as disks or optical discs and their corresponding disk drives) as permanent storage device 1535.

[0107] Other embodiments use removable storage devices (such as floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 1535, system memory 1525 is a read-write memory device. However, unlike storage device 1535, system memory 1525 is volatile read-write memory, such as random access memory. System memory stores some instructions and data required by the processor during operation. In some embodiments, the processes of the present invention are stored in system memory 1525, permanent storage device 1535, and / or read-only memory 1530. One or more processing units 1510 retrieve instructions to be executed and data to be processed from these various memory units to perform the processes of some embodiments.

[0108] Bus 1505 is also connected to input device 1540 and output device 1545. Input device 1540 enables a user to communicate information to the computer system and make requests to the computer system. Input device 1540 includes an alphanumeric keypad and a pointing device (also referred to as a "cursor control device"). Output device 1545 displays images generated by the computer system. Output devices include printers and display devices such as cathode ray tube (CRT) or liquid crystal display (LCD). Some embodiments include devices that serve as both input and output devices, such as touchscreens.

[0109] Finally, as Figure 15 As shown, bus 1505 also couples computer system 1500 to network 1565 via a network adapter (not shown). In this way, the computer can be part of a computer network (such as a local area network (“LAN”), wide area network (“WAN”), or intranet) or a network of networks (such as the Internet). Any or all components of computer system 1500 can be used in conjunction with this invention.

[0110] Some embodiments include electronic components, such as microprocessors, storage, and memory, that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Examples of such computer-readable media include RAM, ROM, read-only optical disc (CD-ROM), recordable optical disc (CD-R), rewritable optical disc (CD-RW), read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard disk drives, read-only and recordable Blu-ray discs. Optical discs, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media may store computer programs executable by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code generated by a compiler, and files that include higher-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0111] While the above discussion primarily refers to microprocessors or multi-core processors that execute software, some embodiments are executed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0112] As used in this specification, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms "display" or "positive display" refer to a display on an electronic device. As used in this specification, the terms "computer-readable medium," "multiple computer-readable media," and "machine-readable medium" are limited entirely to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other transient or temporary signals.

[0113] While the invention has been described with reference to numerous specific details, those skilled in the art will recognize that it can be embodied in other specific forms without departing from its spirit. For example, several figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, a process may be implemented using several sub-processes or as part of a larger macro-process. Therefore, those skilled in the art will understand that the invention is not limited to the foregoing illustrative details but is to be defined by the appended claims.

Claims

1. A method for facilitating the operation of a Network Address Translation (NAT) middlebox service for a first network of a data center at a gateway device in the data center, the method comprising: At the gateway device in the data center: Receive a packet from a second network that includes an Internet Protocol version 4 (IPv4) header, the IPv4 header having a first destination IPv4 address and a first destination port; An IPv6 address is generated as a second destination IP address based on the first destination IPv4 address and the first destination port, wherein the second destination IP address includes an IPv6 address prefix based on the first destination IPv4 address associated with the NAT middlebox service instance and a port number range assigned to the NAT middlebox service instance, wherein the IPv6 address prefix is ​​locally unique for the host computer operating in the data center and executing the NAT middlebox service instance, wherein for a set of IPv6 address prefixes associated with multiple host computers executing the NAT middlebox service instance, the gateway device is configured with a set of IPv6 routes, the set of IPv6 routes indicating the next hop for each IPv6 address prefix, and wherein the set of IPv6 routes is created based on the multiple host computers advertising to the gateway device that the corresponding IPv6 address prefix is ​​available at the corresponding host computer; The received packet is encapsulated with an IPv6 header using the second destination IP address; and Based on the routing entries in the group of IPv6 routes, the encapsulated packets are forwarded to the host computer along the first network of the data center to (i) receive the packets, (ii) remove the IPv6 header, (iii) identify the NAT middlebox service instance using the first destination IP address and port in the IPv4 header, and (iv) guide the NAT middlebox service instance to replace the first destination IP address and port in the IPv4 address with a third destination IP address and a third destination port; (v) Providing the packet to a destination machine that is executed on the host computer and connected to the first network.

2. The method as described in claim 1, wherein, The advertised IPv6 address is an IPv6 address prefix based on a first destination IPv4 address, which is used by the NAT middlebox service instance to replace the source address of packets sent from the NAT middlebox service instance to the first network that provides the NAT middlebox service to it.

3. The method of claim 2, wherein The NAT middlebox service is implemented by multiple distributed NAT (dNAT) middlebox service instances, which are executed on multiple host computers and use a first destination IPv4 address to provide the dNAT middlebox service for packets sent from within a first network. Each dNAT middlebox service instance is assigned a non-overlapping range of port numbers for use when providing the dNAT middlebox service, and Each host computer advertises a different IPv6 address prefix based on a first destination IPv4 address and the port number range assigned to the dNAT middlebox service instance running on that host computer.

4. The method of claim 1, further comprising identifying packets destined for a first destination IPv4 address as requiring the generation and encapsulation operation to forward the packets using the generated second destination IP address, based on NAT records.

5. The method of claim 1, wherein, The second destination IP address is an IPv6 address that includes the first destination IPv4 address and a set of bits for the first destination port.

6. The method of claim 5, wherein, The IPv6 address includes a first set of bits, a second set of bits, and a third set of bits. The first set of bits indicates that the address is not necessarily globally unique. The second set of bits includes the IPv4 address. The third set of bits includes a set of bits belonging to the port number range assigned to the NAT middlebox instance.

7. The method of claim 4, wherein, The NAT record is received from the controller computer of the first network.

8. The method of claim 1, wherein, A set of IPv6 routing entries associated with a set of NAT middlebox service instances are received from the controller computer to be included in the IPv6 routing table of the gateway device, and forwarding the encapsulated packet includes performing a lookup in the IPv6 routing table to identify the next hop for forwarding the encapsulated packet to the NAT middlebox service instance.

9. A method for facilitating the provision of a distributed network address translation (dNAT) middlebox service at a host computer in a first network of a data center, the dNAT middlebox service being implemented by multiple dNAT middlebox service instances executed on multiple host computers in the data center, each dNAT using the same external IPv4 address as the source address of the served packets, the method comprising: A gateway device in the data center, operating between a first network and a second network, receives packets comprising an internal IPv4 packet and an Internet Protocol version 6 (IPv6) encapsulation header having a first destination IP address, wherein the first destination IP address includes an IPv6 address prefix based on a second destination IP address associated with a dNAT middlebox service instance in the IPv4 header of the internal IPv4 packet and a port number range assigned to the dNAT middlebox service instance, wherein the IPv6 address prefix is ​​locally unique for the host computer executing the dNAT middlebox service instance, wherein the gateway device is configured with a set of IPv6 routes for a set of IPv6 address prefixes associated with the plurality of host computers executing the plurality of dNAT middlebox service instances, the set of IPv6 routes indicating the next hop for each IPv6 address prefix, and wherein the set of IPv6 routes is created based on the plurality of host computers advertising to the gateway device that the corresponding IPv6 address prefix is ​​available at the respective host computer; Remove the IPv6 encapsulation header and use the second destination IP address and second destination port number from the IPv4 header of the inner IPv4 packet to identify the dNAT middlebox service instance executed on the host computer; as well as The internal IPv4 packets are forwarded along the first network of the data center to the identified dNAT middlebox service within the first network to receive the packets with the IPv4 header and replace the second destination IP address and second port number with a third destination IP address and port number, so that the packets can then be supplied to the destination machine that is executed on the host computer and connected to the first network.

10. The method of claim 9, wherein, The internal IPv4 packet is sent as an IPv4 packet by the source device in the second network using the second IP address and port number as the destination IP address and destination port number in the IPv4 packet header, and the gateway device receives the IPv4 packet for forwarding to the dNAT middlebox service instance in the first network.

11. The method of claim 10, wherein, The gateway device generates the IPv6 encapsulation header and forwards the encapsulated packet to the host computer based on the first destination IP address in the IPv6 encapsulation header.

12. The method of claim 11, wherein, The gateway device identifies the first destination IP address in the IPv6 encapsulation header based on the second IP address and port number in the IPv4 header.

13. The method of claim 12, wherein, The gateway device is configured by a controller computer in the first network to (1) identify a first destination IP address, (2) encapsulate an IPv4 packet destined for a second IP address with the IPv6 header using the first IP destination address, and (3) forward a data message destined for the first IPv6 address to the host computer.

14. The method of claim 9, further comprising: The gateway device announces that the IPv6 address prefix, including the first IP destination address, is available at the host computer, so that the gateway device forwards packets destined for the IPv6 destination address to the host computer, where the announced IPv6 address prefix is ​​the longest matching prefix for the IPv6 destination address.

15. The method of claim 14, further comprising identifying the IPv6 address prefix based on: (1) an IPv4 address used by the dNAT middlebox service instance as the source IP address of outgoing packets, and (2) a range of port numbers assigned to the dNAT middlebox service instance as the source port of outgoing packets.

16. The method of claim 14, wherein, Announcement of the IPv6 address prefix to the gateway device includes: announcing to the route reflector that the IPv6 address prefix is ​​available at the host computer, and the route reflector then announcing to the gateway device that the IPv6 address prefix is ​​available at the host computer.

17. The method of claim 14, wherein, The notification of the IPv6 address prefix to the gateway device also includes information for configuring the gateway device to perform IPv6 encapsulation on data messages destined for the second IP address.

18. The method of claim 9, wherein, The packet is the first packet in a packet stream between a device in the second network and the destination machine, and the method further includes: The second packet is received from the packet stream from the source machine, which is the destination machine of the first packet. The second packet has the same first source IP address as the second destination IP address and the same first source port as the destination port of the first packet. The second packet is forwarded to the gateway device in the second network without using the first IP destination address as the source IP address or destination IP address to encapsulate the second packet. In this process, the source machine of the destination machine, which is the first packet, sends a second packet. The second packet has a second source IP address that is the same as the third destination IP address and a second source port that is the same as the third destination source port. The second source IP address and the second source port are replaced by the first source IP address and the first source port by the dNAT service operation performed by the identified dNAT middlebox instance.

19. A machine-readable medium storing a program that, when implemented by at least one processing unit, implements the method as described in any one of claims 1-18.

20. An electronic device, comprising: A set of processing units; as well as A machine-readable medium storing a program that, when implemented by at least one of the processing units, implements the method as described in any one of claims 1-18.

21. A system comprising components for implementing the method as described in any one of claims 1-18.

22. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-18.

Images