Route advertisement supporting a distributed gateway service architecture

Abstract

Some embodiments of the invention provide a novel network architecture for announcing routes in an availability zone (e.g., a data center that provides a collection of hardware resources). In some embodiments, the novel network architecture also provides a collection of distributed services at the edge of a virtual private cloud (VPC) implemented in the availability zone (e.g., using the hardware resources of the data center) at a collection of host computers in the AZ. The novel network architecture includes a collection of route servers to receive announcements of network addresses (e.g., Internet Protocol (IP) addresses) available in the availability zone (AZ) from different routers in the AZ. The route servers also announce the received network addresses to other routers in the AZ. In some embodiments, the other routers include routers executing on host computers in the AZ and gateway devices of the availability zone.

Description

Background Art

[0001] Many networks rely on using stateful edge services provided at the edge of an availability zone (e.g., a data center). Stateful services for some networks are provided by a limited number of state-maintaining devices (e.g., specialized gateway devices located at the edge of an availability zone (AZ)). However, providing stateful services on a limited number of devices can lead to a bottleneck because all traffic using the stateful service must be handled by a limited number of devices. To address this bottleneck, some networks use a distributed stateful service architecture. However, a distributed stateful service architecture has its own challenges. For example, because the stateful service operations performed by each instance of a distributed stateful service require state information, in some cases traffic traversing the provider gateway device will be forwarded to a randomly selected stateful service instance. For a system with "N" distributed stateful service instances, this will result in data messages being directed to a distributed stateful service instance that does not store state information for the data message (N-1) of the N times.

[0002] Therefore, a solution is required that solves both the bottleneck and misdirection problems to provide stateful edge services such as SNAT and stateful load balancing. Summary of the Invention

[0003] Some embodiments of the present invention provide a novel network architecture for advertising routes in an availability zone (e.g., a data center that provides a collection of hardware resources). In some embodiments, the novel network architecture also provides a collection of distributed services at the edge of a virtual private cloud (VPC) implemented in an availability zone (e.g., using the hardware resources of a data center). The novel network architecture includes a collection of route servers for receiving advertisements of network addresses (e.g., Internet Protocol (IP) addresses) available in an availability zone (AZ) from different routers in the AZ. The route servers also advertise the received network addresses to other routers in the AZ. In some embodiments, the other routers include routers executed on host computers in the AZ, gateway devices (e.g., routers of availability zone routers in the AZ, and routers that constitute an intermediate architecture between routers executed on host computers, route servers, and gateway devices in the AZ).

[0004] The novel network architecture also includes a plurality of host computers, each host computer executing a router (e.g., a routing instance) that (i) identifies network addresses available on the host computer, (ii) sends an announcement of the identified network addresses to a set of routing servers, and (iii) receives announcements from the set of routing servers about network addresses available on other host computers. In some embodiments, the identified network addresses include network addresses associated with data compute nodes (DCNs) (e.g., virtual machines (VMs), containers, pods, etc.) and network addresses associated with services available at the host computer. In some embodiments, the services are distributed edge services provided for data messages entering or leaving a VPC implemented in an AZ. In some embodiments, the announcements are made using a border gateway protocol (BGP) or other route announcement protocol.

[0005] The novel architecture also includes a set of controller computers that configure the host computers to execute service instances to provide services for a virtual private cloud that includes DCNs executing on the host computers. In some embodiments, the service instances are used to provide distributed services, including distributed edge services provided for data messages entering or leaving a VPC implemented in an AZ. In some embodiments, the distributed edge services are provided at distributed logical routers located at a logical edge of the VPC (i.e., logical routers that process data messages entering or leaving the VPC) as part of logical router processing.

[0006] In some embodiments, a novel method is presented for identifying network addresses available at a host computer. A host executing a routing instance (e.g., a routing machine) detects that a network address has become available on the host computer. In some embodiments, detecting that a network address has become available includes detecting that at least one of a DCN or a service has been added to the host computer. The routing instance then identifies a network address associated with the detected DCN or the service that has been added to the host computer. In some embodiments, the identified network address is an address used by other routers in the AZ. In other embodiments, the identified network address is an address used by routers and machines in a network outside the AZ. The routing instance then announces the identified network address to at least one routing server for announcement to other routers in the AZ.

[0007] A host computer in a novel architecture, upon receiving a data message entering a VPC and destined for a set of one or more DCNs in the VPC, performs a set of processing steps before delivering the data message to the destination DCN. In some embodiments, the host computer receives the data message at a managed forwarding element (MFE) (e.g., a software switch executing on the host computer). In some embodiments, the MFE provides the received data message to a routing instance to determine a next hop. In some embodiments, the next hop for the data message entering the VPC is a tunnel endpoint (e.g., a Virtual Extensible Local Area Network (VXLAN) Tunnel Endpoint (VTEP)) that decapsulates the data message and provides the decapsulated data message to a distributed logical router for processing operations. In some embodiments, the distributed logical router is defined at the logical edge of the VPC such that all traffic entering and leaving the VPC is processed by the distributed logical router.

[0008] In some embodiments, the distributed logical router processing operation identifies a set of services (e.g., stateful or stateless edge services) required to process data messages entering the VPC. In some embodiments, the set of services includes any or all of a firewall service, a network address translation (NAT) service, a load balancing service, a distributed intrusion detection system (IDS) service, and a distributed intrusion protection system (IPS) service. Different services use different information contained in data messages and, in some embodiments, are implemented in different ways based on the type of information used. For example, services that use information contained in headers at Layer 4 of the Open Systems Interconnection (OSI) model (sometimes referred to as "L4 services") execute in kernel space because they do not consume a lot of resources and moving them to user space would make them unacceptably slow. However, in some embodiments, services that use data at Layer 7 of the OSI model (sometimes referred to as "L7 services") execute in user space because their resource-intensive processing would occupy threads in kernel space in order to quickly process data messages and because the cost of moving them into user space is not significant compared to the cost of performing the operation in the first place.

[0009] In some embodiments, a set of services is implemented for multiple tenants in separate VPCs on the same host computer. In some embodiments, different services are provided by different modules (containers, applications, etc.) in a multi-tenant service machine (or pod). In some embodiments using a multi-tenant service machine, the service machine implements separate VTEPs to differentiate traffic for each tenant. In other embodiments, a separate service machine (e.g., virtual machine, container, pod, etc.) is provided for each tenant to implement the tenant's set of L7 services.

[0010] In some multi-tenant embodiments, a routing instance is a multi-tenant routing instance that maintains separate virtual routing and forwarding (VRF) contexts for each tenant. In some embodiments, the routing instance is a BGP instance. In some embodiments, the routing instance is a multi-protocol BGP instance (MP-BGP) that supports multiple addressing protocols (e.g., Multi-Protocol Label Switching (MPLS), BGP Labeled Unicast (BGP-LU), Segment Routing (SR), etc.). In some embodiments, different VRF contexts are distinguished by a route distinguisher, and a route target is used to identify announcements associated with each VRF context. In some embodiments, the routing instance includes a free-range routing daemon that implements MP-BGP announcement and route learning operations for the routing instance.

[0011] The preceding summary is intended to serve as a brief introduction to some embodiments of the present invention. It is not intended to be an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and accompanying figures referenced in the detailed description further describe the embodiments described in the summary, as well as other embodiments. Therefore, a comprehensive review of the summary, detailed description, accompanying figures, and claims is required to understand all embodiments described in this document. Furthermore, the claimed subject matter is not limited by the illustrative details in the summary, detailed description, and accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The novel features of the invention are set forth in the appended claims.For purposes of illustration, however, several embodiments of the invention are set forth in the following drawings.

[0013] Figure 1 An exemplary network in which the novel architecture is implemented is illustrated.

[0014] Figure 2 An embodiment of a host computer within a network executing components for two different VPCs is illustrated.

[0015] Figure 3 A second embodiment of a host computer executing components of multiple VPCs within a network is illustrated.

[0016] Figure 4 Conceptually illustrates a process for configuring components of a host computer to implement a virtual private cloud within an availability zone.

[0017] Figure 5 Conceptually illustrates a process for detecting that a data compute node has been added to a host and advertising the network address associated with the data compute node to other routers in an availability zone.

[0018] Figure 6Conceptually illustrates a process for detecting that a DCN has been removed from a host computer.

[0019] Figure 7 Conceptually illustrates a process for detecting a service that has been added to a host computer and advertising a network address to other routers in an AZ.

[0020] Figure 8 Conceptually illustrates a similar process for detecting that a service has been removed from a host computer and withdrawing a network address associated with the removed service from other routers in an availability zone.

[0021] Figure 9 Conceptually illustrates a process for a routing server that receives a routing advertisement and generates a routing advertisement to other routers based on the received advertisement.

[0022] Figure 10 Conceptually illustrates a process for monitoring the health of individual service instances and advertising a failure of a service instance to routers in an AZ.

[0023] Figure 11 Conceptually illustrates a process for updating routing information at a multi-tenant routing instance.

[0024] Figure 12 Illustrates a routing table for a multi-tenant MP-BGP routing instance receiving a routing advertisement from a routing server at a free range routing (FRR) daemon and updating a virtual routing and forwarding context.

[0025] Figure 13 Conceptually illustrates a process for processing data messages addressed to a service or DCN executing on a host computer.

[0026] Figure 14 Conceptually illustrates a process for generating configuration data for different network elements and for providing a collection of distributed services at the edge of a VPC, where the network elements provide a novel network architecture for advertising routes in an availability zone.

[0027] Figure 15 Illustrates a computer controller cluster in a data center sending different types of configuration data to different network elements.

[0028] Figure 16 Conceptually illustrates a computer system with which some embodiments of the invention are implemented. DETAILED DESCRIPTION

[0029] In the following detailed description of the present invention, many details, examples and embodiments of the present invention are set forth and described. However, it will be apparent to those skilled in the art that the present invention is not limited to the embodiments set forth, and that the present invention can be practiced without discussing some of these specific details and examples.

[0030] Some embodiments of the present invention provide a novel network architecture for advertising routes in an availability zone (e.g., a data center that provides a collection of hardware resources). In some embodiments, the novel network architecture also provides a collection of distributed services at the edge of a virtual private cloud (VPC) implemented in an availability zone (e.g., using the hardware resources of a data center). The novel network architecture includes a collection of route servers (sometimes referred to as route reflectors) that receive network addresses available in an availability zone (AZ) from different routers in the AZ. The route servers also advertise the received network addresses to other routers in the AZ. In some embodiments, the other routers include routers executed on host computers in the AZ, gateway devices (e.g., routers of availability zone routers in the AZ, and routers that constitute an intermediate architecture between routers executed on host computers, route servers, and gateway devices in the AZ).

[0031] The novel network architecture also includes a plurality of host computers, each of which executes a router (e.g., a routing instance) that (i) identifies network addresses available on the host computer, (ii) sends advertisements of the identified network addresses to a set of routing servers, and (iii) receives advertisements of network addresses available on other host computers from the set of routing servers. In some embodiments, the identified network addresses include network addresses associated with data computing nodes (DCNs) (e.g., virtual machines (VMs), containers, pods, etc.) and network addresses associated with services available at the host computers. In some embodiments, the services are distributed edge services provided for data messages entering or leaving a VPC implemented in an AZ. In some embodiments, the advertisements are made using the Border Gateway Protocol (BGP) or other routing advertisement protocols.

[0032] As used in this document, a packet refers to a collection of bits sent across a network in a specific format. In some embodiments, a packet is referred to as a data message. One of ordinary skill in the art will recognize that the terms packet and data message are used in this document to refer to various formatted collections of bits sent across a network. The formatting of these bits can be specified by standardized protocols or non-standardized protocols. Examples of packets that follow standardized protocols include Ethernet frames, IP packets, TCP segments, UDP datagrams, etc. Moreover, as used in this document, references to L2, L3, L4, and L7 layers (or Layer 2, Layer 3, Layer 4, and Layer 7) refer to the second data link layer, the third network layer, the fourth transport layer, and the seventh application layer, respectively, of the OSI (Open Systems Interconnection) layer model.

[0033] Furthermore, in this example, the term managed forwarding element (MFE) refers to a software forwarding element or a hardware forwarding element that is configured by a controller computer cluster (i.e., a collection of one or more controller computers that provide configuration data to network elements) to implement a virtual private cloud (VPC) (e.g., a logical network that includes a collection of logical forwarding elements (LFEs)). In some embodiments, the VPC is implemented as a logical overlay network that uses tunneled packets to traverse the physical underlying network. In some embodiments, each LFE is a distributed forwarding element that spans multiple host computers and is implemented by configuring multiple MFEs on the multiple host computers. To this end, in some embodiments, each MFE or a module associated with the MFE (e.g., a VXLAN tunnel endpoint (VTEP)) is configured to encapsulate the LFE's data messages with an overlay network header that contains a virtual network identifier (VNI) associated with the overlay network. As such, in the following discussion, the LFE is referred to as an overlay network fabric that spans multiple hosts.

[0034] In some embodiments, the LFE also spans configured hardware forwarding elements (e.g., top-of-rack switches). In some embodiments, the collection of LFEs includes a logical switch implemented by configuring multiple software switches or related modules on multiple host computers. In other embodiments, the LFE can be other types of forwarding elements (e.g., logical routers), or any combination of forwarding elements (e.g., logical switches and / or logical routers) that form a VPC (e.g., a logical network) or part thereof. There are many examples of LFEs, logical switches, logical routers, and logical networks, including examples provided by VMware's NSX network and service virtualization platform.

[0035] The novel architecture also includes a collection of controller computers that configure host computers to execute service instances to provide services for a virtual private cloud including a DCN executed on the host computers. In some embodiments, the service instances are used to provide distributed services, including distributed edge services for data messages entering or leaving a VPC implemented in an AZ. In some embodiments, the distributed edge services are provided at a distributed logical router located at the logical edge of the VPC (i.e., a logical router that processes data messages entering or leaving the VPC) as part of logical router processing.

[0036] Figure 1 Illustrated is an exemplary network 100 in which the novel architecture is implemented. Figure 1 The diagram illustrates a network 100 that includes an external network 101 connecting a collection of machines 102 outside of an availability zone 105 (e.g., a data center) with guest machines (GMs) 125 executing on multiple hosts 120 in the availability zone 105 (e.g., a public cloud data center or a collection of data centers). The availability zone 105 includes a collection of gateway devices 110 through which data messages enter and exit the availability zone 105. The availability zone 105 also includes a collection of controller computers 140 for a VPC that provide configuration instructions to the hosts 120. In some embodiments, the controller computers 140 execute in different availability zones and communicate with the hosts 120 through a management plane (e.g., a management VPC) that spans multiple availability zones.

[0037] Figure 1 Also illustrated are a set of route servers 130 that act as route learning (e.g., BGP) agents for other routers in AZ 105 and connections. Figure 1 The intermediate architecture 150 of the different elements shown in FIG. Figure 1 A collection of host computers 120 is illustrated, representing a collection of physical resources of an AZ 105 on which at least one VPC is implemented. Host computers 120 are represented by host 120a, which includes a managed forwarding element 160a that connects components of host computer 120a to each other and to an intermediate fabric 150 for communicating with other host computers 120 and with external machines 102 in an external network 101. In some embodiments, MFE 160a is a software forwarding element that executes in the hypervisor of host 120a.

[0038] Host computers 120 execute guest machines 125 (e.g., DCNs, containers, pods, etc.), which, for simplicity, are assumed to belong to a single tenant's VPC. A VPC implemented across a collection of hosts 120 also includes a collection of services provided to the GMs 125 executing on the collection of hosts 120. In some embodiments, the GMs 125 are applications that service external requests (e.g., web servers) or support other GMs 125 (e.g., database servers).

[0039] In the illustrated example, the set of services for GM 125a includes a set of L4 services 124a and 129a, as well as a guest machine providing a set of L7 services 122a. In some embodiments, L4 services 124a and L7 service GM 122a are connected to the service slice of the VPC, while L4 service 129a is an inline L4 service for the destination GM in GM 125a's set. In some embodiments, the service slice is shared by multiple tenant VPCs using the same L7 service GM 122a. Because L7 service GM 122a is resource-intensive, executing a separate L7 service GM 122 for each tenant would unnecessarily consume resources.

[0040] L4 services 124a and 129a are shown as executing in kernel space, while L7 services are shown as executing in L7 service GM 122a in user space. In some embodiments, inline L4 services 129a include services such as firewall and NAT services that are part of the processing pipeline for a destination GM in the set of GMs 125a. In some embodiments, L4 services 129a are invoked from a port of a logical switching element or from a VNIC of a GM in the set of GMs 125a. For some services that are part of the processing pipeline for a particular destination GM (e.g., an L4 firewall), the service cannot be independently addressed by an external machine and does not need to have any network address advertised by routing machine 123a.

[0041] In some embodiments, the L4 services 124a include at least one of an L4 firewall and a load balancing service, and the L7 services provided in the L7 service guest machine 122a include at least one of a load balancing service, a distributed intrusion detection system (IDS) service, and a distributed intrusion protection system (IPS) service. For some L4 and L7 services, such as an inline NAT service using a specific external IP address or a load balancing service for a collection of computing nodes addressed using a specific virtual IP address (VIP) (either L4 or L7 load balancing), some embodiments advertise a network address associated with the service.

[0042] In some embodiments, the set of network addresses advertised for a particular service includes not only the specific address used by the service but also an additional network address that identifies a specific service instance (e.g., a service executing on a specific host computer) within a set of service instances that use the same network address to provide the distributed service. For example, a distributed NAT service that uses a specific IP version 4 (IPv4) address as its external address across multiple host computers can identify the individual host computers providing the service by having each host computer advertise an IP version 6 (IPv6) address that uniquely identifies the host computer within the AZ (or the distributed NAT service instance executing on the host computer). Similarly, a distributed load balancing service can identify different hosts providing the service by having each host computer advertise an IP version 6 (IPv6) address that uniquely identifies the host computer within the AZ (or the distributed load balancing service instance executing on the host computer). In some embodiments, the IPv6 address advertised for a particular distributed service instance is based on the IPv4 address associated with the distributed service, so that upon receiving a data message addressed to the IPv4 address associated with the distributed service, a gateway device can generate an IPv6 address that identifies the specific host that should receive the data message. Specific examples of using such IPv6 addresses are provided in U.S. patent application Ser. No. 16 / 931,196, filed on July 16, 2020, which is incorporated herein by reference.

[0043] Host computer 120 also includes components for interacting with a collection of controller computers 140. Host computer 120a includes a configuration data store 128a that stores configuration data received from collection of controller computers 140. Local controller 126a uses the information stored in configuration data store 128a to configure GM 125a, routing machine 123a, and services 124a and 122a. In some embodiments, the information stored in configuration data store 128a includes the L2 and L3 addresses associated with GM 125a, routing machine 123a, and services 124a and 122a, as well as the identification of services associated with each GM 125a (e.g., policies defining when a service should process incoming or outgoing data messages). In other embodiments, the local controller also configures MFE 160a, distributed routing (DR) instance 121a, and VXLAN tunnel endpoint (VTEP) 170a to implement a VPC.

[0044] Figure 1 An embodiment is illustrated in which a host computer executes components of a VPC for a single tenant. In an alternative embodiment, components of two or more VPCs operating on behalf of one or more tenants are executed in a single host computer. Figure 2 and 3Two possible embodiments of executing components for multiple VPCs of one or more tenants are illustrated. Figure 2 The diagram shows Figure 2 Example logical networks (VPCs) implemented for two VPCs (i.e., Tenant A and Tenant B) and components such as Figure 1 1. The two VPCs are shown in FIG. 20 ...

[0045] The VPC (or logical network) 200 includes a logical router 271, which, in the depicted embodiment, is an edge (or gateway) logical router for the VPC. The VPC includes a first logical switch 272, which is used to connect a collection of guest machines 225 (e.g., DCNs, VMs, containers, pods, etc.) to the logical router 271. In the depicted embodiment, the VPC also includes a separate service logical switch 273 that connects the logical router 271 to the L7 service GM 222. As discussed above, in some embodiments, the L7 service GM 222 executes multiple applications, programs, or containers to provide multiple different services for the VPC.

[0046] The logical view illustrates a set of inline L4 services 224 provided between the logical switch 272 and the GM 225, and a set of inline L4 services 229 provided between the logical router and the service logical switch. In some embodiments, these services are logically part of the processing pipeline of the logical router 271 or the logical switches 272 and 273 and are not logically distinct. In some embodiments, the L4 services 224 are logically applied at a virtual interface attached to the GM 225 and include at least one of a distributed firewall service and a distributed network address translation service. In other embodiments, additional separate service nodes (e.g., third-party service nodes) are invoked by the logical router 271 or the logical switches 272 and 273. While the VPC shown includes only a single logical router and two logical switches, other VPCs in other embodiments include multiple layers of logical routers and additional logical switches, and different VPCs spanning a particular host computer will have different logical components or logical structures.

[0047] Physical view 202 illustrates the Figure 1 Similar components have similar numbering, but the components for tenants A and B's VPCs are distinguished by using "A" and "B" and different boundary shading. Figure 2The routing machine 223 in the multi-tenant (or multi-VPC) embodiment depicted in FIG is a multi-tenant routing machine that maintains different virtual routing and forwarding (VRF) tables 280A and 280B for tenants A and B, respectively. In some embodiments, the different VRF tables are associated with different route distinguishers and route targets for facilitating packet forwarding using Multi-Protocol Label Switching (MPLS). Additionally or alternatively, in some embodiments, the different VRF tables 280 are associated with different virtual network identifiers (VNIs) for implementing VXLAN encapsulation. The host computer 220 also implements a single VTEP 270 that serves as a tunnel endpoint for data messages for multiple tenants.

[0048] In the illustrated embodiment, host computer 220 executes separate guest machines 225A and 225B (DCNs, VMs, containers, pods, etc.), distributed router instances 221A and 221B, L4 services 224A, 224B, 229A, and 229B, and L7 service GMs 222A and 222B for each tenant (e.g., VPC). Figure 2 In the example embodiment, a managed switching element (MSE) 260 is used to connect the different components of each VPC. In some embodiments, the MSE 260 is configured to implement the logical switching element (e.g., logical switch 272 and service logical switch 273) of each VPC. In some embodiments, the logical switching element spans multiple host computers and is implemented by an MSE on each host computer in the span of the logical switching element. Similarly, the logical router 271 of each VPC spans multiple hosts and is implemented by a distributed router instance 221 for the VPC on each host. In some embodiments, the various elements of the host computers 220 are configured based on configuration data stored in the configuration data storage device 228, which is received from a controller computer cluster that generates configuration data for implementing a set of VPCs across multiple host computers including the host computer 220.

[0049] In addition to the separate VPC components, host computer 220 also executes components that, in some embodiments, are used by all tenants. For example, in some embodiments, routing machine 223, VTEP 270, managed switching element (MSE) 260, local controller 226, communication agent 227, and configuration data storage 228 are commonly used by all tenants. In some embodiments where a single VTEP 270 executes on the host computer, gateway device 110 establishes a tunnel to VTEP 270 to forward packets to L7 services 222A and 222B and GMs 225A and 225B.

[0050] Figure 3Illustrated is a management view 301 (or logical view) of exemplary logical networks (VPCs) 300A and 300B and components such as those that execute multiple VPCs. Figure 1 A physical view 302 of a representative host computer 320 in a network such as network 100 is shown. Figure 3 The host computer 320 implements Figure 2 The exemplary logical network (VPC) is similar to the logical structure of a VPC, but instead of a service logical switch being (1) logically separate for each VPC and (2) exclusively reachable through a logical router, the service logical switch (1) is a logical switch reachable by both VPCs 300A and 300B, and (2) a service VTEP 370b is used to forward incoming data messages addressed to the L7 service directly to the service logical switch 373. In some embodiments, a first VTEP 370a is used for traffic to GMs 325A and 325B and a second VTEP 370b is used for traffic to the L7 service in the multi-context L7 service DNC 350, a gateway device of the AZ establishes split tunnels to each VTEP of the host computer, a first tunnel for forwarding traffic to the GM and a second tunnel for forwarding traffic to an externally addressable L7 service instance in the multi-context L7 service DCN 350. In some embodiments, packets sent from the L7 service to other GMs in the VPC are forwarded to the distributed router using the overlay network via the tunnel between VTEPs 370b and 370a. for Figure 2 In some embodiments, the two VPCs belong to two different tenants, while in other embodiments, the two VPCs belong to a single tenant (e.g., VPCs for two different departments of the same company).

[0051] In some embodiments, the service logical switch (e.g., service plane) is a separate logical switch or other structure that allows each VPC's logical forwarding element (e.g., logical router, logical switch, etc.) to access L7 services (e.g., L7 firewall, load balancing, intrusion detection, etc.). In some embodiments, the L7 service is reached by encapsulating the data message in a header that identifies the logical service switch (or other structure) and the source context (e.g., VPC, logical forwarding element, tenant, etc.) to reach the correct L7 service instance 351 and return to the correct VPC or logical forwarding element. In some embodiments, the L7 service is called as part of the processing pipeline for the logical forwarding element and is returned to the logical forwarding element to complete the logical processing pipeline. In some embodiments, invoking the L7 service as part of the processing pipeline includes intercepting traffic requesting the L7 service, encapsulating the traffic (e.g., using Generic Network Virtualization Encapsulation (GENEVE)) for delivery to the L7 service, wherein the traffic has a logical context that identifies the original traffic (e.g., tenant ID, VNI of the logical forwarding element, etc.) so that the L7 service can provide the service defined for the logical context and return the traffic to the correct logical switch or processing pipeline.

[0052] In other embodiments, instead of providing a logical switch (e.g., a service plane) to which each VPC is connected, the L7 services DCN 350 has a connection to each logical switch for which it provides L7 services. However, because the (re)direction of traffic to the L7 services is performed in kernel space, other configurations are used in other embodiments as will be appreciated by those of ordinary skill in the art.

[0053] Figure 3 The illustrated embodiment illustrates an embodiment in which L4 services 324A and 324B associated with a service logical switch 373 are implemented as a multi-context L4 service 340, which provides L4 services to different tenants based on a tenant or logical network identifier (e.g., a VNI included in a VXLAN header) included in a packet. Furthermore, a collection of L7 services 351A and 351B for tenants A and B, respectively, are executed within a single multi-context L7 service DCN 350. To facilitate providing L7 services to different VPCs within the same L7 service DCN 350, the illustrated embodiment also implements a service VTEP 370b within the L7 service DCN 350 to differentiate traffic for different tenants or VPCs. In some embodiments, the multi-context L7 service DCN 350 is one of a virtual machine or pod that includes multiple L7 service containers or executes multiple L7 service applications for different tenants. Figure 3 Appears in and Figure 1 and 2 Other elements with similar numbers also have similar functions and features.

[0054] Figure 4 A process 400 for configuring components of a host computer (e.g., host computer 120) to implement a VPC within an AZ is conceptually illustrated. In some embodiments, process 400 is performed by a hypervisor of the host computer based on configuration information received from a controller computer (e.g., controller computer set 140). In some embodiments, process 400 is an initialization process performed when the host computer is first configured to implement a VPC. Process 400 begins by receiving (at 410) configuration information from the controller computer set. In some embodiments, the configuration information includes a configuration of a collection of components for implementing a VPC in collaboration with other host computers in the AZ (and possibly other AZs). In some embodiments, the components for implementing a VPC include a routing machine that is used to advertise network addresses associated with the VPC to other routers in the AZ and receive advertisements from other routers in the AZ. In some embodiments, the components for implementing a VPC include a data compute node (DCN) of the VPC and service instances within the VPC that provide services for the DCN of the VPC.

[0055] Based on the received configuration information, process 400 instantiates (at 420) a routing machine on the host computer. As used in this document, the term instantiation is used to refer to the process of making a network component (e.g., a DCN, a service, a logical network element, etc.) added to or available on a host computer. In some embodiments, the routing machine is instantiated on each host computer because it is first configured to implement a VPC, and then a single routing machine is used by each VPC subsequently implemented on the host computer. Thus, the routing machine is a multi-tenant (or multi-tenant capable) routing machine and is configured to advertise routes for VPC components for different tenants using at least one standard route advertisement protocol (e.g., MP-BGP). In some embodiments, the routing machine is one of a standard virtual machine, a lightweight virtual machine, a container, or a pod.

[0056] Before or after instantiating the routing machine on the host computer (at 420), the process identifies (at 430) a set of VPC components to be executed on the host computer. In some embodiments, the identified components include a set of DCNs for the VPC, a set of logical forwarding elements (LFEs) for the VPC, and service compute nodes (e.g., virtual machines, containers, or pods). In some embodiments, the configuration information also includes information about the connections between the DCNs, service compute nodes, and LFEs. In some embodiments where instantiating the routing machine occurs after identifying (at 430) the set of VPC components, identifying the set of VPC components includes identifying that the routing machine needs to be instantiated.

[0057] After identifying (at 430) the components of the VPC to be executed on the host computer, process 400 instantiates (at 440) the identified DCNs for the VPC. In some embodiments, the identified DCNs include DCNs of the VPC associated with providing a specific service to machines in the external network. In some embodiments, the DCNs associated with providing a specific service to machines in the external network include at least one of a DCN that is directly addressable from the external machine and a DCN that is addressable at a network address (e.g., a VIP) associated with a set of DCNs that provide a specific service executed on a set of host computers in one or more AZs. In some embodiments, other identified DCNs include DCNs that provide services to other DCNs in the VPC that are addressable within the private address space of the VPC.

[0058] In addition to identifying (at 430) the components of the VPC, process 400 also identifies (at 450) a set of services associated with the identified VPC components. In some embodiments, the services are identified based on a policy of the VPC that defines a set of services associated with a DCN in the VPC. In some embodiments, the identified services include different types of services, such as firewall services, NAT services, load balancing services, IDS services, and IPS services. In some embodiments, not all services associated with the VPC are required for the set of DCNs executing on a particular host computer, and the set of identified services is a subset of the services associated with the particular DCN executing on the host computer. In some embodiments, the same service (e.g., a firewall service) is called from multiple ports of a single logical switching element or even from ports of multiple logical switching elements of the VPC.

[0059] Once the services associated with the DCN are identified (at 450), the process instantiates (at 460) the identified services. In some embodiments, the identified services include services that are instantiated in different ways. For example, a service provided as an inline service (e.g., provided as part of a processing pipeline) is added as a component (e.g., a program or function call) that can be called from ports of multiple logical switching elements. For example, Figure 1 The L4 services 129a represent a collection of inline services that can be implemented using function calls. In some embodiments, the identified services include services provided based on information at layers 1-4 of the OSI model rather than at higher layers (e.g., L4 services 124a and 129a), which are instantiated for execution in the kernel space of the host computer.

[0060] Other identified services include services provided based on information from Layer 7 of the OSI model (e.g., L7 services) and are instantiated for execution in the user space of the host computer. In some embodiments, the L7 services run in one of a virtual machine, lightweight virtual machine, container, or pod in the user space of the host computer. In some embodiments, multiple L7 services execute in the same virtual machine (e.g., as separate applications within the VM) or pod (e.g., as separate containers within the pod). In some embodiments, other identified resource-intensive services are also configured to execute in the user space of the host computer along with the identified L7 services. In some embodiments, virtual machines or containers executing L7 services are not instantiated (or added) until services requiring VMs or containers are identified. In some embodiments, running only services and VMs or containers identified as necessary conserves host resources for the DCN and services executing on the host. In some embodiments, a local controller (e.g., local controller 126) is responsible for instantiating and configuring the DCN, services, LFEs, and service VMs (or containers / pods) based on configuration data stored in a local storage device (e.g., configuration data storage device 128). In some embodiments, a service executing within a VM or container (or an nginx data path associated with a VM or container) is configured using a virtual machine communication interface (VMCI) agent (e.g., communication agent 127) that provides configuration information from a local storage device.

[0061] After (or in conjunction with) instantiating (at 440 and 460) the identified VPC components, process 400 provides (at 470) routing information for the instantiated components and services of the VPC to a routing instance on the host computer. In some embodiments, providing routing information includes causing the instantiated components to announce their network addresses to the routing machine (e.g., using Gratuitous Address Resolution Protocol (GARP) data messages). In other embodiments, providing routing information to the routing machine additionally or alternatively includes providing routing information from a local storage device via a VMCI agent (e.g., communication agent 127), which retrieves configuration information from the local storage device and provides it to the routing machine. In some embodiments, the routing instance registers with a local configuration data store (via the VMCI agent) or with the VMCI agent to receive updates to configuration information related to the DCN and services operating on the host computer.

[0062] In some embodiments, a novel method for identifying a network address available at a host computer is presented. A host computer (e.g., a routing machine) executing a routing instance detects that a network address has become available on the host computer. In some embodiments, detecting that the network address has become available includes detecting that at least one of a DCN or a service has been added to the host computer. The routing instance then identifies a network address associated with the detected DCN or service added to the host. In some embodiments, the identified network address is an address used by other routers in the AZ. In other embodiments, the identified network address is an address used by routers and machines in a network external to the AZ. The routing instance then advertises the identified network address to at least one routing server for advertising to other routers in the AZ.

[0063] Figure 5 A process 500 is conceptually illustrated for detecting that a DCN has been added to a host computer and announcing the network address associated with the DCN to other routers in an AZ. In some embodiments, process 500 is performed by a routing machine (or a free-range routing daemon of the routing machine) on a host computer that communicates with a set of routing servers using a routing learning protocol (e.g., BGP, MP-BGP, IS-IS, etc.). In some embodiments, the routing machine registers to receive notifications when a DCN is added to or removed from the host computer on which the routing machine is executing. In some embodiments, the registration (e.g., via an API call) is performed with a local controller or VMCI agent executing on the host computer that retrieves configuration data about the DCN on the host computer from a local configuration data store.

[0064] In some embodiments, process 500 begins by receiving (at 510) a notification that a DCN has been added to a host computer. In some embodiments, the notification includes a collection of information about the added DCN that is used to generate routing entries in a routing machine. Additionally, in some embodiments, the collection of information is used to generate advertisements to other routers in the AZ, as described below. One of ordinary skill in the art will recognize that in some embodiments, registration is not required, and the information received in operation 510 is received from the DCN after being added to the host computer (e.g., via Gratuitous Address Resolution Proxy (GARP) information).

[0065] After receiving (at 510) a set of information regarding the added DCN, a network address associated with the DCN is identified (at 520). In some embodiments, identifying the network address includes identifying information from the received set of information related to the DCN's network address. In some embodiments, the set of information includes the network address of the DCN within the VPC for which it is being added, a VPC identifier (e.g., a virtual network identifier (VNI)), and the next hop toward the added DCN. Once the network address is identified (at 520), the routing machine identifies (at 530) a set of network addresses for advertising the added DCN. In some embodiments, the identified set of network addresses for advertising the DCN to other routers in the AZ includes a first network address for advertising the DCN's availability to other components of the VPC and a second network address used by machines outside the VPC. In some embodiments, the second network address is a network address used by the DCN to make itself available to external machines. In some embodiments, the second network address is a network address associated with a service added to a host computer to serve the added DCN.

[0066] In some embodiments, the set of identified network addresses is a single network address (e.g., a / 32 IP address) associated with additional information identifying the DCN's availability to other components of the VPC and to machines outside the VPC. In some embodiments, the additional information (e.g., context information) includes at least one of a virtual network identifier (VNI) or logical network identifier (LNI) associated with the VPC associated with the DCN, an associated tenant identifier (TID), and a route distinguisher value used by other routers to (1) identify advertised routes as being associated with a particular VRF context and (2) distinguish routes for the added DCN from routes for other DCNs in different VPCs having the same network address (e.g., DCNs that share IP addresses in a private IP address space used by multiple VPCs). In some embodiments, the identified network address also includes information identifying the host computer on which the DCN is executing, such as a VTEP network address or IP address of the host computer. In some embodiments, the context information is generated by the host computer on which the newly added DCN is executing based on information stored at the host computer (e.g., configuration information stored in a configuration data storage device).

[0067] In some embodiments, additional information is sent to the routing server along with the identified network addresses, as described further below. In some embodiments, this additional information includes context attributes associated with the machine to which the identified network addresses belong. Examples of context attributes in some embodiments can include any attribute that is not an L2, L3, or L4 header value. For example, in some embodiments, the context attribute sent to the route reflector along with the identified network addresses is a logical port identifier. In some embodiments, the logical port identifier is used to resolve forwarding and service policies (e.g., security policies designated using logical port groups to which the logical port identifier is added). In some embodiments, the logical port identifier is identified for a DCN with a newly added IP address (e.g., for a newly added DCN or a new IP address added to an existing DCN). In some embodiments, the context attribute is included in the GENEVE header of the advertisement.

[0068] Once the set of addresses to be advertised has been identified (at 530), the set of identified network addresses is advertised (at 540) to other routers in the AZ in order to allow data messages to be directed to the DCN and the process ends. In some embodiments, advertising the set of network addresses to other routers is accomplished by advertising the set of network addresses to at least one of the set of routing servers used to propagate routing information within the AZ. In some embodiments, advertising the set of network addresses to the set of routing servers (route reflectors) includes sending additional information (e.g., context information) along with the advertised set of network addresses to the routing servers. In some embodiments, the additional information is included in the GENEVE (or other encapsulation) header of the advertisement of the set of identified network addresses.

[0069] In some embodiments, the set of routing servers receive and send advertisements using standard protocols for communicating routes between routers, such as BGP or MP-BGP, IS-IS, IBGP, eBGP, OSPF, etc., and use these standard protocols instead of the controller computer cluster updating routing information for each host computer as DCNs or services are added or removed from the VPC, or migrated from one host computer to another within the VPC. In some such embodiments, using a mature protocol such as BGP improves convergence speed and reduces the workload of the controller computers. In addition, using routing servers to communicate with routers in the AZ reduces the amount of data needed to update the routers in the AZ compared to a full mesh architecture between all routers in the AZ.

[0070] Figure 6A similar process 600 is conceptually illustrated for detecting that a DCN has been removed from a host computer and withdrawing the network address associated with the removed data compute node from other routers in the availability zone. In some embodiments, process 600 is performed by the same routing machine that performed process 500. As described above, in some embodiments, when a DCN is removed from the host computer on which the routing machine is executing, the routing machine registers to receive notification. In some embodiments, the registration (e.g., via an API call) is performed with a local controller or VMCI agent executing on the host computer, which retrieves configuration data about the DCN on the host computer from a local configuration data store.

[0071] In some embodiments, process 600 begins by receiving (at 610) a notification that a DCN has been or will be removed from a host computer. Removing the DCN includes migrating the DCN to a different host and shutting down the DCN (e.g., removing the DCN from a VPC). In some embodiments, the notification includes a collection of information about the removed DCN that is used to generate a route entry in a routing machine. In some embodiments, the notification includes only a unique identifier (e.g., a universally unique identifier (UUID)) for the removed DCN, which the routing machine uses to identify the route entry and the advertisement made when the DCN was added to the host computer. Additionally, in some embodiments, the collection of information is used to generate an advertisement to other routers in the AZ to withdraw the route. Those skilled in the art will recognize that in some embodiments, registration is not required, and the information received in operation 610 is received from a local controller or VMCI agent as part of the process of removing the DCN from the host computer.

[0072] After receiving (at 610) a set of information regarding the removed DCN, a network address associated with the DCN is identified (at 620). In some embodiments, identifying the network address includes identifying information from the received set of information related to the DCN's network address. If the notification includes a unique identifier for the removed DCN, identifying the network address associated with the removed DCN includes identifying the network address using the unique identifier. In some embodiments, the set of information includes the network address of the DCN within the VPC associated with the removed DCN and a VPC identifier (e.g., VNI). Once the network address is identified (at 620), the routing machine identifies (at 630) a set of network addresses used to advertise the DCN when it is added to a host. In some embodiments, the identified set of network addresses used to advertise the DCN to other routers in the AZ includes a first network address used to advertise the DCN's availability to other components of the VPC and a second network address used by machines outside the VPC. In some embodiments, the identified set of network addresses is a single network address associated with additional information identifying the DCN's availability to other components of the VPC and to machines outside the VPC. In some embodiments, the additional information includes at least one of a VNI associated with the VPC associated with the DCN and a route distinguisher value used by other routers to identify advertised routes as being associated with a specific VRF context and to distinguish routes for the added DCN from routes for other DCNs in different VPCs that have the same network address (e.g., DCNs that share an IP address in a private IP address space used by multiple VPCs). In some embodiments, the identified network address also includes information identifying the host computer on which the DCN is executing, such as the host computer's VTEP network address or IP address.

[0073] Once the set of addresses has been identified (at 630), the identified set of network addresses is withdrawn (at 640) from the other routers in the AZ to stop directing data messages to the DCN located at the host computer and the process ends. In some embodiments, withdrawing the set of network addresses to the other routers is accomplished by withdrawing the set of network addresses from at least one route server in a set of route servers used to propagate routing information within the AZ.

[0074] In addition to detecting the addition of a DCN to a host computer, in some embodiments, the routing machine also detects the addition of a service to the host computer. Figure 7A process 700 is conceptually illustrated for detecting services that have been added to a host and announcing their network addresses to other routers in an AZ. In some embodiments, process 700 is performed not only when a service is added to a host computer, but also after the configuration of a service is updated (e.g., to update the assigned network address or port range). In some embodiments, process 700 is performed by a routing machine on a host computer that communicates with a set of routing servers using a routing learning protocol (e.g., BGP, MP-BGP, IS-IS, etc.). In some embodiments, the routing machine registers to receive notifications when services are added to or removed from the host computer executing the routing machine. In some embodiments, registration is performed (e.g., via an API call) with a local controller or VMCI agent executing on the host computer, which retrieves configuration data for the service on the host computer from a local configuration data store.

[0075] In some embodiments, process 700 begins by receiving (at 710) a notification that a service has been or will be added to a host computer. In some embodiments, the added service is added as a service instance of a distributed service that is provided at multiple host computers using the same network address (or set of network addresses). In some embodiments, the notification includes a collection of information about the added service (or service instance) that is used to generate routing entries in the routing machine. In addition, in some embodiments, the collection of information is used to generate advertisements to other routers in the AZ, as described below. It will be appreciated by those skilled in the art that, in lieu of registration notifications, in some embodiments, the local controller is configured to provide information related to the service added to the host computer to the routing machine. Alternatively, information about the service is received from the service instance after being added to the host computer (e.g., via Gratuitous Address Resolution Proxy (GARP) messages).

[0076] After receiving (at 710) a set of information about the added service, a network address associated with the service is identified (at 720). In some embodiments, identifying the network address includes identifying information in the received set of information associated with the network address associated with the service (e.g., an external IPv4 address associated with a NAT service or a virtual IPv4 address associated with a load balancing service). In some embodiments, the set of information includes a VPC identifier (e.g., a VNI), a set of ports assigned to service instances on host computers (e.g., for a distributed NAT service), and a next hop toward the added service. Once the network address is identified (at 720), the routing machine identifies (at 730) a set of network addresses for advertising the added service. In some embodiments, the identified set of network addresses for advertising the service to other routers in the AZ includes a first network address for advertising the availability of the service to other components of the VPC and a second network address for use by machines outside the VPC. In some embodiments, the identified set of network addresses for advertising the added service is based on the network addresses in the received set of network addresses and on additional information. For example, in some embodiments, a network address advertised by a particular host computer for a distributed NAT service (e.g., a distributed stateful service) executing on multiple host computers is an IPv6 network address prefix that combines an IPv4 address associated with the NAT service and a set of ports assigned to the distributed NAT service instance executing on the particular host computer. Generation of the IPv6 network address is used to identify the service instance executing on a particular host computer that stores state information for flows to avoid redirection within an AZ and is further described in U.S. patent application Ser. No. 16 / 931,196, filed on July 16, 2020, which is incorporated herein by reference.

[0077] For other services, the set of identified network addresses is, in some embodiments, a single network address (i.e., the VTP associated with the service) that is associated with additional information identifying the service's availability to other components of the VPC and to machines outside the VPC. In some embodiments, the additional information includes the VNI associated with the VPC associated with the DCN, a route distinguisher value used by other routers to identify advertised routes as being associated with a particular VRF context, and to distinguish routes for the added DCN from routes for other DCNs in different VPCs that have the same network address (e.g., DCNs that share IP addresses in the private IP address space used by multiple VPCs). In some embodiments, the identified network address also includes information identifying the host computer on which the service is executed, such as the host computer's VTEP network address or IP address.

[0078] In some embodiments, even some stateful distributed services are advertised as available on every host computer using the same network address. For example, in some embodiments, a stateful distributed load balancing service used to distribute requests received from clients in an external network relies on a collection of gateways in an AZ to consistently send the same flow to the same host computer, thereby providing distributed load balancing across the host computers providing the distributed load balancing service based on equal-cost multi-path (ECMP) operations performed by the gateway devices in the AZ. To enable this ECMP operation, in some embodiments, a routing machine on each host executing a distributed load balancer instance advertises the same VIP address as available, and the gateway devices in the AZ record the multiple advertised next-hop addresses as next hops that are as closely associated with the VIP as possible. For received data messages addressed to the VIP, the gateway devices in the AZ select a specific next hop using ECMP operations. In such embodiments, as the number of host computers providing the distributed load balancing service changes, an acceptable number of redirect operations may be required, making the effort of ensuring that different host computers can be deterministically identified for each flow (or data message) unworthy.

[0079] In embodiments that rely on ECMP for an AZ gateway device, after a routing table change at the AZ gateway (e.g., based on a change in the number of hosts providing distributed edge services), ECMP operations may direct data messages of at least some data message flows to a different host computer than the data messages received before the routing table change. To redirect these data messages to a host computer that maintains state information for the associated data message flow, some embodiments provide a redirection operation. In some embodiments, the host computer is aware of the algorithm or hash used by the ECMP operation of the AZ gateway, and upon receiving a data message for an established data message flow for which the host computer does not store state information, the host computer performs an ECMP operation based on a previous set of host computers to determine the host computer that stores the state information and redirects the data message to that host computer.

[0080] In other embodiments where the algorithm or hash of the AZ Gateway ECMP operation is unknown, each new data message stream received at a particular host computer triggers a notification sent to each other host computer providing edge services for the particular host computer to maintain state information for that data message stream. Alternatively, in some embodiments, the notification sent by the particular host computer is sent to a host computer selected using an operation known to each host computer (e.g., consistent hashing). After a membership change (i.e., a change in the number or identity of host computers providing edge services), the known operation is used to redirect data messages to a host selected using a known operation based on the previous membership. The selected host computer will then redirect the data message to the particular host computer that maintains state information based on the notification sent from the particular host computer.

[0081] Once the set of addresses to be advertised has been identified (at 730), the identified set of network addresses for advertising the availability of the service is advertised (at 740) to other routers in the AZ to allow data messages to be directed to the service and the process ends. In some embodiments, advertising the set of group network addresses to other routers is accomplished by advertising the set of network addresses to at least one route server in a set of route servers used to propagate routing information within the AZ. For services available to machines in networks outside the AZ, the set of route servers advertises the network addresses to a set of gateway devices in the AZ that provide access to external networks.

[0082] In some embodiments, a collection of route servers uses a standard protocol, such as BGP or MP-BGP, to receive and send announcements. These standard protocols are used instead of a cluster of controller computers to update routing information for each host computer when a DCN or service is added or removed from a VPC or when a host computer is migrated from one VPC to another. In some such embodiments, the use of a mature protocol, such as BGP, increases convergence speed and reduces the workload on the controller computer. Furthermore, using route servers to communicate with routers in an AZ reduces the amount of data required to update routers in the AZ compared to a fully meshed architecture between all routers in the AZ.

[0083] Figure 8A similar process 800 is conceptually illustrated for detecting that a service has been removed from a host computer and withdrawing the network addresses associated with the removed service from other routers in the availability zone (and external network). In some embodiments, process 800 is performed by the same routing machine that performs process 700. As noted above, in some embodiments, the routing machine registers to receive notifications when a service is removed from the host computer on which it executes. In some embodiments, the registration (e.g., by an API call) is made to a local controller or VMCI agent executing on the host computer, which retrieves configuration data about the services on the host computer from a local configuration data store.

[0084] In some embodiments, process 800 receives (at 810) a notification that a service has or will be removed from a host computer. In some embodiments, removal of a service occurs when the DCN for the VPC that includes the service is no longer executing on the host computer, when the DCN executing on the computer does not require the service, or a service instance for a particular service is removed from a host computer (either migrated to another host or shut down to reduce service capacity). In some embodiments, the notification includes a set of information about the removed service that can be used to generate routing entries in the routing machine. In some embodiments, the notification includes information that the routing machine can use to identify the routing entries and the announcements made when the service was added to the host computer. In addition, in some embodiments, the set of information is used to generate announcements to other routers in the AZ to withdraw the routes (set of network addresses). Those of ordinary skill in the art will recognize that, in some embodiments, registration is not required and the information received in operation 810 is received from the local controller or VMCI agent as part of the process of removing the service from the host computer.

[0085] After receiving (at 810) the set of information about the removed service, the network address associated with the service is identified (at 820). In some embodiments, identifying the network address includes identifying information in the received set of information associated with the network address associated with the service. If the notification includes identifying information for the removed service, then identifying the network address associated with the removed service includes using the identifying information to identify the network address. In some embodiments, the set of information includes a VPC identifier (e.g., VNI), a set of ports assigned to instances of the service on the host computer (e.g., for a distributed NAT service), and a set of ports toward the added service. Once the network address is identified (at 820), the routing machine identifies (at 830) a set of network addresses for advertising the service when the service is added to the host machine. In some embodiments, the identified network address also includes information identifying the host computer on which the service is executed, such as a VTEP network address or IP address of the host computer.

[0086] Once the set of addresses for advertising the service has been identified (at 830), the set of identified network addresses is withdrawn (at 840) from other routers in the AZ in order to stop data messages from being directed to the DCN at the host computer and the process ends. In some embodiments, withdrawing the set of network addresses from other routers is accomplished by withdrawing the set of network addresses from at least one of the set of routing servers used to propagate routing information within the AZ.

[0087] In some embodiments, the set of routing servers receive and send advertisements using standard protocols such as BGP or MP-BGP, and these standard protocols are used instead of the controller computer cluster to update routing information for each host computer when a DCN or service is added or removed in a VPC or migrated from one host computer to another in a VPC. In some such embodiments, using a mature protocol such as BGP improves convergence speed and reduces the workload of the controller computers. In addition, using routing servers to communicate with routers in the AZ reduces the amount of data needed to update routers in the AZ compared to a full mesh architecture between all routers in the AZ.

[0088] In some embodiments, the set of routing servers (e.g., routing servers 130) receive information about Figures 5-8 the advertised service. Figure 9Conceptually, a process 900 for a routing server is illustrated that receives routing advertisements and generates routing advertisements to other routers based on the received advertisements. In some embodiments, the process 900 is performed by a routing server in a set of routing servers for an AZ, each routing server receiving advertisements from a different set of allocated routers in the AZ and generating advertisements for a different set of assigned routers in the AZ. In some embodiments, the different sets of assigned routers overlap to ensure that each router will receive an advertisement even in the event of a routing server failure. In other embodiments, the different sets of assigned routers are different to avoid conflicting advertisements.

[0089] The process 900 begins by receiving (at 910) a set of advertisements for addresses in the AZ from a plurality of routers in the AZ. In some embodiments, the plurality of routers are in a set of assigned routers from which the routing server receives advertisements and to which the routing server generates and sends advertisements. In some embodiments, the received advertisements include virtual IP addresses associated with L4 and L7 services, as well as IP addresses associated with L7 service DCNs on which L7 services execute. In some embodiments, the advertisements identify the IP addresses or VTEPs of L7 service DCNs as addresses on which the VIPs of L7 services are available. Gateway devices of the AZ receive multiple addresses associated with a VIP (e.g., multiple VTEP addresses or L7 service DCN IP addresses), in some embodiments including each received address as a possible next hop to be selected by a load balancing operation (e.g., an equal cost multi-path (ECMP) operation). In some embodiments in which the number of service instances is greater than the AZ gateway device can use in an ECMP operation, an additional load balancing operation is introduced before the service instances, or an additional VIP is associated with the service to allow the ECMP to perform for each VIP on a smaller number of service instances.

[0090] After receiving (at 910) the advertisements from the plurality of routers in the AZ, the routing server exchanges (at 920) routing information with other routing servers to maintain accurate routing information for the AZ. In some embodiments, the exchange of routing between routing servers is performed as each routing server receives an advertisement, such that operation 920 includes only sending information about received advertisements and receiving information from other routing servers independent of the exchange in operation 920. In some embodiments, the exchange occurs through a set of advertisements (e.g., BGP or MP-BGP advertisements). In other embodiments, control plane messages are used to synchronize the routes received at each routing server.

[0091] After exchanging routing information with other route servers (at 920), process 900 generates (at 930) at least one aggregate advertisement based on at least a plurality of received advertisements. In some embodiments, the at least one aggregate advertisement is also based on routing information received from other route servers. In some embodiments, the aggregate advertisement includes only the set of routes not included in previous advertisements, while in other embodiments, all active routes are advertised to identify routes as active and avoid timing-based invalidation of active routes or cleanup operations at routers. In some embodiments, the process generates (at 930) multiple aggregate advertisements. In some embodiments, the multiple aggregate advertisements include advertisements for different routers executing on different host computers, which include only routes relevant to the host computers. In other embodiments, a different aggregate advertisement is generated for each unique route distinguisher. In some embodiments, the multiple aggregate advertisements include different advertisements for different subsets of routers within a designated set of routers executing on host computers in the AZ. In some embodiments, the multiple aggregate advertisements include different advertisements for different subsets of routers in the AZ, such as different designated sets of routers executing on host computers in the AZ, sets of routers in the intermediary fabric, and sets of gateway routers in the AZ, including routes relevant to each set of routers.

[0092] Once at least one aggregate advertisement is generated (at 930), a set of routers in the AZ are identified (at 940) for receiving the at least one generated aggregate advertisement. In some embodiments, identifying the set of routers in the AZ includes identifying a set of routers to receive each generated aggregate advertisement. As described above, in some embodiments, different aggregate advertisements are generated for different sets of routers, and identifying the set of routers includes identifying a set of routers to receive each generated aggregate advertisement. In embodiments where a single aggregate advertisement is generated, identifying the set of routers includes identifying all routers in the AZ's set of routers for the routing server's assignment. Once the set of routers to receive the generated aggregate advertisement is identified (at 940), the aggregate advertisement is sent to the identified set of routers in the AZ and the process ends. In some embodiments, the advertisement is made using a standard advertisement protocol (e.g., BGP, MP-BGP, IS-IS, etc.).

[0093] In some embodiments, the routing server is responsible for monitoring the availability of services in the AZ (e.g., the health of service instances) and handling service instance failure announcements. Alternatively or additionally, in some embodiments, a collection of controller computers or routing instances executing on the same host computer as the service instance is responsible for monitoring the availability of service instances executing on a particular host computer and handling service instance failure announcements. Figure 10A process 1000 for monitoring the health of a single service instance and announcing failures of the service instance to routers in an AZ is conceptually illustrated. Process 1000 begins by establishing a monitoring session with the service instance. In some embodiments, the monitoring session is a Bidirectional Forwarding Detection (BFD) session or any other monitoring protocol known to those skilled in the art.

[0094] Process 1000 then determines (at 1020) whether the service is still available via the monitoring protocol. In some embodiments, if a network element (e.g., a routing server, a routing instance, or a controller computer) receives a heartbeat packet or other indication that the service instance is available based on the monitoring protocol being used, then the service is determined to be still available. If it is determined (at 1020) that the service is still available, then the process continues to determine whether the service is available. This determination can be made periodically based on the period of expected heartbeat packets or other availability indications (e.g., every 50ms, 100ms, etc.). If it is determined (at 1020) that the service is unavailable (e.g., based on the criteria used for the monitoring protocol), then an advertisement is generated to withdraw routes associated with the service instance from routers in the AZ.

[0095] After generating (at 1030) the withdrawal advertisement, the generated withdrawal advertisement is used to announce the withdrawal of the route associated with the service instance from routers in the AZ. In some embodiments where the routing instance or controller computer monitors the availability of the service instance, the advertisement is made to at least one routing server associated with the routing instance or controller computer. In such embodiments, the routing server then propagates the advertisement to other routing servers and routers in the AZ. In embodiments where the routing server monitors the health of the service instance, the routing server announces the withdrawal of the route associated with the service instance to other routing servers and routers in the AZ itself.

[0096] In some multi-tenant embodiments, the routing instance is a multi-tenant routing instance that maintains separate virtual routing and forwarding (VRF) contexts for each tenant. In some embodiments, the routing instance is a BGP instance. In some embodiments, the routing instance is a multi-protocol BGP instance (MP-BGP) that supports multiple addressing protocols (e.g., Multiprotocol Label Switching (MPLS), BGP Labeled Unicast (BGP-LU), Segment Routing (SR)). In some embodiments, different VRF contexts are distinguished by a route distinguisher, and a route target is used to identify advertisements associated with each VRF context.

[0097] Figure 11 A process 1100 for updating routing information at a multi-tenant routing instance is conceptually illustrated. Figure 12Multi-tenant MP-BGP routing instance 1223 is illustrated receiving routing advertisements from route server 1230 at a free range routing (FRR) daemon (ie, MP-BGP daemon) and updating routing tables for VRFs 1280A-1280D. Figure 12 Will be discussed Figure 11 1-4) are associated with at least one VPC in an AZ. In some embodiments, process 1100 is performed by a multi-tenant MP-BGP routing instance (e.g., routing instance 1223 or FRR daemon 1290) executing on a host computer. Process 1100 begins (at 1110) by receiving an advertisement (e.g., MP-BGP advertisement 1231) of routes for a set of addresses (e.g., routes 1-4) associated with at least one VPC in an AZ. In some embodiments, the sets of advertised routes associated with different VPCs are each advertised in separate advertisements. In other embodiments, a single advertisement (e.g., MP-BGP advertisement 1231) includes a set of addresses for different VPCs, where the advertisement for each different VPC is associated with a different VPC identifier (e.g., route distinguisher).

[0098] After receiving (at 1110) the advertisement, a virtual routing and forwarding (VRF) context associated with each advertised address is identified (e.g., by the multi-tenant MP-BGP routing instance 1223 or the FRR daemon 1290). In some embodiments, identifying the VRF context includes identifying a route distinguisher associated with the advertised address and a route target associated with each VRF context. Figure 12 As shown in , a single VRF context can be associated with multiple route discriminators (e.g., by specifying multiple import route targets). For example, VRFs 1280A and 1280B are both associated with the same tenant and are each associated with the route targets of two VRFs associated with the tenant's VPC (i.e., route targets 65000:100 and 65000:200). Other VRF contexts are associated with a single route target (e.g., 65000:300 or 65000:400 for VRFs 1280C and 1280D, respectively).

[0099] After identifying (at 1120) the VRF associated with each advertised route, the routing table of each identified VRF is modified (at 1130) with the associated advertised route. In some embodiments, the modification is to remove the route in the case of a withdrawn advertisement and to add the route in the case of an advertisement that adds a new route to the address. For example, advertisement 1231 includes routes 1 and 2 associated with route distinguishers 65000:100 and 65000:200, respectively, which are associated with Tenant 1's VPC. Based on the route distinguishers, VRFs 1280A and 1280B, which specify import route targets 65000:100 and 65000:200, add routes 1 and 2. Based on the advertisement for route 3 being associated with route distinguisher 65000:300, route 3 is added to the routing table of VRF 1280C based on the specified import route target 65000:300. Furthermore, based on the advertisement for route 4 being associated with the route distinguisher 65000:400, route 4 is added to the routing table of VRF 1280D based on the specified import route target 65000:400.

[0100] After modifying (at 1130) the routing table for the identified VRF, the modified routing table is provided (at 1140) to a local storage device (e.g., a configuration data storage device) for access by a router (e.g., a logical router, a software router, etc.) to update its routing table in order to make forwarding decisions for data messages processed by the router, and the process ends. In some embodiments, providing the routing table to the local storage device includes providing the routing table to a communication agent (e.g., a VMCI agent) that mediates between the routing instance and the local storage device. In some embodiments, each router registers with the local storage device or a local controller that provides access to the local storage device to receive updates to a specific routing table (e.g., a specific VRF routing table). In some embodiments, registering with the local storage device includes initiating a long pull to receive the updated routing table.

[0101] Upon receiving a data message that enters a VPC and is destined for a set of one or more DCNs in the VPC, a host computer in the novel architecture performs a set of processing steps before delivering the data message to the destination DCN. Figure 13Conceptually, a process 1300 for processing a data message addressed to a service or DCN executing on a host computer is illustrated. While the process is described as being performed by the host computer, one of ordinary skill in the art will recognize that, in some embodiments, different operations are performed by different components of the host computer. The process 1300 begins with receiving (at 1310) an incoming data message from a gateway device addressed to a set of DCNs in a logical network. In some embodiments, the host computer receives the data message at a managed forwarding element (MFE) (e.g., a software switch executing on the host computer). In some embodiments, the MFE provides the received data message to a routing instance to determine a next hop. In some embodiments, the data message is received at a routing instance that advertises availability of a destination network (e.g., IP) address.

[0102] In some embodiments, the set of DCNs is a single DCN specifically addressed by the incoming data message. In some embodiments, the set of DCNs is a set of DCNs for which a load balancing service instance executing on the host computer provides load balancing. In some embodiments, the set of DCNs includes a set of one or more DCNs executing on the host computer, while in other embodiments, the set of DCNs includes only DCNs executing on other host computers. In some embodiments in which a load balancing service instance performs load balancing for a set of DCNs including DCNs executing on the host computer, the DCNs executing on the host computer are preferentially selected to receive the data message. In some embodiments, the preference is expressed as a higher weight for a weighted load balancing algorithm, or an additional cost for forwarding the data message to a DCN executing on another host (e.g., adding a constant value to a measure of load of the DCN on another host computer).

[0103] After receiving (at 1310) an incoming data message, a first routing operation is performed (at 1320) at a routing instance that advertises the availability of a destination network (e.g., IP) address to other routers in the AZ. In some embodiments, the routing instance is a multi-tenant routing instance that uses information in the incoming data message's header (e.g., a routing distinguisher in an MPLS header, a VNI in a VXLAN header, etc.) to identify a VRF context for performing routing operations on the incoming data message. The routing operation identifies the next hop for the data message and, in some embodiments, includes any or all of a decapsulation operation, an encapsulation operation, or other operations for modifying the packet as needed to reach the incoming data message's destination. In some embodiments, the next hop for the data message entering the VPC is a tunnel endpoint (e.g., a Virtual Extensible Local Area Network (VXLAN) Tunnel Endpoint (VTEP)). In other embodiments where the routing instance has already advertised the availability of the incoming data message's destination IP address at the VTEP, operations 1310 and 1320 are not performed because the data message is sent directly to the VTEP. In embodiments where both VTEPs are executed on the host computer, the destination address is associated with a specific VTEP executing on the host computer.

[0104] In embodiments where the VTEP receives an incoming data message, the VTEP identifies (at 1330) a logical network associated with the set of DCNs addressed by the incoming data message. In some embodiments, the identification includes identifying a local VNI of a tenant logical network corresponding to a VXLAN VNI included in the incoming data message, removing a VXLAN header of the data message and replacing it with a logical network header (e.g., a VLAN header), the local VNI providing the data message to a distributed logical router associated with the logical router identifier. In other embodiments, the identification includes identifying an interface associated with the logical network based on information included in the incoming data message. In some embodiments, the VTEP acts as a tunnel endpoint for multiple logical networks implemented on a host computer. In some embodiments, the multiple logical networks are for different tenants.

[0105] After identifying (at 1330) the logical network associated with the set of DCNs addressed by the incoming data message, the incoming data message is provided (at 1340) to a distributed logical router instance of the identified logical network executing on the host computer. In some embodiments, the distributed logical router is defined at a logical edge of the VPC, such that all traffic entering and leaving the VPC is processed through the distributed logical router. In some embodiments, the distributed logical router instance is a managed physical routing element (MPRE) that performs logical routing (e.g., implements the logical router) for the distributed logical router of multiple tenants. In other embodiments, the MPRE executes on the host computer for each tenant to perform logical routing for the distributed logical router of the tenant.

[0106] After providing (at 1340) the incoming data message to the distributed logical router instance, a logical routing operation is performed (at 1350) to identify a next hop for the incoming data message toward the set of DCNs. In some embodiments, instead of a set of physical gateway devices of the AZ being configured to perform edge services, the distributed logical router instance implements a distributed logical edge router with distributed logical router instances in other host computers to perform edge services for the logical network. In some embodiments, performing logical routing at the distributed logical route instance includes determining (at 1355) whether a service is logically configured to be performed before reaching the next hop. In other embodiments, this determination is made (at 1355) at a logical interface (e.g., port) of the distributed logical router (or next hop logical router or logical switch) performing the logical routing operation as part of a logical forwarding operation. If the process 1300 determines (at 1355) that no service is logically configured between the logical router and the next hop, then the process proceeds to operation 1370 and forwards the data message to the next hop as discussed below, and the process ends.

[0107] In some embodiments, the distributed logical router processing operation identifies a set of services (e.g., stateful or stateless edge services) required to process data messages entering the VPC. In some embodiments, the set of services includes any or all of a firewall service, a network address translation (NAT) service, a load balancing service, a distributed intrusion detection system (IDS) service, and a distributed intrusion protection system (IPS) service. Different services use different information contained in data messages and, in some embodiments, are implemented in different ways based on the type of information used. For example, services that use information contained in headers at Layer 4 of the Open Systems Interconnection (OSI) model (sometimes referred to as "L4 services") execute in kernel space because they do not consume a lot of resources and moving them to user space would make them unacceptably slow. However, in some embodiments, services that use data at Layer 7 of the OSI model (sometimes referred to as "L7 services") execute in user space because their resource-intensive processing would occupy threads in kernel space in order to quickly process data messages and because the cost of moving to user space is not significant compared to the cost of performing the operation in the first place.

[0108] In some embodiments, the collection of services is implemented for multiple tenants in separate VPCs on the same host computer, as described above with respect to Figure 3 In some embodiments, different services are provided by different modules (containers, applications, etc.) in a multi-tenant service machine (or pod). In some embodiments using a multi-tenant service machine, the service machine implements separate VTEPs to differentiate traffic for each tenant. In other embodiments, a separate service machine (e.g., virtual machine, container, pod, etc.) is provided for each tenant to implement a set of L7 services for the tenant.

[0109] If process 1300 determines (at 1355) that the service is logically configured to be executed before reaching the logical switch, then the service is invoked (at 1360). In some embodiments, invoking the service includes providing an incoming data message to a service instance (e.g., a program, module, application, etc.) executing on a host computer. In some embodiments, the service instance returns the service's data message to the logical interface from which it received the distributed logical routing instance (e.g., back to the logical interface). In other embodiments, the service is part of a processing pipeline (e.g., a distributed logical router processing pipeline) and the service provides the service's data message to the next stage or operation in the processing pipeline. In some embodiments, the service is a load balancing service (e.g., a distributed load balancing service) that executes in the kernel of the host computer and performs a load balancing service (i.e., a Layer 4 (L4) service) based on parameters in header values ​​associated with layers 1-4 of the OSI model. In other embodiments, the service is a firewall service or any other middlebox service that can be executed between a router and a switch.

[0110] After providing the service, the process 1300 determines (at 1365) whether the additional service is logically configured before reaching the next hop. In some embodiments, this determination is made implicitly by the processing pipeline including or not including the additional service before forwarding the data message to the next hop. In other embodiments, the distributed logical router instance performs subsequent routing operations on the service-incoming data message to determine whether the additional service is logically configured before reaching the next hop toward the destination DCN. In some embodiments, the determination made by the distributed logical router instance is made implicitly by determining, based on routing rules (e.g., policy-based routing rules), that the data message should be provided to the service instance.

[0111] If process 1300 determines (at 1365) that the additional service is logically configured before reaching the next hop, the process returns to operation 1360 to invoke the additional service as described above. However, if process 1300 determines (at 1365) that the additional service is not logically configured before reaching the next hop, the data message is forwarded to the next hop and the process ends. In some embodiments, the next hop is a destination DCN (e.g., a container, pod, VM, etc.) associated with the destination IP address of the incoming data message. The destination DCN for some incoming data messages is a specific DCN addressed by the incoming data message. For other incoming data messages addressed to a VIP, the destination DCN identified by the routing operation is a middlebox service DCN that provides load balancing or other services for the set of DCNs associated with the destination address of the ingress data message. In some embodiments, the middlebox service DCN performs a service and identifies a DCN in the set of DCNs associated with the destination address (e.g., performs a load balancing service to identify a DCN to receive the incoming data message) and forwards the incoming data message to the identified DCN. Those skilled in the art will appreciate that additional logical forwarding elements and service instances may exist along the path to the destination DCN and, in some embodiments, will be processed similarly to operations 1350 - 1370 .

[0112] In some embodiments, a controller computer cluster (ie, a collection of one or more controller computers) of a VPC (eg, a logical network) in an AZ provides configuration information to network elements to implement the VPC. Figure 14 A process 1400 is conceptually illustrated for generating configuration data for different network elements that provide a novel network architecture for advertising routes in an availability zone (e.g., a data center that provides a collection of hardware resources) and for providing a collection of distributed services at the edge of a VPC. In some embodiments, process 1400 is performed by a controller computer or a cluster of controller computers. In some embodiments, the cluster of controller computers includes a management plane controller computer. In some embodiments, process 1400 is performed whenever a new distributed edge service is added to a logical network or a distributed edge service instance (e.g., a middlebox service instance) is launched or added to a host computer in an AZ for an existing distributed edge service.

[0113] Process 1400 begins by receiving (at 1410) an instruction to modify a distributed edge service in a logical network (e.g., a VPC). In some embodiments, the modification instruction is an instruction to add a distributed edge service that was not previously provided in the logical network. In other embodiments, the modification instruction is an instruction to add a distributed edge service instance to the set of distributed edge service instances currently providing the distributed edge service. In some embodiments, the received instruction is generated by a controller computer or cluster of controller computers executing process 1400 based on a policy that specifies when a particular distributed edge service or distributed edge service instance should be added to the logical network.

[0114] After receiving (at 1410) an instruction to modify a distributed edge service, process 1400 determines (at 1420) a number of distributed service instances to execute to implement the distributed edge service for the logical network. In some embodiments, determining the number of distributed service instances includes identifying (1) active distributed service instances and (2) distributed service instances that have been requested to be activated (e.g., started). In some embodiments, identifying the distributed service instances includes identifying the number of end machines (e.g., DCNs, workload VMs, containers, etc.) that each distributed service instance supports (i.e., provides a distributed service for). In some embodiments, in addition to identifying the number of end machines, identifying the distributed service instances also includes identifying one or both of (1) the total number of connections being handled by the distributed service (i.e., the sum of all distributed instances) and (2) the number of connections handled by each distributed service instance.

[0115] After determining (at 1420) the number of distributed service instances, process 1400 determines (at 1430) the number of port ranges or the size of the port ranges that will be available for assignment to the distributed service instances for a particular distributed service (such as a distributed network address translation service). In some embodiments, the number of port ranges or the size of the port ranges are determined based on input from a user (e.g., an administrator) of the VPC or a logical subnet within the VPC. The input from the user can be based on a maximum amount of resources that the user expects the distributed service instance to consume when providing the distributed service. In some embodiments, the user input specifies any one or all of (1) the maximum number of distributed service instances that can be instantiated, (2) the maximum number of ports that can be assigned to a single distributed service instance, or (3) a policy for determining the number of ports assigned to a particular distributed service instance. In some embodiments, the policy is based on any one or all of: (1) the number of active distributed service instances, (2) the number of compute nodes for which each active distributed service instance provides a distributed service, (3) the number of connections handled by the distributed service, and (4) the number of connections handled by each distributed service instance.

[0116] For example, a policy may specify that the entire range of possible port numbers be divided into powers of two that are at least twice the number of distributed service instances (or as large), and that the port number ranges are adjusted based on the policy as the number of distributed service instances increases or decreases (e.g., going from 4 to 5 distributed service instances results in each of the 8 port ranges being divided into two smaller port number ranges, or going from 17 to 16 distributed instances results in the 64 port number ranges being merged into a 32 port number range). In some embodiments, the policy specifies that non-adjacent port number ranges are assigned to each distributed service instance (e.g., 0-8191 assigned to the first distributed service instance, 16384-2475 assigned to the second distributed service instance, etc.). This policy allows the number of hosts to be increased and decreased without having to constantly reassign port number ranges.

[0117] In some embodiments, the policy may specify that when a particular distributed service instance uses a portion of the port numbers assigned to the distributed service instance above a threshold fraction (e.g., 0.8 or 0.9), (a) an adjacent available range will be assigned to the distributed service instance, (b) workload compute nodes will be migrated from the host computer on which the distributed service instance is executing, or (c) a new distributed service instance will be added to another host computer. Alternatively or additionally, in some embodiments, the policy may specify that when a particular distributed service instance uses a portion of the port numbers assigned to the distributed service instance below a threshold fraction (e.g., 0.3 or 0.2), the range of assigned port numbers will be reduced, or additional terminal machines will be migrated to the host computer on which the distributed service instance is executing (e.g., from the host computer on which the distributed service instance is executing using a majority of its assigned port numbers). Other possible policies for handling excess capacity of distributed services include a policy that specifies that when the total number of connections handled by a distributed service instance falls below a certain threshold fraction based on the number of distributed service instances and the capacity of the assigned port range that the distributed service instance will be deactivated, or a smaller port range will be assigned to each active distributed service instance. Other strategies may specify a port range to be assigned based on the number of workload computing nodes providing distributed services by the distributed service instance (e.g., for 0-10 workload computing nodes, 256 port numbers are assigned; for 11-20 workload computing nodes, 512 port numbers are assigned; etc.). Those skilled in the art will appreciate that these strategies are merely examples of possible strategies, and that different strategies may be used in different embodiments based on user requirements.

[0118] After determining (at 1430) the port number ranges, process 1400 selects (at 1440) at least one port range to assign to each distributed service instance. As discussed above, in some embodiments, the initial port range assignment assigns non-adjacent, non-overlapping port ranges to each distributed service instance. In some embodiments, subsequent assignments assign at least one additional port number range to specific distributed service instances that use a number of port numbers greater than a threshold fraction of the assigned port numbers. In some embodiments, other subsequent assignments remove a portion of a range of port numbers from the initial assignment to specific distributed service instances that use fewer than a threshold number of port numbers within the initially assigned port number range.

[0119] In some embodiments, the size of the port number range assigned to the distributed service instance is determined by an administrator based on the expected maximum number of distributed service instances (e.g., for an expected maximum of 64 distributed service instances, 64 different port number ranges are created, each range including 1024 ports, each port being assigned to a distributed service instance at startup). In other embodiments, the size of the port number range is dynamic and can change based on the number of active distributed service instances, active connections, or workload compute nodes using the distributed service. The size of the port number range can also vary between distributed service instances. For example, a larger port number range is assigned to a first distributed service instance executing on a host that executes a greater number of workload compute nodes using the distributed service than to a second distributed service instance executing on a host that executes a smaller number of workloads, and can change as the number of workload compute nodes changes.

[0120] After selecting (at 1440) a range of port numbers to use for each distributed service instance, process 1400 identifies (at 1450) a set of host computers to host the distributed edge service instances. In some embodiments, this determination includes identifying host computers currently executing elements of the logical network to which the distributed edge service instances are to be added. After the host computers are identified, the resources of each host computer available for executing additional network elements of the logical network are determined to identify host computers with sufficient resources to execute the distributed edge service instances. In some embodiments, additional considerations include the number of workloads DCN executing on the host requiring the distributed edge service, the number of other tenants executing network elements on the host computer, the number of connections currently being handled by the host computer, and the like.

[0121] After identifying (at 1450) a set of host computers to execute a distributed edge service instance, process 1400 generates (at 1460) configuration data for implementing the desired distributed service instance. In some embodiments, the generated configuration data for executing the distributed edge service instance includes a number of cores (e.g., the cores of the service GM on which the service instance executes) or the amount of other resources assigned to the service instance. In some embodiments, the resources assigned to the service instance are specified using Kubernetes (k8s) annotations and converted into configuration data to be provided to host computer 1520 (e.g., to configuration data storage 1528). In some embodiments, the configuration data generated for a first L7 service to be executed in a service DCN (e.g., L7 service GM 1522a) executed in host computer 1520 includes configuration data for adding the service DCN to an IP address in a service fragment subnet for a service interface of the DCN in which the L7 service executes.

[0122] In some embodiments, the generated configuration data includes multiple sets of configuration data for different network elements (eg, host computers, gateway devices) and for different purposes. Figure 15 The diagram illustrates a computer controller cluster 1540 in a data center 1505 sending different types of configuration data to different network elements. Figure 15 The elements are generally related to Figure 1 Same elements as discussed in . Figure 15 A collection of configuration data 1542 (received at configuration data storage 1528) is illustrated for each host computer 1520. In some embodiments, configuration data 1542 includes configuration information for: (1) configuring a distributed service instance to provide a distributed service, (2) configuring other network elements (e.g., GM 1525 and MFE 1560) executing on the host computer to communicate with the distributed service instance (e.g., 1524), and (3) configuring a routing instance (e.g., routing machine 1523) executing on the host computer to advertise an IPv6 address associated with the distributed service instance executing on the host computer.

[0123] In some embodiments, configuration data storage 1528 receives configuration data and identifies configuration data for each module executing on host computer 1520, such as the configuration data for each module executing on host computer 1520. Figure 4In some embodiments, configuration data is pushed from the configuration data store 1528 to the local controller 1526 and the communication agent 1527 to be propagated to the different elements configured on the host computer 1520. In some embodiments, the local controller 1526 is responsible for pushing the L4 service configuration data to the data plane and configuring the host computer to execute the DCN (GM 1525 and L7 service GM 1522a), while the communication agent 1527 (e.g., VMCI agent) is responsible for pushing data from the configuration data store 1528 to the data path (e.g., nginx) of the DCN (e.g., L7 service GM 1522a) where the L7 service executes.

[0124] Configuration data (e.g., configuration data 1542) includes configuration data for configuring at least one distributed service instance executing on at least one host computer to provide the distributed service using a range of assigned port numbers. In some embodiments, the configuration data for initializing a new distributed service instance on the host computer includes an IPv4 address associated with a distributed NAT service used in performing distributed service operations (e.g., replacing the source IP address of data messages from a first network to an external network) and the assigned port number range. In some embodiments, the configuration data includes a set of policies for providing the distributed service (e.g., firewall rules, load balancing criteria, or a policy for selecting a DCN to receive data messages, etc.). In some embodiments, additional configuration information (e.g., the logical overlay network element to which the distributed instance is connected) is sent to the host computer to configure other elements of the host to communicate with the new distributed service instance, as will be appreciated by one of ordinary skill in the art. For L7 services executing in a user-space DCN (e.g., a VM, container, or pod), in some embodiments, the configuration data includes a VIP associated with the service and a service IP address for the user-space DCN in which the L7 service executes.

[0125] In some embodiments, the additional configuration data sent to the host computer includes configuration data sent to the host computer to configure the host computer (or an MFE or BGP instance executing on the host computer) to recognize and advertise IPv6 address prefixes associated with the distributed service instance or the added DCN, such as with respect to Figures 4-8 As mentioned above. Figure 4 As discussed, in some embodiments, the configuration data also includes information used within the host computer to address the distributed service instance and configure a machine executing on the host computer to use the distributed service instance for specific packets (e.g., packets sent to an external network).

[0126] In some embodiments, the generated configuration data includes configuration data (e.g., configuration data 1541) generated for providing to the gateway device. In some embodiments, the controller computer cluster 1540 sends configuration data 1541 to a collection of gateway devices for configuring the gateway devices to perform IPv4 to IPv6 encapsulation, and in some embodiments, for configuring the gateway devices with IPv6 routing table entries. In some embodiments, the gateway devices are partially or fully programmable gateway devices that can be programmed by the controller computer cluster to implement IPv4 to IPv6 translation and encapsulation based on PBR rules specified based on the IPv4 address and the destination port in the IPv4 header. In other embodiments, the gateway devices are off-the-shelf gateway devices (e.g., dual-stack routers) that can be simply programmed to be sufficient to configure the gateway devices to implement IPv4 to IPv6 encapsulation.

[0127] For both programmable and off-the-shelf gateway devices, the configuration data includes a collection of what will be called distributed service records and IPv6 routing table entries. In some embodiments, the distributed service record maps the combination of the IPv4 address and destination port number used by a specific distributed service operation to an IPv6 destination address. In some embodiments, the distributed service record is provided as a lookup table and instructions for using the lookup table to route data messages using the IPv4 address associated with the distributed service. In some embodiments, the distributed service record is a PBR rule (or similar rule or policy) that defines an algorithm for generating an IPv6 address from an IPv4 destination address and a port number. In some embodiments, the PBR rule specifies the IPv4 destination address to which the algorithm should be applied, while in other embodiments, the IPv4 address and port number are specified. In some embodiments, the distributed service record is an instruction for configuring an off-the-shelf gateway device to perform IPv6 encapsulation of an IPv4 packet destined for the IPv4 used by a specific distributed service operation according to a specified algorithm. In some embodiments, the instruction is based on functionality (e.g., an exposed API) provided by the off-the-shelf gateway device.

[0128] In some embodiments, each of the IPv6 routing table entries identifies an IPv6 address prefix associated with a particular host computer from a set of host computers executing a distributed service instance and a next-hop interface for reaching the particular host computer. The IPv6 address prefix specified for the particular host in the IPv6 routing entry is based on an IPv4 address associated with the distributed service and a port number range assigned to the distributed service instance executing on the host. If multiple non-contiguous port ranges are assigned to a particular host computer, then the set of IPv6 routing table entries includes multiple entries for the particular host computer.

[0129] Once the configuration data is generated (at 1460), the configuration data generated for each network element is forwarded (at 1470) to the appropriate network element for use in configuring the network elements. Figures 4-8 The network elements described. In some embodiments, the configuration data, after being received by the local agent for communication with the controller computer cluster (e.g., configuration data storage device 1528), is used by the local controller (e.g., local controller 1526) and VMCI agent (e.g., communication agent 1527) at the host computer, and the local agent communicates with the controller computer cluster using control plane messages. The local controller and VMCI agent then provide configuration data or configure elements on the host computer to implement the distributed service (e.g., instantiate a distributed service instance, configure the GM to use the distributed service instance, configure the DCN with a service IP and a set of L7 service instances, and configure the MFE or routing instance to advertise the IPv6 address prefix associated with the distributed service instance, etc.). The configuration data generated for the gateway device is forwarded to the gateway device to configure the gateway device to identify a specific host associated with a specific received packet (e.g., by using the provided IPv6 routing table entry). After forwarding (at 1470) the configuration data, the process ends. One of ordinary skill in the art will appreciate that in some embodiments, process 1400 is performed for each distributed service using the same IPv4 address as the source address of outgoing packets at each of a plurality of distributed service instances.

[0130] In some embodiments, the controller computer cluster periodically or based on a schedule monitors the load on the distributed service instance and the distributed service in the aggregation. In some embodiments, monitoring is based on a program executed on the host computer identical to the distributed service instance. In some embodiments, the program monitors a set of metrics associated with the distributed service instance (e.g., time delay, the number of connections handled, the number of packets per second, the number of terminal machines using the distributed service instance, etc.). In some embodiments, as long as a new distributed service instance or workload machine is requested to be initialized, operations 1420 and 1430 are performed. In some embodiments, operations 1420 and 1430 are also performed periodically or based on a schedule set by an administrator to determine whether monitoring information indicates that there has been a change in the size of a required reallocation port number range or any port number range. If such a change occurs, operations 1440-1470 are performed to update the allocation of the port range and provide updated configuration data to the network element.

[0131] Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer readable media do not include carrier waves and electronic signals over wire, fiber optic, or other communication media.

[0132] In this specification, the term "software" is meant to include firmware residing in read-only memory or applications stored in magnetic storage which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the application. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

[0133] Figure 16 A computer system 1600 is conceptually illustrated, with which some embodiments of the application are implemented. The computer system 1600 can be used to implement any of the above-described hosts, controllers, and managers. As such, it can be used to perform any of the above-described processes. This computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 1600 includes a bus 1605, processing unit(s) 1610, a system memory 1625, a read-only memory 1630, a permanent storage device 1635, an input device 1640, and an output device 1645.

[0134] The bus 1605 serves as the main information highway connecting the various islands of circuitry together and facilitating communication therebetween. For example, the bus 1605 couples the processing unit(s) 1610 to the system memory 1625, the read-only memory 1630, and the permanent storage device 1635.

[0135] The processing unit(s) 1610 retrieve instructions to execute and data to process from these various memory units, in order to execute the processes of the present application. In different embodiments, the processing unit(s) can be single or multi-core processors. The read only memory (ROM) 1630 stores static data and instructions that are needed by the processing unit(s) 1610 and other modules of the computer system. The permanent storage device 1635, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the computer system 1600 is off. Some embodiments of the present application use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device 1635.

[0136] Other embodiments use a removable storage device (such as a floppy disk, flash drive, etc.) as the permanent storage device. Like the permanent storage device 1635, the system memory 1625 is a read-and-write memory device. However, unlike the permanent storage device 1635, the system memory is a volatile read-and-write memory, such as a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the processes of the present application are stored in the system memory 1625, the permanent storage device 1635, and / or the read only memory 1630. The processing unit(s) 1610 retrieve instructions to execute and data to process from these various memory units, in order to execute the processes of some embodiments.

[0137] The bus 1605 also connects to the input and output devices 1640 and 1645. The input devices enable the user to communicate information and select commands to the computer system. The input devices 1640 include alphanumeric keyboards and pointing devices (also called “cursor control devices”). The output devices 1645 show information generated by the computer system. The output devices include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as a touchscreen that functions as both input and output devices.

[0138] Finally, as shown in Figure 16 The bus 1605 also couples the computer system 1600 to a network 1665 through a network adapter (not shown). In this manner, the computer can operate in a networked environment using the network 1665. The computer system 1600 is in some embodiments a part of the network 1665. The network 1665 is home to a plurality of computer systems. These computer systems include, for example, a server computer, a client computer, a peer computer, and / or other computer systems that are communicatively coupled to the network 1665. The network 1665 and its components can be used to implement the present application.

[0139] Some embodiments include electronic components, such as a microprocessor, that store computer program instructions on a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disc-read only (CD-ROM), compact disc-recordable (CD-R), compact disc-rewritable (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini-SD card, micro-SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code (e.g., generated by a compiler) and files including high-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0140] While the above discussion primarily refers to microprocessors or multi-core processors executing software, some embodiments are performed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves.

[0141] As used in this specification, the terms "computer," "server," "processor," and "memory" refer to electronic or other technical devices. These terms do not include a person or group of people. For the purposes of this specification, the term "display" refers to displaying on an electronic device. As used in this specification, the terms "computer-readable medium," "computer-readable medium," and "machine-readable medium" are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other short-lived or transient signals.

[0142] Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. For example, several figures illustrate the process conceptually. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a continuous series of operations, and different specific operations may be performed in different embodiments. In addition, the process may be implemented using several sub-processes or as part of a larger macro-process. Therefore, it will be appreciated by those skilled in the art that the present invention is not limited to the foregoing illustrative details, but is defined by the appended claims.

Claims

1. A method for routing instances executed on a host computer among a plurality of host computers in a data center, the plurality of host computers executing a plurality of data compute nodes (DCNs) and collectively implementing a first virtual private cloud (VPC) and a second virtual private cloud (VPC) for a first tenant and a second tenant in the data center, the method comprising: At the routing instance executing on the host computer, detecting that a first DCN and a second DCN for the first VPC and the second VPC, respectively, have been added to the host computer; identifying a first network address and a second network address associated with the added first DCN and second DCN, respectively; as well as announcing the identified first network address and second network address to a routing server, so that the routing server announces them to a first set of host computers and a second set of host computers, respectively implementing the first VPC and the second VPC, among the plurality of host computers. wherein the routing instance is a multi-tenant routing instance that maintains a first virtual routing and forwarding (VRF) context and a second VRF context for the first tenant and the second tenant and uses Border Gateway Protocol (BGP) to advertise available addresses, and The routing instance is a multi-protocol BGP (MP-BGP) instance that associates each VRF context with a set of different routing discriminators and routing target values ​​to identify data messages associated with each VRF context. 2 . The method of claim 1 , wherein detecting that the first DCN has been added comprises receiving a notification from a controller computer.

3. The method of claim 2, wherein the notification is received via a communication agent executing on the host computer to communicate with the controller computer via a management network. 4 . The method of claim 1 , wherein detecting that the first DCN has been added comprises receiving a notification from the instantiated first DCN.

5. The method of claim 1, wherein the identified first network address is an internal network address used by other DCNs in the logical network.

6. The method of claim 5, wherein the data center comprises a gateway device located between an external network and the logical network, the method further comprising: identifying an external network address associated with the added first DCN used by a machine in the external network; as well as The external network address is announced to a routing server, so that the routing server announces it to the gateway device of the data center.

7. The method of claim 1, wherein the routing instance registers for notifications related to adding a service instance to a host computer.

8. A method for routing instances executed on a host computer among a plurality of host computers, the plurality of host computers collectively implementing a distributed edge service of a virtual private cloud (VPC) implemented by the plurality of host computers, the method comprising: At the routing instance executing on the host computer, detecting that a service instance is available on the host computer to provide a distributed edge service for incoming data messages that enter the VPC from a machine outside the VPC and are destined for a machine inside the VPC; identifying an external network address associated with the distributed edge service used by a machine external to the logical network; as well as announcing the identified external network address to a routing server operating external to the host computer, for the routing server to announce to a gateway device in a data center, for the gateway device to use in directing at least a subset of the incoming data messages to the host computer for the distributed edge service, The external network address is a first IP version 6 (IPv6) network address based on an Internet Protocol version 4 (IPv4) address associated with the distributed edge service, the IPv6 network address being used to address the service instance executing on the host computer, and other IPv6 network addresses based on the IPv4 address associated with the distributed edge service being used to address other service instances executing on other host computers among the plurality of host computers.

9. The method of claim 8, wherein detecting that the service instance is available comprises receiving a notification from a controller computer.

10. The method of claim 9, wherein the notification is received via a communication agent executing on the host computer to communicate with the controller computer via a management network. The method of claim 8 , wherein detecting that the service instance is available comprises receiving a notification from the service instance.

12. The method of claim 8, wherein the logical network is a first logical network of a first tenant, the service instance is a first service instance of the first tenant, the distributed edge service is a first distributed edge service, and a set of host computers in the plurality of host computers implement a second distributed edge service for a second logical network, the second logical network being implemented by the set of host computers in the plurality of host computers, the second distributed edge service being provided for data messages associated with a plurality of DCNs that enter and leave the second logical network through the gateway device of the data center, the method further comprising: detecting that a second service instance for providing a second distributed edge service for a second logical network is available on the host computer; identifying a network address associated with the instantiated second service instance; as well as The identified network address associated with the second service instance is announced to the routing server, so that the routing server announces it to the gateway device of the data center.

13. The method of claim 12, wherein the routing instance is a multi-tenant routing instance that maintains a first virtual routing and forwarding (VRF) context and a second VRF context for a first tenant and a second tenant and uses a Border Gateway Protocol (BGP) to announce available addresses.

14. The method of claim 13, wherein the BGP instance is a multi-protocol BGP (MP-BGP) instance that associates each VRF context with a different set of route discriminators and route target values ​​to identify data messages associated with each VRF context.

15. The method of claim 8, wherein the routing instance registers for notifications related to adding a service instance to a host computer.

16. A machine-readable medium storing a program, which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 15.

17. An electronic device comprising: A collection of processing units; as well as A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1 to 15.

18. A system comprising means for implementing the method according to any one of claims 1-15.

19. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 15.

Images