parsing logical network definitions for different sites

Abstract

As more and more networks move to the cloud, it is more common for an enterprise or other entity to have a network that spans multiple sites. While logical networks operating within a single site are well established, there are various challenges in making logical networks span multiple physical sites (e.g., data centers). Sites should be self-contained while also allowing data to be easily sent from one site to another. Various solutions are needed to address these issues.

Description

Background Technology

[0001] As more and more networks move to the cloud, it's becoming increasingly common for companies or other entities to have networks spanning multiple sites. While logical networks operating within a single site have been well-established, there are various challenges in enabling logical networks to span multiple physical sites (e.g., data centers). Sites should be independent, yet also allow for easy data transfer from one site to another. Various solutions are needed to address these challenges. Summary of the Invention

[0002] Some embodiments of the present invention provide a network management system for managing a logical network spanning multiple federated sites (e.g., multiple data centers). The network management system in some embodiments includes a global manager that manages the entire logical network spanning all sites, and a local manager that directly manages the logical network at each site. The logical network includes logical network elements spanning one or more sites and logical network policies applied to the elements at those sites. In some embodiments, the global manager receives a global desired configuration for the logical network (e.g., from a network administrator), identifies relevant portions of the global desired configuration for each site in the federation, and provides the identified portions to the corresponding local manager for that site. Furthermore, the global manager collects information from the local managers regarding the implemented state of the logical network across sites and provides this information (e.g., to the logical network administrator) for troubleshooting and management purposes.

[0003] In some embodiments, the global manager executes on a computing device at one of the sites spanned by a logical network, and each local manager also executes on a computing device at its respective site. In some embodiments, the global manager executes on the same computing device at one of the physical sites as the local manager that manages that site. In some embodiments, the global manager and the local managers are separate modules of a single application. Some embodiments deploy each manager at a physical site as a cluster of machines, each machine executing on a different computing device at the same site.

[0004] Some embodiments employ a primary global manager and a secondary global manager in an active-standby deployment. The primary global manager is asynchronously synchronized with the secondary global manager as a backup for failover scenarios. The secondary global manager executes on different computing devices, and in some embodiments is located at a different site from the primary global manager for resilience, and maintains an independent database. In some embodiments, the secondary global manager is also deployed as a collection of machines, and may also execute on the same computing device as the local manager.

[0005] In some embodiments, the primary global manager's database is a distributed shared log implemented across a collection of storage devices at the physical site where the primary global manager resides. A series of database transactions are used to receive and store data related to the desired global configuration in the database; these transactions are initiated via a series of application programming interface (API) calls to the global manager. In some embodiments, the database generates update streams from the database transactions, which are used to provide secondary global managers with data about the desired configuration for database replication. The update streams also include metadata associated with each transaction, such as timestamp information that can be used for data sorting, and database state to prevent access race conditions. In some embodiments, the database is shared by the primary global manager and other applications (e.g., local managers) on the same computing device. In some such embodiments, data to be replicated to secondary global managers is tagged so that only data associated with the primary global manager is replicated, and other data associated with other applications on the computing device is not replicated.

[0006] In some embodiments, the database of each global manager also includes a collection of queues, each queue being reserved for the local manager at each physical site, and a queue being reserved for auxiliary global managers. The database at the auxiliary global manager also includes these queues. When the primary global manager receives a global desired configuration for a logical network, the global manager stores the portion of the global configuration in each queue based on the relevance of that portion to the configuration of the logical network at the corresponding physical site in the queue. In some embodiments, a broker service of the global manager identifies the relevant portion for each physical site (e.g., based on the span of logical network elements), as described in further detail below.

[0007] In some embodiments, a collection of asynchronous channels connects a primary global manager to local managers and secondary global managers at different sites. Some embodiments execute a collection of site managers at each global manager to manage the channels, with each site manager corresponding to one of the physical sites. Channels retrieve data from queues in a database and provide the retrieved data to a destination database at the corresponding physical site. These channels maintain connectivity between physical sites and, in some embodiments, guarantee various connectivity parameters (e.g., minimum bandwidth, maximum round-trip time, etc.) required to replicate data to secondary global managers and propagate data to secondary local managers. When the managers are implemented as a cluster of machines, the channels also identify the active machine for each manager.

[0008] In some embodiments, the globally desired configuration of a logical network is expressed as a hierarchical tree (also known as a global policy tree) with nodes and connections between nodes. In some embodiments, a node represents a logical network element spanning one or more sites and a logical network policy applied to these elements, and a connection represents a relationship between nodes (e.g., parent-child relationship, logical network connection, etc.). Logical network elements include logical forwarding elements that forward data in the logical network, such as logical routers, logical switches, etc. These logical network elements also include, in some embodiments, logical constructs such as logical ports associated with logical forwarding elements, domains that are logical groups of one or more sites (e.g., geographic regions), and groups of logical network endpoints that share one or more attributes (e.g., operating system, region, etc.). In some embodiments, each logical network element is implemented by a physical forwarding element (PFE) executed on a computing device at the site spanned by that logical network element. Logical network policies include forwarding policies, service policies, and security policies, and are applied in some embodiments to govern the behavior of logical forwarding elements.

[0009] In some embodiments, the global manager performs span calculations for each logical network element to determine which logical network elements are implemented at each site. In some embodiments, the span calculation is based on relationships between nodes in the global policy tree. For example, when the relationship between two nodes is a parent-child relationship, the child node inherits its span from the parent node. However, in other cases, the child node does not inherit the entire span of its parent node. As another example, when the relationship between two nodes is a dependency of one node on another, expressed as a reference of one node to another, the span of the dependent node will depend on the referenced node. In some embodiments, some logical network elements also have predefined spans, for example, defined by the network administrator.

[0010] In some embodiments, the relevant portion of the globally desired configuration provided to each local manager is a site-specific policy subtree of the global policy tree, corresponding to the logical network element implemented at that site. The subtree for each site includes only the nodes that span the logical network element across that site. In some embodiments, the subtree is generated by first identifying the span of each node in the global policy tree. The global policy tree is then resolved using the span identified for each node to generate the policy subtree for each site.

[0011] At each site, a local manager uses a relevant portion of the globally desired configuration received from a global manager to manage the logical network at that site. For example, in some embodiments, the local manager uses the relevant portion to generate configuration data and provide it to the control plane of the logical network (e.g., a cluster of controllers at each site). In some embodiments, these controllers identify the computing devices that perform physical forwarding elements at the site and distribute the configuration data to the identified computing devices. Some embodiments have a local controller (also called a chassis controller) that performs on one or more computing devices and receives configuration data from a cluster of controllers. The local controller uses the configuration data to configure the physical forwarding elements to implement logical network elements. The controller cluster at each site also creates a mapping between logical addresses (e.g., the MAC address of a logical network endpoint performing on a computing device) and physical addresses (e.g., the IP address of a tunnel endpoint on a computing device) and distributes these mappings to each of their associated computing devices, as well as to other controller clusters at other sites that require data.

[0012] In some embodiments, the computing device at each site also acts as a machine, along with a physical forwarding element and a local controller. These machines include logical network endpoints, which are the source and destination of data message traffic (e.g., computers, virtual machines, containers, etc.), and service machines (e.g., firewalls, load balancers, etc.) that perform services on the data traffic. Machines reside at a single site at a given time, but in some embodiments may migrate between sites. In some embodiments, these machines are connected to physical forwarding elements to exchange data messages within the network. In some embodiments, the logical network endpoints at each site are logically organized into groups that can span multiple sites. In some embodiments, the service machine applies some of the logical network policies to the data messages exchanged between groups of endpoints based on policies defined according to these groups.

[0013] In some embodiments, policies are defined at a global manager, for example, by an administrator of the logical network. In some embodiments, a policy is one or more service rules enforced at a site on a data message flow based on a set of flow attributes. In some embodiments, the global manager distributes service rules to local managers at the sites where policies are to be enforced (i.e., the policy span). The local managers use the service rules to generate configuration data for controller distribution, configuring the data plane (i.e., forwarding elements and service machines) to enforce the received service rules on data message flows associated with groups of logical network endpoints.

[0014] In some embodiments, policies are defined by referencing groups of logical network endpoints spanning one or more sites. In some embodiments, these groups are defined at a global manager, such as by an administrator of the logical network. In some embodiments, service rules reference these groups using group identifiers assigned at the global manager when the groups are defined. The definitions of these groups are distributed to the sites across which the policies are spanned so that controllers at these sites can configure the data plane to enforce the policies.

[0015] As discussed above, service rules for a given policy are enforced on data messages associated with a set of logical network endpoints spanning one or more sites. In some embodiments, service rules have multiple fields, including a source field, a destination field, and an action field. Some embodiments reference these groups in these fields by using a group identifier assigned to the group (e.g., assigned by a global manager when the group is defined). For example, when a group identifier for a group is referenced in the source field of a service rule, the service rule enforces on data messages originating from machines in that group. When a group identifier for a group is referenced in the destination field of a service rule, the service rule enforces on data messages directed to machines in that group.

[0016] Service rules are enforced on data messages whose attributes match those specified in the rules. For example, the rule might specify a flow 5-tuple (source IP address and port number, destination IP address and port number, and transport protocol). Alternatively, the rule might specify other attributes of the flow. The PFE at each site identifies matching flows by performing a match classification operation on each data message to determine whether the flow attributes match those specified in the service rules. In some embodiments, when a data message matches a service rule, it is provided to a service machine that performs the action specified by the rule on the matching data message. These actions include dropping the data message, allowing the data message, or performing other service operations on the data message. For example, a service operation might modify the data message header to redirect the message to a different destination. These service operations include load balancing, firewalls, Dynamic Host Configuration Protocol (DHCP), Network Address Translation (NAT), and other services.

[0017] Some embodiments define a domain as a group of one or more sites. In some embodiments, the domain is defined at a global manager, for example, by an administrator of a logical network. Some domains are specific to a single physical site and are referred to as locations. Other domains are logical groups of one or more sites and are referred to as zones. Some embodiments restrict each site to membership in a single location and a single zone. In other words, a location may have no more than one site, and a site may have no more than two zones. The member sites of each domain define the span of the domain. In some embodiments, the domain is defined at a global manager and represented as a node in a policy tree. As mentioned above, policies are also defined as nodes in the policy tree, and in some embodiments, policies are defined as child nodes of domain nodes. In such cases, policies (and therefore any service rules of the policies) inherit the span of their parent domain, i.e., policies are enforced at sites that are members of their parent domain.

[0018] Furthermore, in some embodiments, groups are also represented as child nodes of a domain. In such cases, the group inherits the span of its parent domain; that is, the group is defined at a site that is a member of its parent domain. If the group is defined at a site (i.e., if the group spans that site), then logical network endpoints located at that site can join the group.

[0019] In some embodiments, the service rule distributed to the first set of sites to apply a policy to a first group of machines defined at those sites refers to a second set of machines defined at a different second set of sites. In other words, the span of the referenced group does not include the span of the policy. Because such a service rule references a group of machines that are not defined for one or more sites in the first set of sites, not all machines in the first group can enforce the policy. Therefore, some embodiments identify which sites in the first set of sites do not have a definition for the referenced group and distribute the definition of the referenced group to those identified sites.

[0020] For example, a policy can be defined in a first domain that has a first set of sites as members, while the groups referenced by the policy are defined in a different second domain that has a second set of sites as members. Thus, the policy inherits a span from the first domain, and the referenced groups inherit a span from the second domain. In such embodiments, the definition of the referenced group is distributed to the identified sites by expanding the span of the group to include the policy's span. In some embodiments, this is done at the global manager (e.g., by an administrator of the logical network) by creating a reference group in the first domain that references the group in the second domain. Upon receiving this modification, the global manager repeats the span calculation and determines that the group now spans the first set of sites. The global manager then provides the portion to be configured (which now includes the referenced groups) to the local managers at the first set of sites, who are now able to configure their respective controllers to configure the physical forwarding elements and service machines at the first set of sites to enforce the rules.

[0021] Some embodiments provide the implemented status of one or more logical network elements of a logical network upon request. In some embodiments, a global manager receives a request for the implemented status of a logical element from the administrator of the logical network. For each logical network element in the request, the global manager identifies the sites that the element traverses and sends the request to the local managers of those identified sites. In some embodiments, these requests are sent in parallel. After receiving the implemented status from each site traversed by each logical network element, the global manager combines the received implemented status from each site and presents the implemented status in response to the initial request. For example, in some embodiments, the implemented status may be presented to the administrator of the logical network in a report.

[0022] The realized state of a logical network element, in some embodiments, indicates whether the desired configuration (also referred to as the desired state) of the logical network element has been implemented at each site that the element traverses. Some embodiments describe the realized state in part based on deviations from the desired state. In some embodiments, the realized state for a logical network element (e.g., a logical router, logical switch, etc.) describes how the logical network element is implemented at each physical site it traverses. For example, the realized state includes data from the PFE executed on the computing device at each traversed site to implement the logical network element. In some embodiments, the traversed sites are identified by performing the span calculation as described above, or determined according to the definition of the logical network element (e.g., by the administrator of the logical network). For each logical network element, the realized state received from each local manager is a subset of all realized states for that logical network element, just as the desired state of the logical network element provided to the local manager is a subset of the global desired state of the logical network element defined at the global manager.

[0023] In some embodiments, the Global Manager also receives notifications from the Local Manager. Unlike on-demand requests for implemented states (e.g., pulled from the Local Manager), these notifications are automatically pushed from the Local Manager. In some embodiments, the notifications relate to site-specific implementation issues or situations. In some embodiments, notifications are provided to the GM when the events they describe occur (e.g., in real-time or near real-time rather than on demand). In some embodiments, these notifications are also displayed as reports (e.g., for review by administrators of the logical network).

[0024] In some embodiments, one type of notification is an alert from the local manager regarding a failure of the implementation of a logical network element at a specific site. For example, the failure could be due to an IP address conflict with another network element locally defined by the local manager, which the global manager is unaware of.

[0025] Another type of notification in some embodiments is a message from the local manager indicating that the network administrator has overridden the desired state of a logical network element at that particular site. This occurs in some embodiments when an administrator (e.g., different administrators possibly located in different regions or even countries) modifies the logical network using the local manager instead of the global manager. Some embodiments limit the overriding of desired states to networking-related configurations and prevent such overriding of security-related configurations (e.g., security policies).

[0026] In some embodiments, the third type of notification is a message from the local manager regarding the failure of a logical network element removal at that site. In other words, the logical network element was removed at the global manager (e.g., because it is not needed or is not used), but it remains in use at the local manager of one site. An example could be a first logical router locally defined at the local manager of one site, connected to a second logical router defined at the global manager. The global administrator is unaware that the first logical router does not exist at other sites, and the global administrator could attempt to remove the second logical router. The local manager at the site where the second logical router is in use as a link for the first logical router then sends a notification to the global manager informing them that the second logical router has not yet been removed from that site.

[0027] An additional example of the type of notification in some embodiments is a message from the local manager regarding the migration of a logical network endpoint (e.g., a virtual machine) from one site to another. Such endpoints are typically attached to logical ports of logical network elements (e.g., logical switches) spanning two sites. In some embodiments, when an endpoint is migrated to a different site, it maintains its association with the same logical port and seamlessly inherits the same security policies associated with that port, even at the new site.

[0028] In some embodiments, a machine can be migrated from one physical site to another while retaining the state of the network policies applied to that machine. For example, for a logical segment (logical switch) spanning at least two physical sites, in some embodiments, a machine attached to a logical port of the segment at one of the spanned sites is migrated to a new logical port of the same segment at another of the spanned sites. In some embodiments, migrating a machine includes creating a new machine on a new host computer (at the new site), transferring configuration data from the original machine to the new machine, and removing the original machine from its host computer. In some embodiments, the configuration data includes physical state data and logical network policies.

[0029] In some embodiments, machine state (e.g., runtime data) is transferred from a computing device executing the original machine at a source site (i.e., the source host) to a computing device executing the new machine at a destination site (i.e., the destination host). In some embodiments, the state includes a local port identifier assigned by a local manager at the source physical site and a global port identifier assigned by a global manager. In some embodiments, the global port identifier is associated with a logical network policy defined by a user client (e.g., by a network administrator) as part of the desired configuration of the logical network. Upon receiving the transferred state data, the destination host overwrites at least a portion of the new machine's state with the received state of the old machine.

[0030] In some embodiments, the destination host extracts a global port identifier from the received state data and provides it to the local manager of the destination site. In some embodiments, the local manager uses the global port identifier to retrieve logical network policies associated with the global port identifier from the global manager and applies these policies to new machines. In some embodiments, the local manager generates configuration data from the policies and provides the configuration data to the control plane of the logical network at the destination site (e.g., a set of controllers), and then distributes the configuration data to the destination hosts.

[0031] The foregoing overview is intended as a brief introduction to some embodiments of the invention. It is not intended to be an introduction or overview of all the inventive subjects disclosed in this document. The following detailed description and the accompanying drawings, with reference to the detailed description, will further describe the embodiments described in the summary, as well as other embodiments. Therefore, a full review of the summary, detailed description, and drawings is necessary to understand all the embodiments described in this document. Moreover, the claimed subject matter is not limited to the illustrative details in the summary, detailed description, and drawings, but is defined by the appended claims, as the claimed subject matter can be implemented in other specific forms without departing from the spirit of the subject matter. Attached Figure Description

[0032] The novel features of the present invention are set forth in the appended claims. However, for illustrative purposes, several embodiments of the invention are illustrated in the following drawings.

[0033] Figure 1 The diagrams illustrate simple examples of logical networks from some embodiments.

[0034] Figure 2 The logical routing components of a logical router are conceptually illustrated.

[0035] Figure 3 The concept illustrates the physical sites spanned by the logical network, as well as the host computers and edge devices that implement the logical network.

[0036] Figure 4 The diagrams conceptually illustrate examples of network management systems for some embodiments of a logical network spanning multiple physical sites.

[0037] Figure 5 A conceptual illustration is provided of a global manager running on a computing device separate from any computing device that executes the local manager at a given site.

[0038] Figure 6 The concept illustrates a global manager that runs on a computing device at a separate site that is not traversed by a logical network, and therefore there is no local manager.

[0039] Figure 7 Conceptual illustration Figure 4 The different components of the network management system described in the document.

[0040] Figure 8 The concept illustrates a cluster of managers and a distributed database system at physical sites spanned by a logical network.

[0041] Figure 9 The concept illustrates the generation of an update stream for use by the primary global manager to copy the desired configuration to the secondary global manager.

[0042] Figure 10 The illustration shows a process performed by a database instance to generate an update stream in some embodiments.

[0043] Figure 11 The concept illustrates an agent service that retrieves events from a database to create, update, or delete them and populates the associated persistent queue.

[0044] Figure 12 The illustration depicts a process performed by an agent service to populate a persistent queue in some embodiments.

[0045] Figure 13 The conceptual illustration depicts, in some embodiments, a process performed by the sending AR module at the main global manager for providing retrieved data to the auxiliary global manager and the local manager.

[0046] Figure 14 The conceptual diagram illustrates the architecture by which the local manager receives data from the main global manager via channels.

[0047] Figure 15 The illustration shows a process performed by the local manager when receiving data from the global manager in some embodiments.

[0048] Figure 16 The concept illustrates, in some embodiments, the process performed by the local manager when it receives CUD events directly from the user client rather than from the global manager.

[0049] Figure 17 The following is a conceptual illustration of an example of a global policy tree from some embodiments.

[0050] Figure 18 The concept illustrates an example of a security group spanning one or more physical sites.

[0051] Figure 19 The conceptual illustration illustrates, in some embodiments, the process performed by the global manager to parse the global desired configuration in order to populate the global policy tree and to identify the relevant portions of the global desired configuration for each physical site.

[0052] Figure 20 The process of generating a policy subtree, performed by the global manager in some embodiments, is conceptually illustrated.

[0053] Figure 21 A conceptual illustration shows an example of a global policy subtree for a single physical site.

[0054] Figure 22 A conceptual illustration of the local policy tree at a single physical site is provided.

[0055] Figure 23 The matching-action table of some embodiments is conceptually illustrated.

[0056] Figure 24 The concept illustrates, in some embodiments, a process performed by the global manager to expand the span of a group for applying cross-domain policies.

[0057] Figure 25A -B conceptually illustrates an example of expanding group span due to a cross-domain strategy.

[0058] Figure 26 The conceptual illustration depicts a process performed by the global manager in some embodiments to retrieve the implemented state of logical network elements.

[0059] Figure 27 The concept illustrates the distributed and collected operations of the global manager to retrieve the implemented state of logical network elements across multiple physical sites on demand.

[0060] Figure 28A -C conceptually illustrates an example of a user interface for presenting the implemented state of a logical router to an administrator of a logical network.

[0061] Figure 29A -B conceptually illustrates examples of user interfaces used in some embodiments to present the implemented state of logical switches to administrators of logical networks.

[0062] Figure 30 The diagram illustrates an example of an alert caused by the failure to create the requested logical network element at the local manager.

[0063] Figure 31 The diagram illustrates an example of an alert caused by an attempt to modify global configuration.

[0064] Figure 32A -B conceptually illustrates an example of an alarm caused by the failure to remove a logic network element.

[0065] Figure 33 Examples of some implementations for migrating virtual machines between sites within a region while preserving the machine’s group membership and associated policies are illustrated.

[0066] Figure 34A -C conceptually illustrates the information exchange during the creation and migration of virtual machines between the global manager, local manager, and source and destination hosts.

[0067] Figure 35The conceptual illustration depicts a process performed by the source host during the creation and migration of virtual machines in some embodiments.

[0068] Figure 36 The conceptual illustration depicts a process performed by the destination host during the creation and migration of virtual machines in some embodiments.

[0069] Figure 37 An electronic system that implements some embodiments of the present invention is conceptually illustrated. Detailed Implementation

[0070] In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the described embodiments and can be practiced without discussing some of the specific details and examples.

[0071] Some embodiments of the present invention provide a network management system for managing a federated logical network spanning multiple physical sites (e.g., data centers). The desired configuration of the logical network is defined by one or more network administrators using a set of network managers. The logical network includes logical network elements spanning one or more sites and logical network policies applied to the elements at those sites. Any such elements or policies spanning multiple physical sites are defined through a global manager.

[0072] In some embodiments, the global manager (e.g., via a user client) receives the global desired configuration for the logical network, identifies the relevant portions of the global desired configuration for each physical site, and provides the identified portions to the corresponding local manager for each site. Furthermore, the global manager collects information from the local managers regarding the implemented state of the logical network across sites and provides this information (e.g., to the administrator of the logical network) for troubleshooting and management purposes.

[0073] Some embodiments of the logical network elements include logical forwarding elements (LFEs), such as logical switches (to which logical network endpoints are attached) and logical routers. Each LFE (e.g., a logical switch or a logical router) is implemented across one or more physical sites, depending on how the network administrator defines the LFE. In some embodiments, the LFE is implemented within a physical site by physical forwarding elements (PFEs) that also host logical network endpoints (e.g., PFEs executed in virtualization software on host computers) and / or on edge devices within the physical site. The LFEs transmit data traffic between logical network endpoints (e.g., data compute nodes (DCNs))—(i) within a data center, (ii) between logical network endpoints in different data centers, and (iii) between logical network endpoints within a data center and endpoints outside the logical network (e.g., outside the data center). In some embodiments, the edge device is a compute device, which may be a bare metal machine on which the data path is executed and / or a computer on which the logical network endpoint executes the data path. In some embodiments, these data paths perform various gateway operations (e.g., gateways for extending logical switches across physical sites, gateways for performing centralized features of logical routers, such as performing stateful services and / or connecting to external networks).

[0074] Figure 1 A simplified example of a logical network 100 according to some embodiments is conceptually illustrated. This logical network 100 includes a Layer 0 (T0) logical router 105, a Layer 1 (T1) logical router 110, and logical switches 115 and 120. Although not shown, various logical network endpoints (e.g., virtual machines, containers, or other types of data compute nodes) are attached to the logical ports of logical switches 115 and 120. These logical network endpoints execute on host computers in the physical sites spanned by the logical switches to which they are attached. In this example, both the T0 logical router 105 and the T1 logical router 110 are defined to span three physical sites, but in other embodiments, such routers can span any number of sites. In some embodiments, logical switches 115 and 120 inherit the span of the logical router 105 to which they are connected.

[0075] As in this example, in some embodiments, a logical router may include a T0 logical router (e.g., router 105) that is directly connected to the external network 125 and a T1 logical router (e.g., router 110) that isolates a set of logical switches from the rest of the logical network and can perform stateful services for endpoints connected to those logical switches. In some embodiments, these logical routers 105-110 are defined by the network administrator as having one or more routing components, depending on how the network administrator configures the logical routers.

[0076] Figure 2 A logical network 200 is conceptually illustrated, showing logical routing components of logical routers 105 and 110, and various logical switches connected to these logical components and interconnecting the logical components with each other. Logical routers 105-110 span three physical sites 201-203, with some components of each router spanning all three sites, while other components span only a single site. As shown, T1 logical router 110 includes a collection of distributed routing components (DR) 205 and centralized routing components (also referred to as serving routers or SRs) 210-220. In some embodiments, the T1 logical router may have only a DR, or it may have both a DR and an SR. For the T1 logical router, the SR allows centralized (e.g., stateful) service to be performed on data messages sent between (i) logical network endpoints connected to logical switches connected to the T1 logical router and (ii) logical network endpoints connected to other logical switches not connected to the Layer 1 logical router or from external network endpoints. In this example, data messages sent to or from logical network endpoints connected to or from logical switches 115 and 120 will have stateful services applied by one of the SR 210-220 of the T1 logical router 110 (specifically, by the main SR 215).

[0077] In some embodiments, a T1 logical router may be connected to a T0 logical router (e.g., T1 logical router 110 is connected to T0 logical router 105). As mentioned, these T0 logical routers 105 handle data messages exchanged between logical network endpoints and external network endpoints. As shown, T0 logical router 105 includes a set of DR 225 and SR 230-240. In some embodiments, the T0 logical router includes an SR (or multiple SRs) operating in each physical site traversed by the logical router. In some or all of these physical sites, T0 SR 230-240 are connected to external routers 241-243 (or top-of-rack (TOR) switches providing connectivity to external networks).

[0078] In addition to logical switches 115 and 120 (which span all physical sites traversed by T1 DR 205), Figure 2Various automatically defined logical switches are also illustrated. Within each physical site, T1 DR 205 is connected to its corresponding local T1 SR 210-220 via corresponding transit logical switches 245-255. Similarly, within each physical site, T0 DR 225 is connected to its corresponding local T0 SR 230-240 via corresponding transit logical switches 260-270. Furthermore, router link logical switch 275 connects the primary T1 SR 215 (which performs stateful services for T1 logical router 110) to T0 DR 225. In some embodiments, similar router link logical switches are defined for each other physical site, but are marked as off.

[0079] Finally, the network management system also defines the backplane logical switches connecting each set of SRs. In this case, there is a backplane logical switch 280 connecting T1 SRs 210-220 and a backplane logical switch 285 connecting T0 SRs 230-240. Unlike transit logical switches, these backplane logical switches are extended across the physical sites spanned by their respective logical routers. When an SR for a particular logical router routes a data message to another SR for the same logical router, the data message is sent according to the appropriate backplane logical switch.

[0080] As mentioned, the LFE of a logical network can be implemented by the PFE executed on the source host computer as well as by the edge device. Figure 3 The diagram conceptually illustrates physical sites 305-315 spanned by a logical network 300, and host computer 320 and edge devices 325 implementing the logical network. Virtual machines (VMs) (in this example) or other logical network endpoints operate on host computer 320, which executes virtualization software to host these VMs. In some embodiments, the virtualization software includes PFEs, such as virtual switches and / or virtual routers. In some embodiments, one PFE (e.g., a flow-based PFE) executes on each host computer 320 to implement multiple LFEs, while in other embodiments, multiple PFEs execute on each host computer 320 (e.g., one or more virtual switches and / or virtual routers). In still other embodiments, different host computers execute different virtualization software with different types of PFEs. In this application, "PFE" is used to refer to a collection of one or more PFEs that execute on a host computer to implement an LFE for one or more logical networks.

[0081] In some embodiments, edge device 325 executes data paths 327 that implement one or more LFEs (e.g., Data Plane Development Kit (DPDK) data paths). In some embodiments, logical router SRs are assigned to and implemented by edge devices (SRs are centralized and therefore distributed in a different manner than DRs or logical switches). In different embodiments, the data paths 327 of edge device 325 may execute in the main operating system of a bare-metal computing device and / or in a VM or other data computing nodes (which are not logical network endpoints) operating on the edge device.

[0082] In some embodiments, as shown, edge device 325 connects physical sites 305-315 to each other (and to an external network 125). In such embodiments, host computers 320 within a physical site can send data messages directly to each other, but send data messages to host computers 320 in other physical sites via edge device 325. When a source logical network endpoint (e.g., VM 330) in the first physical site 305 sends a data message to a destination logical network endpoint (e.g., VM 335) in the second physical site 310, the data message is first processed by PFE 340, which is running on the same host computer 320 as the source VM 330, then by edge device 325 in the first physical site 305, then by edge device 325 in the second physical site 310, and finally by PFE 345, which is running on the same host computer 320 as the destination logical network endpoint 335.

[0083] More specifically, when a logical network endpoint sends a data message to another logical network endpoint, the PFE executing on the host computer where the source logical network endpoint resides performs logical network processing. In some embodiments, the set of source host computer PFEs (collectively referred to herein as source PFEs) performs processing on as many logical networks as possible (referred to as first-hop logical processing). That is, the source PFEs process the data message through the logical networks until (i) a destination logical port for the data message is determined or (ii) the data message is logically forwarded to an LFE (e.g., a logical router's SR) to which the source PFE cannot perform processing.

[0084] Figure 4A conceptual illustration shows an example of a network management system 400 for some embodiments of a logical network spanning physical sites 405-415. The network management system 400 includes (i) a global manager 420 that manages the entire logical network spanning all physical sites 405-415 and (ii) local managers 425-435 for each site that manages the logical network at their respective sites. In addition to the local managers, each physical site also includes a central controller, a host computer, and edge devices (not shown). In some embodiments, the global manager 420 executes on a computing device 445 at one of the sites 405 spanned by the logical network, and the local managers 425-435 also execute on computing devices 450-455 at their respective sites 410-415.

[0085] In some embodiments, the global manager 420 receives a globally desired configuration for the logical network via one or more user clients 440. In some embodiments, each local manager 425-435 may also receive a (site-specific) desired configuration for the logical network via user client 440. In some embodiments, the desired configuration is provided from user client 440 to managers 420-435 and 460 using a representative state transition (REST) ​​application programming interface (API), and... Figure 4 The middle part is indicated by a dashed line. Global Manager 420 also provides each of the Local Managers 425-435 with site-specific portions of the globally desired configuration, such as... Figure 4 The dotted lines are shown in the diagram.

[0086] In some embodiments, such as Figure 4 As shown, the global manager 420 executes on the same computing device 445 at a given physical site 405 as the local manager 425 that manages that site. In other embodiments, such as Figure 5 As shown, the global manager 420 executes on a computing device 505 at a given physical site 405, separate from any local manager 425 that manages that site. In other embodiments, such as... Figure 6 As shown, the global manager 420 is executed on a computing device 605 at a separate site 610 that is not traversed by a logical network, and therefore there is no local manager.

[0087] Some embodiments employ a secondary global manager 460, which is in an active-standby arrangement with the (primary) global manager 420. The primary global manager 420 is asynchronously synchronized with the secondary global manager 460 (e.g., not in real time) as a backup for failover scenarios. This asynchronous replication is handled by... Figure 4The dashed lines in the diagram represent the active global manager. For resilience, the secondary global manager 460 is located on a different physical site 410 than the primary global manager 420 on site 405, and maintains an independent database. This ensures that failover scenarios due to connectivity issues with the active global manager's physical site do not automatically affect the secondary global manager's physical site, and leverages a multi-site architecture with a federated logical network.

[0088] In some embodiments, the auxiliary global manager 460 executes on the same computing device 450 as the local manager 430 that manages its site 410, such as Figure 4 As shown in the illustration. In other embodiments, such as... Figure 5 As shown, the auxiliary global manager 460 executes on a different computing device 510 at its physical site 410, separate from any local manager 430 that manages that site 410. In other embodiments, such as... Figure 6 As shown, the auxiliary global manager 460 executes on a computing device 615 at a separate site 620 that is not traversed by a logical network, and therefore there is no local manager. Although Figure 4-6 The illustration shows an example where the primary global manager 420 and the secondary global manager 460 have exactly the same managed arrangement; however, in some embodiments, any combination or arrangement of managed functions may be used as needed. As an example only, the primary global manager 420 may be co-located with a local manager (e.g., Figure 4 (in the middle), and the auxiliary global manager 460 can be executed at physical sites that are not crossed by logical networks (such as...). Figure 6 (as shown in the image).

[0089] The primary global manager 420, secondary global manager 460, and local managers 425-435 are, in some embodiments, separate modules of a single application, while in other embodiments, they are separate applications. In some embodiments, these applications execute as one or more processes within a machine, running on a host computer at each physical site. Some embodiments deploy one or more of managers 420-435 and 460 as a cluster of machines at their physical sites, each running on a different computing device at the same site, as referenced below. Figure 8 Further detailed description.

[0090] Figure 7 Conceptual illustration Figure 4The network management system 400 described herein comprises various components. The desired configuration of the logical network is received by the primary global manager 420 from user clients 440 (e.g., via a wide area network 442 such as the Internet), stored in database 710, and copied to a secondary global manager 460 for storage in a separate database 712. Site-specific portions of the desired configuration are also provided to each local manager (via a dedicated asynchronous channel) 425-435 for storage in their respective databases (not shown).

[0091] Global Manager 420 also includes several additional modules, including an API processor 706 for receiving user client input via a REST API, a core global manager service 707 for writing data to a database 710, a persistent work queue 711 for maintaining the causal relationships of incoming create / update / delete (CUD) events in the database 710, a log replication module 730 for replicating CUD events to a secondary global manager 460 at the database 460, an agent / span service 735 for performing span calculations on CUD events (and the logical network elements referenced therein), an asynchronous replication (AR) module 740 including dedicated persistent queues 725-727 for propagating CUD events to different local managers at different physical sites, and a site manager 745 for maintaining connection parameters used by the AR module to establish channels with other local managers. Each of these modules and its function will be described in further detail below.

[0092] In some embodiments, databases 710 and 712 are distributed databases (e.g., shared logs) implemented on a collection of storage devices at the respective physical sites of the administrator. Furthermore, in some embodiments, global managers 420 and 460 are implemented as clusters of machines running on separate computing devices at their respective physical sites. Figure 8 This conceptual illustration depicts a manager cluster and a distributed database system located at a physical site spanned by a logical network. The manager cluster shown in this example can serve as a primary global manager in active mode, a secondary global manager in standby mode, or a local manager in different embodiments. Managers are applications running on several manager machines 805-815, executed on individual host computers 820-830 at the physical site, communicating with each other via a physical network 735 (e.g., a data center network architecture) at the physical site. In some embodiments, one of the manager machines 805 is designated as the leader of the cluster, sharing tasks with the other manager machines in active / active mode (e.g., using load balancing), active / standby mode, or some combination of both. The cluster of manager machines 805-815 appears as a single logical manager to the other managers in the logical network.

[0093] In some embodiments, the manager application running on machines 805-815 has separate modules for a global manager and a local manager, which can be enabled or disabled as needed. Alternatively or in combination, in some embodiments, at least one of machines 805-815 is a dedicated global manager machine with a separate machine (not shown) for running the application for the local manager, on the same host computer 820-830 or a different host computer (as referenced above). Figure 4-6 Executed on (as described above).

[0094] In some embodiments, the manager cluster stores desired configuration data in a distributed database system managed by one or more instances 840-850 of a database running on host computers 820-830. In some embodiments, the databases run within a local manager machine on the host, but for clarity they are shown separately in the diagram. Database instances 840-850 communicate with each other via a physical network 835 at a physical site (e.g., a data center network architecture) used by the manager cluster. The database instances 840-850 collectively appear to the manager cluster as a single logical database 710. In some embodiments, instances are fragments or slices of the database. In other embodiments, each instance is a node with a complete copy of the data (e.g., such as...). Figure 8 (As shown in the example). Redundancy allows for durability in the event of a failure in one of the host 820-830.

[0095] In some embodiments, database instances 840-850 are nodes of a distributed log stored on host computers 820-830. Entries in the distributed log provide an ordered, persistent update history of the state of different logical network elements and logical network policies, which the manager cluster accesses via an application programming interface (API) provided by database instances 840-850. The distributed log and database API are described in more detail in U.S. Patent 10,540,119, which is incorporated herein by reference.

[0096] Return to Figure 7For example, data describing the expected global configuration is received from user client 440 and stored in database 510 in some embodiments using a series of transactions initiated by user client 440 via a series of REST API calls to the main global manager 420. In some embodiments, these API calls are received and processed by API processor module 706, which then provides the received data to manager service 707, which performs the core functions of global manager 420. Manager service 707 stores the received data in database 710. In some embodiments, the data is stored in the database in the form of tables that store configuration parameters of logical network elements of the logical network. In some such embodiments, the data in the tables is expressed as a hierarchical tree of user intent, as referenced below. Figure 17 As described. Furthermore, in some embodiments, the manager service 707 also stores (e.g., replicates) a series of incoming transactions in a work queue 711 to maintain their causal relationships and order. In some embodiments, the work queue is a persistent queue in a database 712 and is also replicated across database nodes for redundancy.

[0097] In some embodiments, manager service 707 is a cluster of manager machines 805-815, as referenced above. Figure 8 As described. Figure 7 As shown, in some embodiments, the secondary global manager also has corresponding API processors and manager services that are activated in a failover scenario when the primary global manager fails (e.g., becomes inaccessible) and the secondary global manager becomes the active manager.

[0098] In some embodiments, the database 710 generates one or more update streams from a series of transactions. Figure 9 A conceptual illustration shows the generation of an update stream for use by the primary global manager 420 to replicate the desired configuration to the secondary global manager 460. Figure 10 refer to Figure 9 The illustration shows a process 1000 performed by database instance 840 to generate an update stream in some embodiments.

[0099] Process 1000 begins at 1005 with the receipt of data describing the desired configuration of the logical network. In some embodiments, the received data consists of one or more Create, Update, or Delete (CUD) events received as a series of API transactions at the Global Manager 420, each CUD event affecting one or more logical network elements across one or more physical sites. For example, in Figure 9 In this context, database instance 840 receives CUD event 905 from global manager service 707.

[0100] At 1010, process 1000 uses metadata associated with each transaction to tag the received CUD events, such as timestamp information that can be used for data sorting and database status to prevent race conditions for access. In some embodiments, the metadata also includes parameters associated with API calls, such as user ID, source address, etc. Furthermore, in some embodiments, the metadata includes span information specified (e.g., by an administrator) when the CUD event is defined by the user client 440. Figure 9 In the example, database instance 840 appends metadata 910 (represented by "m") to the data before storing it in database 710.

[0101] In some embodiments, not all data stored in the database by the global manager is required for replication. Data to be replicated in some embodiments includes policy tables, licenses, physical site information, and other data that the secondary global manager will need to present an active state in the event of a failure of the primary global manager. Other database tables, such as those related to network management in active mode, are not required for replication to the secondary global manager in standby mode. Furthermore, state information regarding the implementation state of the logical network does not need to be replicated to the secondary global manager, as the implementation state will become outdated in a failover scenario. Some embodiments distinguish between data to be replicated and data not to be replicated by marking the data to be replicated.

[0102] In some embodiments, process 1000 also marks the CUD event (at 1010) with an identifier indicating that the event to be copied. For example, in Figure 9 In this embodiment, database instance 840 appends an identifier 915 (represented by "R") to an event intended for replication before storing it in database 710. In some such embodiments, database instance 840 does not append any identifier to data not intended for replication.

[0103] At point 1015, process 1000 stores the tagged CUD events along with at least some of the metadata in database 710. Specifically, in some embodiments, process 1000 stores the tagged data in each of database nodes 840-850. In embodiments where database 710 is a distributed shared log, the tagged data is appended to the log. Figure 9 In the example, the tagged data is appended to position 917 of database 710, which is the latest position in the database.

[0104] As described above, in some embodiments, database 710 is shared by the primary global manager 420 and the local manager 425 (e.g., on the same computing device 445, such as...). Figure 4(As shown in the example). In such embodiments, the local manager also writes the data corresponding to CUD events to the database, separately from the data written by the global manager. Figure 9 In this example, database instance 840 also receives CUD event 920 from local manager service 910, which receives the event from user client 440 via a series of API transactions. Unlike event 905 received from global manager service 707, event 920 affects only one or more logical network elements spanning physical site 405. In this example, database instance 840 receives event 920 from local manager service 910 before it receives event 905 from global manager service 707. Before storing event 920 in the database, database instance 840 appends metadata 925 and a distinct identifier 930 (represented by "L") to event 920. Thus, the tagged event is written to location 932 in database 710, which is then the most up-to-date location in the database.

[0105] At 1020, process 1000 pushes CUD event 905 and associated metadata 910 to log replication module 730. In some embodiments, process 1000 pushes the CUD event to work queue 711 in the database. Figure 9 (Not shown in the diagram), the log replication module 730 (and the agent service 735 described below) retrieves it from the work queue, for example, after notification. In some embodiments, when a CUD event is pushed to the log replication module 730, the database instance 840 does not include identifier 915. In some embodiments, process 1000 also notifies the global manager that CUD event 905 has been pushed to the queue for propagation to the relevant local managers, as referenced below. Figure 12-11 As described. Figure 9 As shown, data marked with different identifiers or without any identifiers will not be pushed to the log replication module 730. Then process 1000 ends.

[0106] As described above, database 710 generates an update stream that pushes newly written CUD events to log replication module 730 for replication to auxiliary global manager 460. Return to Figure 7In some embodiments, the global manager 420 also includes an asynchronous replication (AR) module 740 having several persistent queues 725-727. Each of these persistent queues 725-727 is dedicated to propagating desired configuration data to one of the local managers 425-435 at each physical site 405-415. In some embodiments, these queues are replicated across the manager cluster, so that in the event of a failure, the causality of database events is preserved and the network management system can resume from the point of interruption after the failure. In some embodiments, these queues are stored in active storage (e.g., rather than on disk).

[0107] When the primary global manager 420 receives the global desired configuration for the logical network, the global manager stores the portions of the global configuration in persistent queues 725-727 based on the relevance of each portion of the global configuration to the configuration of the logical network at the corresponding physical site in the queue. In some embodiments, the agent service 735 of the global manager identifies the relevant portions of the global desired configuration for each physical site, for example, based on the span of the logical network elements, as described in further detail below. In some embodiments, the span is determined by a span service, and in some embodiments, the span service is the agent service 735 (e.g., Figure 7 It is part of what is depicted in the text, but in other embodiments it is a separate service (as described below). Figure 11 (As shown in the diagram). In some embodiments, the desired configuration is received as one or more Create, Update, or Delete (CUD) events as a series of API transactions at the Global Manager 420, each CUD event affecting one or more logical network elements across one or more physical sites. In some embodiments, each CUD event is part of the global desired configuration.

[0108] Figure 11 The conceptual illustration shows the global manager's agent service 735 retrieving CUD events from database 712 and populating the relevant persistent queues 725-727 of AR module 740. Figure 12 refer to Figure 11 The example illustration depicts a process 1200 performed by agent service 735 to populate persistent queues 725-727 in some embodiments.

[0109] Process 1200 begins at 1205 with receiving a notification of a new CUD event already stored in database 710. In some embodiments, this notification is sent from global manager 420 or database instance 840 (e.g., as part of operation 1020 of process 1000). In response to this notification, agent service 735 retrieves the CUD event. In other embodiments, agent service 735 periodically queries the database for new CUD events. In any case, in some embodiments, CUD events are stored in work queue 711 in database 712 to maintain their causal relationships and order. Agent service 735 then retrieves each CUD event sequentially from this queue. Figure 11 In the example, after receiving CUD events 1125-1145 (labeled "A" to "F"), agent service 735 retrieves CUD event 1120 (labeled "F") from work queue 711.

[0110] At 1215, process 1200 determines whether database 710 has already stored span information 1147 for the CUD event. In some cases, the span for the CUD event is predefined (e.g., when the CUD event is specified via user client 440) and stored in database 712 as metadata associated with the event (e.g., metadata 910). If the span for the CUD event is available, then at 1217, process 1200 retrieves the span information and proceeds to operation 1230, which will be described below.

[0111] If the span for the CUD event is unavailable, then process 1200 determines at 1220 which physical sites the CUD event spans. In some embodiments, taking into account the definitions of the logical network elements referenced by the CUD event and the relationships between those logical network elements and other logical network elements in the logical network, agent service 735 invokes span service 736 to perform span calculation to determine the span. (See below for reference...) Figure 17 The span calculation is described in more detail. After determining the span for the CUD event, at 1225, in some embodiments, process 1200 also updates the database with the span information by storing the span information as metadata associated with the event.

[0112] Based on the span information, at point 1230, process 1200 stores a copy of the event in each persistent queue corresponding to one of the physical sites spanned. Figure 11In the example, after retrieving or determining the span 1147 for CUD event F 1120, agent service 735 determines that the span equals site A 405 and site C 415. Agent service 735 then copies CUD event F 1120 to the corresponding queues 725 and 727 respectively. Then process 1200 ends.

[0113] As described above, in some embodiments, a CUD event refers to one or more logical network elements. If a CUD event refers to different logical network elements with different span attributes, then in some embodiments, the agent service 735 copies the relevant portion of the CUD event (corresponding to different logical network elements) to the relevant queue, rather than the entire event.

[0114] As mentioned above, a persistent queue is a first-in, first-out (FIFO) queue. Figure 11 In the example, agent service 735 first receives CUD event A 1145 and determines that the span is site A 405 and site B 410. Therefore, event A 1145 is copied to queues 725 and 726. Since event A 1145 is the first event already queued in the sequence, it is the first event to be transferred from the queue to the corresponding local manager at the appropriate site (e.g., ...). Figure 11 (As shown in the diagram). Since the main global manager is also located at site A 405, event A 1145 is transmitted to local manager 425 via the physical network at the physical site (e.g., a data center network architecture). However, since site B 410 is a different physical site, event A 1145 must be transmitted to local manager 430 via a wide area network (e.g., the Internet).

[0115] Subsequent CUD events BF 1120-2940 are queued in a similar order to those received, such as... Figure 11 As shown, event B 1105 spans sites A 405 and C 415, event C 1135 spans site A 405, event D 1130 spans sites B 410 and C 415, event E 1125 spans sites A 405 and B 410, and event F 1120 spans sites A 405 and C 415 (as discussed above). Therefore, queue 1105 stores events B 1140, C 1135, E 1125, and F 1120; queue 1110 stores events D 1130 and E 1125, and queue 1115 stores events B 1140, D 1130, and F 1120. Events are transmitted from each FIFO queue in the same order as received.

[0116] Return to Figure 7In some embodiments, the AR module 740 at the global manager 420 maintains a set of asynchronous channels connecting the main global manager 420 to the local managers 425-435. These channels are depicted as dotted lines from persistent queues 725-727 to corresponding AR modules 755-765 at the local managers 425-435.

[0117] In some embodiments, AR module 740 maintains a dedicated asynchronous channel connecting the primary global manager 420 to the secondary global manager 460. In other embodiments, replication to the secondary global manager 460 is handled by a dedicated log replication module 730, which refers to... Figure 9 As described above, the log replication module receives data to be replicated from the work queue 711 in the database 712, and in some embodiments, the data (as depicted by the dotted line) is directly replicated to the database 712 of the auxiliary global manager 460, bypassing the AR module.

[0118] In some embodiments, the auxiliary global manager 460 also includes a receiver module 742 for establishing a channel with the log replication module 730 of the primary global manager, receiving replicated data, and writing the data to the database 712. If the auxiliary global manager 260 becomes active (e.g., due to failover), then in some embodiments, the receiver module 742 assumes the role of the replication module.

[0119] AR modules 740 and 750-765 maintain channels between physical sites and, in some embodiments, guarantee various connection parameters (e.g., minimum bandwidth, maximum round-trip time, etc.) required for data replication to the auxiliary global manager and distribute data to the local manager. In embodiments where the auxiliary global manager and / or local manager are implemented as a cluster of machines, the channels also identify the leader machine of each manager cluster.

[0120] In some embodiments, a set of site managers 745 are executed at the main global manager 420, which provides the AR module 740 with information about physical sites (e.g., connectivity statistics, IP addresses, etc.) for channel maintenance. Each site manager corresponds to one of physical sites 405-415. In some embodiments, the site managers 745 are executed separately from the AR module 740 (e.g., ...). Figure 7 (As depicted in the text), or in other embodiments as a submodule of the AR module. In such embodiments, the auxiliary global manager 460 also executes a collection of site managers (not shown), but these site managers are inactive prior to the failover scenario.

[0121] As described above, the agent service 735 queues CUD events to persistent queues 725-727 of the AR module 740. In some embodiments, the AR module 740 polls its persistent queues 725-727 to determine if a new CUD event exists. If data is found in the queue (e.g., corresponding to a CUD event), the AR module 740 retrieves the data (i.e., dequeues the CUD event) and transmits the retrieved data to the AR module of the local manager at the corresponding site via the channel corresponding to that queue. Figure 13 A conceptual illustration shows a process 1300 performed by the sending AR module 740 at the main global manager 420 in some embodiments for providing retrieved data to one of the local managers 425-435. Reference is made below. Figure 7 Describe process 1300.

[0122] Process 1300 begins at 1305 by querying a persistent queue dedicated to one of the local managers 425-435 at one of the physical sites 405-415. Process 1300 queries the selected queue to see if there are any new CUD events in the queue. The queue will potentially include CUD events copied to the queue by agent service 735 after span calculation, as described above. Figure 12 As described in [the document]. In some embodiments, a CUD event references one or more logical network elements, each spanning one or more physical sites.

[0123] If process 1300 determines (at 1310) that there is no new data in the selected queue, then process 1300 returns to 1305 after a specified time period. In some embodiments, the time period is a timeout value that varies depending on the specific connection parameters from the channel to the physical site. This, in some embodiments, depends in part on the location of both the global manager 420 and the specific local manager at the site associated with the persistent queue (e.g., as described above). Figure 4-6 As described in the text, different scenarios are used to place the global manager 420 and the local manager. If process 1300 determines (at 1310) that there is new data in the selected queue, then the process retrieves the data from the queue (at 1315).

[0124] At 1320, process 1300 then provides the retrieved data to the corresponding physical site via a channel connecting the AR module 740 of the main global manager 420 and the corresponding AR module of the local manager of the physical site. For example, if the selected queue is queue 725, then process 1300 uses the network architecture of site 405 through its dedicated channel (provided by...). Figure 7The dotted lines (represented in the diagram) transmit data to the AR module 755 of the local manager 425 at the same site 405. As another example, if the selected queue is queue 726, then process 1300 transmits data via WAN 771 through its dedicated channel (by...). Figure 7 (The dotted lines in the diagram represent the AR module 760 that transmits data to the local manager 435 at physical site 415.)

[0125] In this way, AR module 740 sends data to the destination corresponding to the queue from which data is retrieved via any necessary intermediate network infrastructure, whether that destination is at the same physical site or a different physical site. Then process 1300 ends.

[0126] Figure 14 A conceptual illustration shows an architecture in which the local manager receives data from the main global manager 420 via a channel. In some embodiments, the local manager 430 includes multiple modules, including a receiving AR module 665 for receiving data from the global manager 420, an entry queue 1402 for storing received events in the AR module, a local manager service 1410 that performs core local manager functions, and a database 1405 (in some embodiments, a distributed database, as referenced above). Figure 8 (as described above). In some embodiments, the local manager 430 also includes an API processor 1415 that receives input from the user client 440 via a REST API, and a handler 1417 that sends configuration data to the control plane to achieve the desired logical network configuration. These modules and their functions will be described in more detail below.

[0127] exist Figure 14 In the example, local manager 430 is located at a different physical site 410 than the primary global manager's site 405, so data is received via a wide area network 770 (e.g., the Internet). However, in some embodiments, the communication medium depends on the location of the local manager relative to the global manager 420. For example, if the local manager (e.g., local manager 425) is at the same physical site as the global manager, then in some embodiments data is received via the local physical network (e.g., data center architecture) at that site. As another example, if the local manager is located on the same host computer (e.g., such as...),... Figure 4 As shown in the figure, in some embodiments, data is received via inter-process communication methods.

[0128] As referenced above Figure 12In some embodiments, agent service 735 retrieves new data from database 710 and performs span calculations to determine queues 725-727 where data should be stored. Global manager's sending AR module 740 then retrieves data from each queue, as described above. Figure 13 The data is transmitted to the corresponding physical site via a dedicated channel. Each channel connects the transmitting AR module 740 of the main global manager 420 to one of the receiving AR modules 755-765 of the local managers 425-435. Figure 15 refer to Figure 14 The example illustration depicts a process 1500 performed by the local manager when receiving data from the global manager in some embodiments.

[0129] Process 1500 begins at 1505 with receiving data from the main global manager. Specifically, this is done via the sending AR module and receiving AR module (e.g., ...) of the local manager. Figure 14 A dedicated channel between the receiving AR module 765 of the local manager 430 and the sending AR module 740 of the main global manager receives data. In some embodiments, the receiving AR module 765 maintains a persistent entry queue 1402 to store the received data; in some such embodiments, this data is stored in active memory. In some embodiments, the received data corresponds to one or more CUD events, each CUD event referencing one or more logical network elements spanning one or more physical sites, including physical site 410. In some embodiments, the order and causality of the received data are maintained by the entry queue 1402.

[0130] At 1510, process 1500 determines whether the CUD events (one or more events) associated with the received data are valid. Verification is based on the absence of any errors or inconsistencies in the logical network configuration applied to the physical site. Furthermore, in some embodiments, other verification is the rule governing whether a logical network element can be updated to extend its span to a new site. For example, before extending an element to a new site, the security policy affecting that element must first be extended to the new site. Otherwise, a vulnerability will arise where the security policy applicable to the logical network element is not fully applied. In some embodiments, verification is performed by a local manager service 1410, which retrieves CUD events from the ingress queue 1402 and stores them in a database 1405.

[0131] If process 1500 determines that the CUD event is invalid, then a notification is generated at 1520 for the main global manager's failure to validate the CUD event. In some embodiments, the notification is a notification event queued in the exit queue (not shown) of AR module 765, which will be transmitted via the same asynchronous channel (e.g., Figure 14The notification event (the dotted line in the image) is sent back to the AR module 740 at the global manager 420. In other embodiments, the notification event is sent via an out-of-band notification channel. As part of the core functionality of the local manager service 1410, or as another module (not shown), notification events in the outgoing queue are retrieved and sent via a channel separate from process 1500.

[0132] In some embodiments, invalid events are not discarded but are persisted to maintain causality. Invalid events will still be accepted, and intervention (e.g., by a network administrator) will be required to resolve the invalidity. See below for reference. Figure 30-3 2. Describe the notification event in more detail. Therefore, process 1500 then continues to 1530.

[0133] At 1530, procedure 1500 applies the CUD event to the local desired configuration of the logical network at the physical site. In some embodiments, the desired configuration of the logical network is expressed as a policy tree, which will be referenced below. Figure 17-20 To describe it in more detail. Figure 14 In the example, the desired configuration of the logical network, for example, the configuration of the logical network elements spanning the physical site 410, is stored in the local database 1405.

[0134] For example, if the verified CUD event is a creation event, the logical network element defined by the event is created within the expected configuration stored in database 1405. If the CUD event is an update event, the expected configuration of the logical network element referenced by the event is updated within the expected configuration stored in database 1405. If the CUD event is a delete event, the logical network element referenced by the event is deleted within the expected configuration stored in database 1405.

[0135] At 1540, process 1500 uses the (now modified) desired configuration of the logical network to generate configuration data and provides it to the control plane of the logical network (e.g., a central controller or controller cluster at each site). In some embodiments, these controllers identify computing devices at the sites that execute physical forwarding elements and distribute the configuration data to the identified computing devices. In some embodiments, different logical network elements span different computing devices (e.g., host computers, edge devices, etc.). In some embodiments, each logical network element is implemented by a physical forwarding element that executes on a computing device identified at the site that the logical network element spans. In other words, a logical network element is implemented by at least one physical forwarding element at each site it spans. Some embodiments have a local controller (also called a chassis controller) that executes on one or more computing devices adjacent to the physical forwarding elements and receives configuration data from the controller cluster. The local controller uses the configuration data to configure the physical forwarding elements to implement the logical network elements at each computing device. Process 1500 then ends.

[0136] Figure 16 A conceptual illustration depicts a process 1600 performed by a local manager at a physical site in some embodiments when it receives CUD events directly from user client 440 rather than from global manager 420. This scenario occurs, for example, when a local administrator at a physical site (who may be the same as or different from the administrator of the globally federated logical network) modifies the desired configuration of a logical network implemented at the local site (e.g., by specifying a series of create, update, or delete events for logical network elements whose span includes the local site).

[0137] Procedure 1600 begins at 1605 by receiving CUD events directly from user client 440. For example, as... Figure 14 As shown, data describing CUD events is received from user client 440 and, in some embodiments, stored directly in database 710 using a series of transactions initiated by a series of REST API calls from the user client to the main global manager 420. In some embodiments, user client 440 is not on the same physical site, so local manager 430 receives CUD events via wide area network 442 (e.g., the Internet). In some embodiments, these API calls are received and processed by API processor module 1415 of local manager 430, which then provides the received data to local manager service 1410, which performs the core functions of local manager 430.

[0138] Process 1600 determines at 1610 whether a CUD event (one or more events) is valid. Verification is based on the absence of any errors or inconsistencies in the logical network configuration applied to the physical site. In some embodiments, verification is performed by the local manager service 1410 either directly upon receipt from the API processor 1415 or after retrieving the event from the database 1405.

[0139] If process 1600 determines that the CUD event is invalid, a notification of failure to validate the CUD event is generated at 1617. In some embodiments, the notification is a notification event that is provided to user client 440 for intervention (e.g., by a network administrator). Process 1600 then terminates.

[0140] If the process determines that the CUD event is valid, then at 1615 the process determines whether the event is a local event. In other words, it determines whether the CUD event only references logical network elements defined at the local site. If these elements are defined by the local manager, then these elements do not extend beyond the physical site span and are unknown to the global manager 420 in some embodiments. If the CUD event is a local event, then process 1600 continues to 1635, which will be described below.

[0141] If process 1600 determines that the CUD event is not a local event, i.e., it references a logical network element defined at global manager 420, then the process at 1620 determines whether the event overrides the expected configuration of the globally defined logical network element. In some embodiments, this determination is made by applying a set of priority rules to the CUD event to determine whether the CUD event is allowed to override the expected configuration of the globally defined CUD event. For example, some embodiments only allow local CUD events that override the expected configuration by means of network-related configurations (e.g., message forwarding rules and policies) or configuration profiles (timers, etc., which are affected by parameters of the local site, such as latency). In such cases, the local CUD event will have priority.

[0142] As another example, some embodiments prevent desired configurations from being overridden by local CUD events targeting security-related configurations. In such cases, globally defined desired configurations will take precedence. Furthermore, in some cases, the event is related to an emergency situation and can only be recognized by the local manager, thus overriding any relevant global configurations. If the event does not have priority to override global configurations (e.g., according to priority rules), then the process continues to 1617 as defined above.

[0143] At 1625, if process 1600 determines that the CUD event does indeed have a priority that overrides the desired configuration of the global definition, then at 1625 a notification for the main global manager to override the event is generated. In some embodiments, the notification is a notification event that is queued in the exit queue (not shown) of AR module 765 to be transmitted via the same asynchronous channel (e.g., Figure 14 The notification event (the dotted line in the image) is sent back to the AR module 740 at the global manager 420. In other embodiments, the notification event is sent via an out-of-band notification channel. For example, as part of the core functionality of the local manager service 1410, or as another module (not shown), notification events in the outgoing queue are retrieved and sent separately from process 1600 via a channel.

[0144] At 1630, process 1600 creates a local copy of the logical network element, whose configuration will be overridden by the CUD event. The original logical network element from global manager 420 is retained as a read-only object in the local manager's database 1405. This local copy (also known as the shadow object) is the target of the CUD event, not the original.

[0145] At 1635, process 1600 applies the CUD event to the local desired configuration of the logical network at the physical site. If the event is an overriding event, then process 1600 applies the CUD event to a shadow copy of the object instead of the original object received from global manager 420. In some embodiments, the desired configuration of the logical network is expressed as a policy tree, which will be referenced below. Figure 17-20 To describe it in more detail. Figure 14 In the example, the desired configuration of the logical network, for example, the configuration of the logical network elements spanning physical site 410, is stored locally in database 1405. CUD events are applied to the local desired configuration.

[0146] For example, if the CUD event is a creation event, the logical network element defined by the event is created within the local expectation configuration stored in database 1405. If the verified CUD event is an update event, the expectation configuration of the logical network element referenced by the event is updated within the local expectation configuration stored in database 1405. If the verified CUD event is a delete event, the logical network element referenced by the event is deleted within the local expectation configuration stored in database 1405.

[0147] At 1640, process 1600 uses the desired configuration of the logical network to generate configuration data and provides it to the control plane of the logical network (e.g., the central controller or controller cluster at each site). Then process 1600 ends.

[0148] As referenced above Figure 15-16The local manager 430 generates and provides configuration data based on the desired configuration of the logical network stored in the local database 1405. Figure 14 In the illustrated embodiment, the local manager service 1410 generates configuration data from the stored desired configuration and provides the generated data to the processor module 1417. The processor module 1417 then distributes the configuration data to a central controller cluster 1420 of one or more controllers. The controller cluster 1420 identifies the host computer 320 and edge devices 325 to which the configuration data is distributed. At each edge device 325, the configuration data is used to configure the edge's data path 327. At each host computer 320, the configuration data is received by the chassis controller 1425, which uses it to configure at least one PFE 345 executing on the host 320 to forward data messages to and from one or more attached VMs 1430.

[0149] In some embodiments, in addition to distributing configuration data to computing devices such as host computer 320 and edge device 325, controller 1420 also receives physical network-to-logical network mapping data from the computing devices and shares this information across data centers. For example, in some embodiments, these mappings are between logical addresses (e.g., the MAC address of a logical network endpoint, i.e., the VM 1430 running on the computing device) and physical addresses (e.g., the IP address of a tunnel endpoint at the computing device). Controller 1420 retrieves and shares tunnel endpoint-to-logical network address mapping data between host computer 320 and edge device 325 at physical site 410, and also shares that data with controllers at other physical sites 405 and 415, allowing those controllers to share this data with the hosts and edge devices at their own sites.

[0150] In some embodiments, the computing device at each site also acts as a machine, along with a physical forwarding element and a local controller. These machines include logical network endpoints (e.g., computers, virtual machines, containers, etc.) that are sources and destinations of data message traffic, and service machines (e.g., firewalls, load balancers, etc.) that perform services on the data traffic. Machines reside at a single site at a given time, but in some embodiments may migrate between sites. In some embodiments, these machines are connected to a physical forwarding element to exchange data messages within the network.

[0151] In some embodiments, the globally desired configuration of the logical network is expressed as a hierarchical tree (also known as a global policy tree) with nodes and connections between nodes. The global policy tree is stored in its database 710 by a primary global manager 420. A replicated global policy tree is also stored in its database 712 by a secondary global manager 460. In some embodiments, nodes represent logical network elements spanning one or more sites and logical network policies applied to those elements, and connections represent relationships between nodes (e.g., parent-child relationships, logical network connections, etc.). Cross-references between nodes are implemented by referencing paths through the tree hierarchy (e.g., global architecture / site / London / enforcement point / edge cluster), which provides information about the span of each node. For logical networks spanning multiple physical sites, Figure 17 A conceptual illustration shows an example of such a global policy tree 1700 from some embodiments.

[0152] Logical network elements include logical forwarding elements that forward data within a logical network, such as logical routers and logical switches. For example, in... Figure 17 In this example, the global policy tree root 1702 connects a single Layer 0 logical router T0 1705, two Layer 1 logical routers T1A 1710 and T1B 1715, and two different types of network segments. These segments are overlapping network segment 1720 and VLAN segment 1725. The node for router T0 1705 has multiple child nodes, including static route definition 1730 and locale services 1735 and 1740 referencing physical sites A 1760 and B 1777. In this example, router T0 1705 also spans site C, but for clarity, the corresponding locale service referencing site C 1765 is not shown in the diagram. The node for router T1A 1710 has a child node for logical switch 1745. The node for router T1B has a child node 1715 referencing locale service 1750 for physical site A.

[0153] The Local Area Service (LAS) nodes used for T0 and T1 routers define the span of these routers. For example, router T0 1705 spans sites A 1760, B 1777, and C 1765, while router T1B 1715 spans site A 1760. As more LAS are added to a T0 or T1 router, the router is extended to the corresponding sites. Unlike router T1B 1715, router T1A 1710 does not have a LAS child node, but instead has a reference to router T0 1705 (dashed line). Therefore, router T1A 1710 inherits the span of router T0 1705 (i.e., router T1A spans sites A 1760, B 1777, and C 1765). Some child nodes also automatically inherit that span in some embodiments. Thus, static route definition 1730 under T0 router 1705 also spans sites A 1760, B 1777, and C 1765. Logical switch 1745 inherits the span of its parent router T1A 1710, which in turn derives its span from a reference to router T0 1705. Therefore, logical switch 1745 also spans sites A 1760, B 1777, and C 1765.

[0154] Each node in the global policy tree 1700 has multiple attributes that define configuration parameters, some of which are user-defined and others inherited. In some embodiments, span is not the only attribute that child nodes inherit from their parent nodes. For example, certain T0 or T1 routers spanning more than one site may assign one physical site as the primary site and the others as secondary sites. If such a logical router has multiple Serving Router (SR) components, the SR component at the primary site will take precedence over certain operations. This configuration is specified for the router (e.g., by the network administrator) and is not part of the locale service configuration under the router.

[0155] Localization service nodes 1735, 1740, and 1750 have references (dashed lines) to edge clusters 1751 and 1752 at corresponding sites A 1760 and B 1777. As mentioned above, in this example, T0 router 1705 also spans site C 1765, but for clarity, the localization service for that site's router is omitted, and therefore the corresponding reference to the edge cluster under site C node 1765 is omitted. In some embodiments, the localization service nodes are related to the above references. Figure 2The described service is associated with a router. The edge cluster is described below with reference to site nodes. In some embodiments, the local service node also has various types of child nodes, defining various types of configuration information available at each site, including interfaces (e.g., logical ports), L2 VPNs, BGP services, and IPSec VPNs. Even if the zoning service is a child node of other elements, it does not necessarily inherit the entire span of those elements. In some embodiments, the zoning service node has a single site span (i.e., the site of the edge cluster node referenced by the local service node), so all child nodes only inherit the span of the single site referenced by the local service node.

[0156] Logical switch 1745 is shown as a child node under router T1A 1710. If they are connected as child nodes (e.g.) Figure 17 As shown in the diagram, such logical switches (also called fragments) are limited to the parent router. However, in some embodiments, logical switches are also directly connected to the global root 1702. For example, overlapping fragment 1720 is directly connected to the global root 1702 and has a reference to router T1B 1715 (dashed line). This allows overlapping fragment 1720 to move to a different router if desired, by simply changing the reference to another logical router located at the top level below the global root 1702. Overlapping fragment 1720 inherits the span of router T1B 1715, such as site A, but if another locale service is added to router T1B 1715, the overlapping fragment can automatically extend.

[0157] In some embodiments, another type of segment is a VLAN-enabled segment. These are defined for transport zones, which are groups of host devices at a single physical site. Therefore, a VLAN-enabled segment can only span the single site that defines the transport zone. In some embodiments, the VLAN-enabled segment is used as an uplink to connect a logical router to an external physical router outside the logical network. In other words, the VLAN resides between the Layer 0 router and the external router. Since multiple Layer 0 routers can connect to the same external physical router, VLAN-enabled segments are used in some embodiments to differentiate their traffic. Typically, connecting a logical Layer 0 router to a physical router occurs at a single physical site because each site has its own WAN (e.g., the Internet) connecting to the sites, i.e., a single Internet Service Provider (ISP). Thus, VLAN-enabled segments provide a way to logically isolate traffic from different Layer 0 routers to the same external router, even if the Layer 0 routers can extend across multiple sites and overlap in their spans.

[0158] exist Figure 17In the example, VLAN fragment 1725 references interface 1755 of localization service 1750 under router T1B 1715 (dashed line). Interface 1755 is restricted to the span of localization service 1750, so by connecting VLAN fragment 1725 to interface 1755, the span of the VLAN fragment is restricted to site A 1760 only as needed. If another localization service is added under router T1B 1715, the span of router T1B will extend to include the new site, but the span of VLAN fragment 1725 will remain unchanged because it references interface 1755 of local service 1750.

[0159] In some embodiments, the interface is an uplink or service port. The interface connects to a logical switch or segment, and then logical network endpoints (such as virtual machines, data compute nodes, or other types of workloads) are attached to those logical switches and segments. These endpoints also have their own services, such as DNS, TCP, etc.

[0160] Furthermore, logical network elements include nodes for each physical site. For example, in Figure 17 In this example, under global root 1702, there are nodes for sites A 1760, B 1777, and C 1765. Each site has enforced point child nodes, under which specific resources are assigned, such as edge clusters, transport zones, etc. In the example, edge cluster 1751 at site A has incoming references from: localization service 1735 attached to router T0 1705 and localization service 1750 attached to router T1B 1715. Edge cluster 1752 at site B 1777 has localization service 1740 attached to router T0 1705. In some embodiments, the edge cluster also has a child corresponding to edge node 1753, which actually performs services such as firewalls, DHCP, etc.

[0161] In some embodiments, logical network elements also include logical constructs, such as domains as logical groups of one or more sites (e.g., geographic regions), and groups of logical network endpoints sharing one or more attributes (e.g., operating system, region, etc.). Domains are defined and represented as nodes in the global policy tree 1700 below the global root 1702. In some embodiments, domains are defined, for example, by the administrator of the logical network at the global manager 420. Unlike sites, which represent physical constructs, domains are logical constructs that serve as envelopes to group different logical entities together (e.g., for security purposes). For example, in some embodiments, firewall policies or other policy microsegments applied to a domain will be automatically applied to all logical endpoint groups defined within the domain.

[0162] In some embodiments, different types of domains exist. For example, some domains are specific to a single physical site and are referred to as locations. This type of domain acts as a container for both site-wide and site-specific configurations and policies. In some embodiments, a location domain is automatically created for each physical site in a federated logical network and cannot be modified by the user.

[0163] Other domains are logical groups of one or more sites and are referred to as regions. In some embodiments, regions are assigned to geographic areas. For example, in Figure 7 In the example, physical site A405 could be in Paris, physical site B410 in London, and physical site C415 in New York. These are respectively related to... Figure 17 In the example, site nodes A 1760, B 1777, and C 1765 correspond. A region (e.g., Europe) can then be defined, which includes physical sites A and B, and different regions (e.g., North America) can be defined, including physical site C. This is useful, for example, in situations where different regulatory environments exist (e.g., the EU's General Data Protection Regulation or GDPR). As with all domains, regions and locations are attached to the global root 1702 and are not attached as child nodes to other domains. Some embodiments restrict each physical site to membership in a single location and a single region. In other words, a location may not have more than one physical site, and a physical site may not be a member of two regions.

[0164] In some embodiments, a domain is created only as a top-level node under the global root 1202 and cannot be a child of another domain or inherit spans from another domain. Instead, in some embodiments, the span of a domain is manually defined at the global manager (e.g., by the administrator of the logical network) as the sites that are members of the domain. In some embodiments, the span is represented by a domain enforcement point configured to reference a site enforcement point intended to make the domain span any site. In some embodiments, these domain enforcement points are used only by the global manager and not by the local manager. For example, in Figure 17 In this context, the domain enforcement point for domain A 1770 references the site enforcement point for site A 1760 (e.g., Paris) and the site enforcement point for site B 1777 (e.g., London). Therefore, domain A 1770 is a domain spanning sites A and B, and... Figure 17 Other sites not shown (e.g., Berlin) represent regions (e.g., Europe). Furthermore, the domain enforcement point for domain B 1775 references the site enforcement point for site C1765 (e.g., New York). Therefore, domain B 1775 spans site C 1765. In this example, domain B1775 is a region (e.g., North America), which can also span... Figure 17Other physical sites not shown (e.g., Chicago, Los Angeles, etc.). Alternatively, domain B 1775 is a location specific only to site C 1765. For a given domain, in some embodiments, a set(s) of site implementation points referenced by the domain's mandatory implementation points is also referred to as a domain deployment map.

[0165] In some embodiments, logical network endpoints at each site are logically organized into security groups that can span multiple sites. In some embodiments, based on policy rules defined according to these groups, the service machine and managed forwarding elements executing on the host computer apply logical network policies (such as network policy 1773) to data messages exchanged between the security groups of endpoints. Such security groups and network policies are defined at global manager 420 by user client 440 (e.g., by the administrator of the logical network). In some embodiments, security groups and network policies are represented as child nodes of a domain in global policy tree 1700 and thus inherit the span of their parent domain. In some embodiments, the span of a network policy is defined not only by its parent domain but also by the site and / or domain referenced by the policy.

[0166] For example, in Figure 17 In this context, domain A 1770 has child nodes corresponding to security group A 1771, which accordingly inherit the spans of sites A 1760 and B 1777 (i.e., the spans defined by the domain deployment graph of domain A). Furthermore, domain B 1775 has child nodes corresponding to security group B1772, which accordingly inherit the span of site C 1765 (i.e., the spans defined by the domain deployment graph of domain B 1775).

[0167] Domain A 1770 also has child nodes corresponding to network policy 1773. A network policy applies to any group defined within the same domain (e.g., group A 1771). In order to apply a policy to a security group, in some embodiments the span of the security group must include the span of the policy.

[0168] In some embodiments, network policies may also refer to security groups that are not in the same domain. For example, network policy 1773 also references security group B 1772 in domain B 1775, even though the domain deployment graph of parent domain A 1770 does not include domain B 1775. This cross-domain policy is discussed in more detail below with reference to Figure 25.

[0169] In some embodiments, some nodes derive their spans from the span leader. Certain types of nodes are span leaders, such as T0 routers, T1 routers located below the global root, domains, or regional settings services. In some embodiments, a TI router defined below a T0 router is not a span leader. A domain is the span leader for all security policies defined under that domain. In some embodiments, groups are also span leaders and do not strictly inherit spans from their parent node (e.g., the domain), because a group's span can be extended beyond its parent domain by referencing the group, which will be described in more detail below with reference to Figure 25.

[0170] Figure 18 Five examples of security groups spanning one or more physical sites are conceptually illustrated. The diagram illustrates a group of host computers 1805 located at physical site A (e.g., Paris), a group of host computers 1810 located at physical site B (e.g., London), and a group of host computers 1815 located at physical site C (e.g., New York). Host computer 1805 at site A runs a group of virtual machines (VMA-VMD), each running either operating system OS1 (e.g., Microsoft Windows Server) or operating system OS2 (e.g., Linux). Similarly, host computer 1810 at site B runs VMVME-VMH, and host computer 1815 at site C runs VM VMI-VML.

[0171] As referenced above Figure 17 As noted in the discussion, in some embodiments, groups are also represented as child nodes of a domain. In such cases, the group inherits the span of its parent domain; that is, the group is defined at a site that is a member of its parent domain. If the group is defined at a site (i.e., if the group spans the site), then logical network endpoints located at that site can join the group.

[0172] As referenced above Figure 17 As described, sites A and B (e.g., Paris and London) are members of domain A 1770, which is a region (e.g., Europe). This logical grouping of the two physical sites is represented by a dashed line surrounding host computer 1805 and host computer 1810. Furthermore, site C (e.g., New York) is a member of domain B 1775, which is also a region (e.g., North America). This logical grouping is represented by a dashed line surrounding host computer 1815.

[0173] The first security group is group A 1817, which is defined for all VMs running operating system OS1 in Paris. This security group is logically defined at Global Manager 420 (e.g., by the administrator of the logical network) as having a span across domain A 1770 (i.e., sites A and B). Therefore, VMs A and D are automatically added to security group A 1817. The network administrator can apply security policies spanning domain A 1770 to security group A 1817, thus automatically applying them to VMs A and D.

[0174] The second security group is group B 1819, which is defined for all VMs running the OS2 operating system in New York. This security group is logically defined at Global Manager 420 (e.g., by the administrator of the logical network) as having a span across domain B 1775 (i.e., site C). Therefore, VMs J and K are automatically added to security group B 1819. The network administrator can apply security policies spanning site C (i.e., domain B 1775) to security group B 1819, thus automatically applying them to VMs J and K.

[0175] The third security group is group C 1820, which is defined for all VMs running the OS1 operating system in Europe. This security group is logically defined at Global Manager 420 (e.g., by the administrator of the logical network) as having a span of domain A 1770 (i.e., sites A and B). Therefore, VMs A, D, E, and G are automatically added to security group C 1820. The network administrator can apply security policies spanning site A, site B, or sites A and B to security group C 1820, thus automatically applying them to VMs A, D, E, and G.

[0176] The fourth security group is group D 1825, which is defined for all VMs running the OS2 operating system in Europe. This security group is logically defined at Global Manager 420 (e.g., by the administrator of the logical network) as having a span between sites A and B. Therefore, VMs B, C, F, and H are automatically added to security group D 1825. The network administrator can apply security policies spanning site A, site B, or sites A and B to security group D 1825, thus automatically applying them to VMs B, C, F, and H.

[0177] The fifth security group is group E 1830, which is defined as all VMs running operating system OS1 in all regions globally. This security group is logically defined at Global Manager 420 (e.g., by the administrator of the logical network) as having a span across sites A, B, and C. Therefore, VMs A, D, E, G, I, and L are automatically added to security group E 1830. Alternatively, security policies spanning any combination of sites A, B, C, or AC can be applied to security group E 1830 by the network administrator and will therefore be automatically applied to VMs A, D, E, G, I, and L.

[0178] As mentioned above, in some embodiments where network policies apply to security groups, the span of the security group must include the span of the network policy. For example, Figure 17 In the example, network policy 1773 will apply to security group E 1830 because the span of security group E 1830 includes the span of network policy 1773. The applicability of network policies to security groups is discussed in further detail below with reference to Figure 25. Furthermore, in some embodiments, VMs can be migrated between physical sites; for example, VM C is migrated from host machine 1805 (dashed arrow) to host machine 1810.

[0179] As discussed above, in some embodiments, the global manager 420 performs span calculations for each logical network element to determine which logical network elements are implemented at each physical site. This is performed to provide each physical site with the relevant desired global configuration (e.g., via an asynchronous replicator channel, as referenced above). Figure 13 As described in some embodiments).

[0180] Figure 19 A conceptual illustration depicts a process 1900 performed by a global manager 420 in some embodiments to parse the global desired configuration to populate the global policy tree 1700 and identify the relevant portions of the global desired configuration for each physical site. This process begins at 1905 by receiving the global desired configuration of the logical network from a user client 440 (e.g., from an administrator of the logical network). In some embodiments, the received configuration is a complete configuration, such as the initialization of the logical network. In other embodiments, the received configuration is a modification of a previously received global configuration, such as creation, update, or deletion events for one or more logical network elements.

[0181] At 1910, process 1900 populates the global policy tree based on the received global expectation configuration (e.g., Figure 17(Global policy tree 1700 in the global policy tree). For example, process 1900 identifies each logical network element referenced in the desired configuration and creates a node corresponding to the logical network element in the global policy tree. Furthermore, in some embodiments, each logical network element is associated with configuration parameters in the received desired configuration. These associated parameters are used to define the attributes of the created node. In some embodiments, the associated parameters in the desired configuration include references to other logical network elements. These references are used to define the references between the created node and other nodes. In some embodiments, nodes are defined hierarchically, starting from the global root 1702 and iterating through each branch of the policy tree from the top-level node (e.g., logical router, logical switch and fragment, domain, site, etc.). Some embodiments first create nodes for sites based on automatic discovery of physical sites or before or after receiving the global desired configuration.

[0182] At 1915, process 1900 selects the physical site implementing the logical network. In some embodiments, the physical site is identified through automatic discovery as discussed above. For the selected physical site, process 1900 parses the global policy tree at 1920 to identify policy nodes whose span includes the selected physical site. Process 1900 uses the identified nodes to generate (at 1925) the relevant desired configuration for the selected physical site. In some embodiments, this process generates the relevant configuration by identifying portions of the global desired configuration. Process 1900, as referenced above... Figure 13 In some embodiments, the asynchronous replicator channel described herein provides the generated relevant configuration (e.g., a portion of the globally desired configuration) to the local manager at the selected physical site.

[0183] At 1935, process 1900 determines whether an additional physical site exists. If an additional site exists, process 1900 returns to 1915, as described above. If no additional site exists, process 1900 terminates.

[0184] In some embodiments, the relevant portion of the globally desired configuration provided to each local manager is a site-specific policy subtree of the global policy tree, corresponding to the logical network element implemented at that site. The subtree for each site includes only nodes of the logical network element spanning the site. The global policy tree 1700 is stored by the global manager 420 in its database 710.

[0185] In some embodiments, the global policy tree is parsed to identify the span of each node in order to generate a policy subtree for each physical site. In some embodiments, a subtree is generated by first identifying the span of each node in the global policy tree. The global policy tree is then parsed using the span identified for each node to generate a policy subtree for each site. The local manager (or management plane application, which may be detached from the local manager) at each site uses the relevant portions of the global desired configuration received from the global manager, as well as any desired configurations received directly by the local manager itself, to manage the logical network at the site. Figure 20 A conceptual illustration is provided for a process 2000 performed by a global manager 420 to generate a policy subtree in some embodiments. Figure 21 The diagram illustrates an example of a policy subtree for a single site generated by process 2000.

[0186] Process 2000 begins at 2005 by receiving a global policy tree representing the desired configuration of a multi-site logical network (e.g., Figure 17 The global policy tree 1700 is used in the process 2000. At 2010, one of the nodes in the global tree is selected, which in some embodiments corresponds to one of the logical network elements. In some embodiments, the node is selected iteratively in the hierarchy of the global tree, starting from the top-level node under the global root 1702 (e.g., logical router, logical switch and fragment, domain, etc.).

[0187] Process 2000 determines (at 2015) the span attribute for the selected node. In some embodiments, the span attribute is determined by a span calculation performed by the agent service 735 of the global manager 420. In some embodiments, the span calculation is based on the relationships between nodes in the global policy tree. For example, when the relationship between two nodes is a parent-child relationship, the child node can inherit its span from the parent node. However, in other cases, the child node does not inherit the entire span of its parent node. As another example, when the relationship between two nodes is a dependency of one node on another, expressed as a reference from one node to another, the span of the dependent node will depend on the referenced node. In some embodiments, some logical network elements also have predefined spans, for example, defined by the network administrator and stored in the database 710. In such embodiments, process 2000 retrieves the span attribute from the database.

[0188] At 2020, process 2000 uses the span attribute to assign the selected node to a policy subtree of each physical site spanned by that node. These policy subtrees are stored in database 710 in some embodiments. In some embodiments, the policy subtrees are associated with dedicated persistent queues 725-727 corresponding to their respective physical sites.

[0189] Process 2000 determines (at 2025) whether an additional node exists in the global policy tree. If an additional node exists, the process returns to 2010 as described above. If no additional node exists, the process continues to 2030. At 2030, the process uses the generated policy subtree to generate the relevant desired configuration for each physical site. In some embodiments, the process generates the relevant configuration from the policy subtree by identifying portions of the global desired configuration. Process 2000 then terminates.

[0190] Figure 21 The diagram illustrates a global policy subtree 2100 for physical site A 405, stored in the database of the local manager (i.e., local manager 425) for site A. Since subtree 2100 is site A-specific, all top-level nodes with a span including site A under the global root 2102 are retained, while top-level nodes associated with sites B and C are omitted. For example, the top-level nodes for sites B 1777 and C 1765 are omitted, as are their corresponding child nodes. However, the top-level node for domain B 1775 remains because in some embodiments, all local managers are informed of all available domains. Furthermore, for router T0 1705, the localization service node for site B 1740 is also omitted. In some embodiments, router T0 1705 still spans sites A and B because its span attribute is associated with its definition at global manager 420 rather than at local manager 425.

[0191] Network policy 1773 is also stored in the global policy subtree 2100. This policy 1773 is defined under domain A 1770, and therefore in some embodiments, it has a span between site A and site B, even though this subtree is specific to site A. Furthermore, as referenced above... Figure 21 As stated, network policy 1773 also references group B 1772. Therefore, the scope of policy 1773 also includes site C, even if that site is located in a different domain. This reference to group B 1772 is also preserved in policy subtree 2100, and is discussed in more detail below with reference to Figure 25.

[0192] In some embodiments, the local manager also stores a separate policy tree that is generated based on the desired configuration received directly from the local manager rather than from the global manager 420. Figure 22A local policy tree 2200 for site A is conceptually illustrated, which in some embodiments is generated by one or more operations of process 1900. This local policy tree 2200 is different from the global policy subtree 2100 received from global manager 420. In this example, the local root 2202 of the local policy tree 2200 connects the site node for site A 0560, two Layer 1 logical routers TIC 2205 and T1D 2210, and an overlap segment 2215. The overlap segment 2215 includes a reference to router T1D 2210, which has a locale service corresponding to site A.

[0193] In some embodiments, logical network elements defined in the local policy tree 2200 can reference logical network elements defined in the global policy tree 1700. For example, in Figure 22 In this context, router TIC 2205 references router T0 1705 defined from global manager 420. Therefore, router TIC 2205 is able to utilize static route 1730 associated with router T0 1705, even though these routes are not defined locally at site A.

[0194] The global policy tree 1700 is stored in its database 710 by the primary global manager 420. A copy of the global policy tree 1700 is also stored in its database 712 by the secondary global manager 460. As described above, in some embodiments, nodes also represent logical network policies applied to logical network elements. Logical network policies include forwarding policies, service policies, and security policies, and in some embodiments are applied to govern the behavior of logical forwarding elements (e.g., by governing the behavior of the physical forwarding elements that implement the logical forwarding elements).

[0195] In some embodiments, the policy is defined at the global manager 420 via a user client 440, for example, by an administrator of the logical network. In some embodiments, the policy is one or more service rules enforced at a site based on a set of flow attributes on a data message flow. In some embodiments, the global manager 420 distributes the service rules to local managers 425-435 at sites 405-415 (i.e., the policy span) where the policy is to be enforced. Each local manager uses the received service rules to generate configuration data for distribution by controllers 1420-1425 to configure the data plane (i.e., physical forwarding elements 345 and edge devices 325) to enforce the received service rules on data message flows between logical network endpoints (such as virtual machines and other data compute nodes).

[0196] In some embodiments, the strategy is defined by referencing a group of logical network endpoints spanning one or more sites, for example, as referenced above. Figure 18Security groups 1817-1830 are described, and in some embodiments these security groups are defined at global manager 420 (e.g., by an administrator of a logical network). In some embodiments, service rules refer to these groups using group identifiers assigned at global manager 420 when the groups are defined. The definitions of these groups are distributed to sites 405-415 across which the policies are spanned, enabling controllers 1420-1425 at these sites to identify hosts for member VMs of each group and enforce policies on the identified hosts.

[0197] As discussed above, service rules for a given policy enforce on data messages associated with a set of logical network endpoints (e.g., VMs) spanning one or more sites. In some embodiments, service rules have multiple fields, including a source field, a destination field, and an action field. Some embodiments reference these groups in these fields using a group identifier assigned to the group (e.g., by a global manager when the group is defined). For example, when a group identifier for a group is referenced in the source field of a service rule, the service rule enforces on data messages originating from machines in that group. When a group identifier for a group is referenced in the destination field of a service rule, the service rule enforces on data messages targeting machines in that group. In addition to the group identifier, in some embodiments, service rules also enforce on data message streams having other attributes that match those specified in the rule. For example, a rule might specify a stream 5-tuple (source IP address and port number, destination IP address and port number, and transport protocol). Alternatively or in combination, a rule might specify other attributes of the stream or the referenced group identifier.

[0198] In some embodiments, controllers 1420-1425 provide service rules to the identified host that performs the PFE, and the PFE stores the service rules as a match-action table. Figure 23 A matching-action table 2300 of some embodiments is conceptually illustrated. In this example, fields for source group identifier, destination group identifier, and action are shown. For clarity, other attribute fields used for matching data message streams as discussed above in some embodiments are not shown.

[0199] Match-Action table 2300 has multiple flow entries 2305-2315, each specifying a different service rule. The number of entries shown is simplified for illustrative purposes only; in reality, there are hundreds or even thousands of such entries. These entries refer to those mentioned in the previous reference. Figure 18 The security group definitions are introduced. For example, the first entry 2305 indicates that matching data messages originating from group C (OS1 machines in Europe) and destined for group D (OS2 machines in Europe) should be discarded. In some embodiments, the basic principle used for such a rule can be different security guidelines between two different operating systems.

[0200] Item 2310 instructs that matching data messages originating from Group A (OS1 machines in Paris) and destined for Group E (OS1 machines globally) should be permitted. This rule instructs that OS1 machines at one site in the domain should be allowed to send data messages to machines at other sites in the domain.

[0201] Item 2315 instructs that matching data messages originating from Group B (OS1 machines in New York) and destined for Group A (OS1 machines in Paris) should be discarded. This rule instructs OS1 machines in one area not to send data messages to OS2 machines in another area. In this example, the rule specified by item 2315 is a possible rule for network policy 1773, such as... Figure 17 As shown, the rule is defined under domain A 1770, alongside group A 1771, but references group B 1772, which is defined under domain B 1775. In this example, the policy tree reference points from policy 1773 to group B 1772, even though group B is the source group of the policy rather than the destination group. This is because the policy tree reference is related to the policy span, regardless of whether the group is the source or destination of data message traffic within that span.

[0202] The PFE identifies matching flows by performing a matching classification operation on each data message to determine whether flow attributes and / or group identifiers match those specified in the service rules. In some embodiments, when a data message matches a service rule, it is provided to a service machine that performs the action specified by the rule on the matching data message. These actions include discarding the data message, allowing the data message, or performing other service operations on the data message. For example, a service operation may modify the header of the data message to redirect the message to a different destination. These service operations include load balancing, firewalls, Dynamic Host Configuration Protocol (DHCP), Network Address Translation (NAT), and other services.

[0203] In some embodiments, a policy is defined as a node in a policy tree and as a child node of a domain node. In such cases, the policy (and therefore any service rules of the policy) inherits the span of its parent domain; that is, the policy is enforced on sites that are members of its parent domain. Furthermore, in some embodiments, groups are also represented as child nodes of a domain, and groups inherit the span of their parent domain. For policies applicable to group members, the group's span must, in some embodiments, include the policy's span. This allows the controller to identify the identifiers referenced by service rules and to identify which groups members to enforce the service rules.

[0204] In some embodiments, service rules distributed to a first set of sites (e.g., a first domain) to apply policies to a first group of machines defined at those sites reference a second group of machines defined in a second, different set of sites (e.g., a second domain). In other words, the span of the referenced second group does not include the span of the policy. Because such service rules reference a group of machines that are not defined for one or more sites in the first set of sites (e.g., the first domain), not all machines in the first group are able to enforce the policy. Therefore, some embodiments identify which sites in the first set of sites (e.g., the first domain) do not have a definition of the referenced group and distribute the definition of the referenced group to those identified sites.

[0205] Figure 24 The process 2400, performed by the global manager 420 in some embodiments to extend the span of a group for applying a cross-domain policy, is conceptually illustrated. Figure 25 conceptually illustrates an example of extending a group span due to a cross-domain policy. The policy in this example is the same as above. Figure 17 The strategy described in 1773 is the same, and is aimed at the above. Figure 18 Define the security groups as described in the document.

[0206] Figure 25A A simplified version 2500 of the global tree 1700 is conceptually illustrated, defined before any network policies are defined, such as in the global manager 420, the local manager 425 at site A 1760 (e.g., Paris), and the local manager 435 at site C 1765 (New York). The Paris site is a member of the European region (domain A 1770), and the New York site is a member of the North American region (domain B 1775). (See above.) Figure 18 As mentioned above, the European domain also extends to site B (London).

[0207] Two groups are defined under these regions: group A 1817 (OS1 machine in Paris) under domain A 1770 (Europe) and group B 1819 (OS2 machine in New York) under domain B 1775 (North America). Domains A 1770 and B 1775, as well as groups A 1817 and B 1819, are defined at global manager 420. Group A 1817 is defined at local manager 425 in Paris, and group B 1819 is defined at local manager 435 in New York. However, both domains 1770-1775 are defined at two sites, because in some embodiments, domains are defined at all sites. For clarity, the corresponding site nodes under each domain are omitted from the tree in Figure 25.

[0208] Process 2400 begins at 2405 with receiving network policies defined in a domain (e.g., a first set of sites spanning that domain) for enforcement against members of security groups within that domain. In some embodiments, network policies are received at global manager 420 via user client 440 (e.g., from a network administrator). Figure 25B In this embodiment, security policy R1773 is now defined at global manager 420 as a policy under domain A 1770 (Europe) for enforcement against members of group A 1817. Policy 1773 is also distributed to local manager 425 in Paris. Although not shown, policy 1773 is also distributed to other local managers at other sites in domain A 1770 (Europe) (e.g., London). In the illustrated embodiment, policy 1773 includes service rule 2315, which is distributed to local managers in the domain.

[0209] At 2410, process 2400 identifies a reference in a network policy to another security group in another domain. For example, in Figure 25B In this context, policy R 1773 defines service rules for a source group (group B 1819) and a destination group (group A 1817). Since group B 1819 is not in the same domain as policy 1773, references to group B 1819 are indicated by a dotted line in the policy tree at global manager 420, from policy R 1773 to group B 1819. The action specified by policy R 1773 is a drop operation. In other words, policy R 1773 specifies that incoming data message traffic from a machine running OS2 in New York to a machine running OS1 in Paris is not allowed. This traffic will be dropped in Paris.

[0210] Procedure 2400 determines at 2415 whether the span of the other group includes the policy span. If the group span includes the policy span, then procedure 2400 ends. If the group span does not include the policy span, then... Figure 25B As shown in the example, the process then continues to 2420, which will be described below. For example, in Figure 25B In the example, the policy's span is the first domain A 1770, for example, Paris and London. The span of the referenced group B 1819 is the domain B 1775, for example, New York. Because the policy references a group that is not in its parent domain, the span of the referenced group does not include the policy's span.

[0211] At 2420, process 2400 expands the span of the referenced group, including the policy span. In effect, this distributes the group definitions from other domains to sites in the original domain where the policy is defined. In some embodiments, this is accomplished by performing a new span calculation for the policy based on a separate span calculation of the group referenced by the policy and the policy's parent domain. The span of the referenced group is then expanded by sending an update event via a dedicated AR channel from global manager 420 to local manager 425, as referenced above. Figure 13 As stated above.

[0212] As described above, in some embodiments, the span is also determined based on the span leaders of some nodes used in the global policy tree 1700. In some embodiments, a security group is a node type whose span leader is not its containing domain. For example, some groups act as span leaders for other groups. In the case of cross-domain policies, the span of the referenced group is extended by designating the span leader of the referenced group as the first group in the same domain as the policy. In this way, the span of the referenced group is wider than just its parent domain.

[0213] Upon receiving an update event, in some embodiments, the local manager 425 creates a reference group under the global policy tree in Paris. For example, in Figure 25B In this process, reference group 2550 is defined by the global manager 425 in the local manager 425 in Paris. In some embodiments, this reference group 2550 is not a complete group definition like group A 1817, but rather a symbolic path reference to group B 1819. As a result of this reference, for the local manager 425, group B 1819 now effectively spans New York, London, and Paris (i.e., spans domains A and B). Then process 2400 ends.

[0214] Some implementations use procedure 2400 to extend group definitions for all local managers of all sites (e.g., London) in Domain A (Europe). For example, if the group definition changes, i.e., a VM migrates from one site in the domain to another site in the domain, then the destination site also needs a referenced group in the global tree of its local manager.

[0215] In some embodiments, the policy tree at the local manager 435 in New York is not updated or modified in any way due to expanding the span of group 1819 at the global manager 420 and adding reference group 2550 to the local manager 425. In some embodiments, the administrator of the New York site is unaware that group 1819 at that site is referenced by policy 1773 at a different domain 1770, and may even decide to delete the group. However, the global manager 420 is aware of the reference due to the reference group 2550 created at the reference domain 1770. The global manager 420 then prevents the deletion of group 1819 and, in some embodiments, displays an alert to the administrators of both the reference and referenced sites, thereby resolving the conflict.

[0216] After receiving the reference group, the local managers in the original domain generate configuration data and distribute it to their respective site controllers to configure the physical forwarding elements and service machines at the first set of sites to enforce the policy.

[0217] In some embodiments, the configuration of the logical network can be described according to several different types of states. The desired state represents the user's intent; for example, the state of the network that a user (e.g., an administrator) expects to achieve through the global manager 420. The realized state is the actual implementation of the desired state, including the configuration of physical forwarding elements in the data plane. The realized state relates only to the logical network elements created by the user and includes configuration details automatically determined by the network management and control system on behalf of the user.

[0218] The third type of state is the discovered state, which represents configuration details of logical network elements not specified as part of the user's intent. For example, in some embodiments, a type of discovered state is site-specific information, such as the available physical resources at a physical site. In some embodiments, the desired state depends on, or is notified by, the discovered state. As the desired state is realized, in some embodiments, some portions of the discovered state are also modified, and these modifications are then rediscovered, thus notifying the next iteration of the desired state and the realized state. In some embodiments, the global manager 420 automatically discovers the discovered state based on information from physical sites, propagating it upwards from the control plane of those sites to their local managers, and from the local managers to the global manager.

[0219] For example, in some embodiments, the structure of the global policy tree 1700 under each site node is automatically discovered from the controller cluster 1420 at the physical site to the local manager 430 and then to the global manager 420. Once the global manager 420 is configured with the access credentials of the local manager 430, it communicates with the local manager (e.g., via a REST API in some embodiments) to automatically populate the enforcement point nodes and identify which transport zones, edge clusters, and edge nodes are configured and available. In some embodiments, these are previously configured by the local administrator of the physical site. Corresponding child nodes are created accordingly for each.

[0220] In some embodiments, the creation of logical ports and fragment ports from the global manager 420 is not permitted. In such embodiments, these ports are site-specific, have unique identifiers assigned by the management / control plane at their home physical site, and the span of these ports cannot be extended. Only one logical network endpoint can be attached to a port, and a logical network endpoint can only be part of one site at a given time. When the span of a switch / fragment extends to multiple sites, active logical ports (e.g., those with attached workloads) are automatically discovered as child nodes (e.g., as interface nodes) of that logical switch or fragment in that tree.

[0221] The desired configuration of a logical network represents the intent of a user (e.g., an administrator). Users specify their intent by specifying the desired configuration, which is why the desired configuration is also called a user intent. The global manager 420 is an intent-based policy manager that receives the user intent (internally represented as the global policy tree 1700) and communicates it to the local manager at each site. The local manager then interprets the received user intent to generate configuration data and provides the configuration data to the network manager and controller as described above to implement the desired configuration. In some embodiments, this is referred to as the implementation of the user intent (e.g., in the control plane and data plane). The actual implementation details (e.g., configuration parameters of forwarding elements, IP addresses, internal UUID identifiers of ports, etc.) are not known or specified by the user or the global manager when the intent is specified, but rather are the details necessary to implement that intent. These details include data from the management plane and control plane, such as the site manager and controller.

[0222] In the idealized scenario, the realized state of the logical network would be identical at each physical site. However, because each physical site is independently configurable (e.g., via local managers 425-435), and because certain configuration details are site-specific (e.g., unique identifiers for objects used in the management and control planes for system creation, such as logical port identifiers), in practice the realized state will differ for each physical site, even for the same logical network element. Even for a single site, the amount of implementation data is far greater than the amount of data required to specify user intent, multiplied by the number of sites in the federated logical network. Therefore, in some embodiments, the global manager does not automatically maintain information about the implementation state (also referred to as "realized state") of any given logical network element. Instead, the realized state can be queried from the global manager as needed for troubleshooting or other investigative purposes.

[0223] The realized state of a logical network element, in some embodiments, indicates whether the desired configuration (also referred to as the desired state) of the logical network element has been implemented at each site that the element traverses. Some embodiments describe the realized state in part based on deviations from the desired state. In some embodiments, the realized state for a logical network element (e.g., a logical router, logical switch, etc.) describes how the logical network element is implemented at each physical site it traverses. For example, the realized state includes data from the PFE (Physical Externality Provider) executed on the computing device at each of the traversed sites to implement the logical network element.

[0224] For each logical network element, in some embodiments the realized state received from each local manager is a subset of the total realized state of that logical network element, just as the expected state of the logical network element provided to the local manager is a subset of the global expected state of that logical network element defined at the global manager.

[0225] In some embodiments, the achieved status received from each local manager 425-435 is an achieved status message indicating whether the logical network element has been successfully achieved at the corresponding physical site. For example, if the logical network element has not been successfully achieved at all physical sites, the status message may simply be a flag indicating an error or pending achieved status.

[0226] Figure 26 A conceptual illustration is provided for a process 2600 executed by a global manager in some embodiments to retrieve the implemented state of logical network elements. Figure 27The diagram conceptually illustrates the distributed and collected operations of a global manager to retrieve the implemented state of logical network elements across multiple physical sites on demand. Figures 28-29 conceptually illustrate examples of user interfaces for presenting the implemented state to an administrator of the logical network, according to some embodiments. In some embodiments, the interface described in Figures 28-29 is a single interface presented on a user client 440, which can be selected between operating modes for accessing the global manager 420 or any of the local managers 425-435.

[0227] Process 2600 begins at 2605 with receiving a query for the implemented state of logical network elements (e.g., logical routers, logical switches, services such as BGP or VPN, security policies, etc.). In some embodiments, the global manager receives the query from user client 440 (e.g., via a wide area network 442 such as the Internet), and in some embodiments, it is initiated by the administrator of the logical network. In some embodiments, the query is received and processed as one or more REST API calls from user client 440 to API processor module 706.

[0228] In some embodiments, the query includes a reference to a logical network element (e.g., a path identifier that uniquely identifies the element). For example, a query for the implemented state of logical switch 1745 includes a path identifier, such as / global-infra / router_T1A / , which describes the location of the node corresponding to the logical switch in the global policy tree 1700.

[0229] In some embodiments, the query is for the implemented state of a logical network element at a single physical site, rather than across all sites that the element spans. In such embodiments, the query includes an identifier to specify the site. Process 2600 determines at 2607 whether the query includes a site identifier. If process 2600 determines that the query includes a site identifier, then the process provides the site identifier to the implementation state processor 2705 and proceeds to 2615, as described below.

[0230] If process 2600 determines at 2607 that the query does not include a site identifier, then process 2600 identifies the physical site spanned by the logical network element at 2610. In some embodiments, this is achieved by performing the above-referenced... Figure 12The span calculation is used to identify the sites spanned. Specifically, in some embodiments, API processor 706 provides parameters associated with the query to span service 736 to perform span calculation. For example, in some embodiments, the parameters include path identifiers for logical network elements. In some embodiments, the span is determined from the definition of the logical network element (e.g., the span is specified by the administrator of the logical network). After determining the span, span service 736 provides the site identifiers corresponding to the span of the logical network element to implementation state processor 2705.

[0231] At 2615, process 2600 sends a request for the implemented state of a logical network element to each local manager 425-435 at each identified site. In some embodiments, the request is sent from an implementation state processor 2705, which receives one or more site identifiers from an agent service 735 or an API processor 706. Some embodiments send the requests in parallel (e.g., “spread” the requests) to local managers 425-435.

[0232] In some embodiments, the request is made via the API of local managers 425-435. In such embodiments, the implementation state processor 2705 employs multiple implementation API clients 2710 to perform API calls, with one client assigned to each local manager. In some embodiments, the request is transmitted via the same AR channels 725-727 used to provide expected configuration data to each local manager.

[0233] At 2615, process 2600 receives the implemented state from each local manager in response to a request (e.g., "collecting" the implemented state). After receiving the implemented state from the local manager at each site traversed by each logical network element, process 2600 combines the implemented states received from each site at 2620. In some embodiments, the received implemented states are received by implementation API client 2710 and combined by implementation state processor 2705 in a single implementation state for the logical network element.

[0234] At 2625, process 2600 processes the realized state of a single combination and presents the realized state in response to the initial request. For example, in some embodiments, the realized state may be presented to the administrator of the logical network in a report on user client 440. Process 2600 then ends.

[0235] Figures 28A-28CA conceptual illustration shows an example of a user interface 2800 for querying the implementation status of logical network routers from a global manager 420. In some embodiments, interface 2800 is presented on a user client 440 accessed by an administrator of the logical network. User interface 2800 is, in some embodiments, a complete user interface for network management and control systems, and is used to define the desired state of the logical network (e.g., by issuing events for the creation, update, and deletion of logical network elements) and to query the implemented status of one or more logical network elements. The following discussion focuses on using interface 2800 to query the implemented status of logical routers (e.g., Layer 0 routers or Layer 1 routers). Although Figure 28A The examples in the text are for illustrative purposes and relate to Layer 0 routers, but the following discussion also applies to Layer 1 routers and fragments.

[0236] Figure 28A In some embodiments, the side panel 2805 is a menu that allows administrators to select different categories of logical network routers to query their implementation status. In this example, a Tier 0 gateway has been selected in the side panel, causing the main panel 2810 to display a list of all currently active Tier 0 routers for the logical network definition. A single active Tier 0 router is labeled Tier0Gateway1 and is therefore highlighted by default.

[0237] The main panel 2810 provides various configuration information sections for this router, including interface, routing, and Border Gateway Protocol (BGP) settings. In some embodiments, one or more of these sections cannot be configured to the desired state via the global manager, but are instead discovered based on the configuration at the physical site.

[0238] In some embodiments, the main panel 2810 also displays information related to the desired state of logical network elements, such as the span of the elements. In the example, the "Location" section 2815 shows a list of all locations currently spanned by the Tier0 Gateway1 router (e.g., London, Paris, and New York). Other available desired configuration information includes Domain Name System (DNS) information, Network Address Translation (NAT) settings, and firewall configuration. Administrators can modify some or all of these settings to define the desired state of the logical network.

[0239] In some embodiments, the implementation status of the Tier0 Gateway1 router is visually represented as a status indicator 2820. In different embodiments, the status indicator may display different text, symbols, colors, or some combination thereof. In some embodiments, this status indicator 2820 represents a global status, indicating the global implementation status at all stations across the span of the element.

[0240] In some embodiments, the global status indicator 2820 implements status on demand. In some such embodiments, selecting a logical network element from the interface is the action that initiates an implementation status request. In the example of FIG28, only a single Tier 0 router (Tier 0 Gateway 1) is active, so selecting the side panel item for the Tier 0 router also defaults to selecting this router and initiating a global implementation status request. If multiple Tier 0 routers exist, then in some embodiments additional selection is required to initiate the request, or alternatively, one of the routers is selected by default and a global implementation status request is issued. In other such embodiments, even after selecting a Tier 0 router, the global status indicator 2820 does not display any implementation status, and additional user interaction is required to initiate a request for global implementation status. For example, in such embodiments, the global status indicator 2820 displays text labeled “Check Status” or a user interface element (button, etc.). The global implementation status request is then initiated by interacting with the indicator (e.g., a reload button to the right of the text).

[0241] If the router has been successfully implemented at all physical sites, such as Figure 28B As shown, the global status indicator 2820 may display a colored icon (e.g., green) and / or the word "Success". If the router fails to achieve success at one or more physical sites across the span, the global status indicator 2820 will display a different color (e.g., yellow or red) and / or different text (such as "Pending" or "Error"). In some embodiments, if the global status is not "Success", one or more alarms may be present. In some embodiments, the number of alarms is also displayed next to the global status indicator, but in this example, since the router has achieved success, the number of alarms is zero.

[0242] In some embodiments, the global status indicator 2820 is also a user interface element. Interaction with the global indicator (i.e., clicking, mouse hovering, or touching (if the interface is on a touchscreen) the global status indicator) creates pop-ups in such embodiments, such as… Figure 28CAs in the example. Pop-up window 2830 displays the local implementation status of the logical network element at each individual physical site that the element traverses. In this example, in response to an implementation status query from the global manager, each local manager at each traversed site has reported "success" to the global manager (as indicated by the status indicator 2825 next to each listed site). If the router fails to implement successfully at one or more physical sites in the span, the local status indicator 2825 for that site will display a different color (e.g., yellow or red) and / or different text (such as "pending" or "error"). In some embodiments, the status can also be refreshed by selecting a refresh icon, either for a single physical site or for a global refresh, which initiates a new request for implementation status at one or more specified sites. If a refresh is selected for a specific site, then in some embodiments the request includes a site identifier, which is used as described in operation 2607 in reference procedure 2600 above.

[0243] Figures 29A-29B A conceptual illustration shows another example of a user interface 2900 for querying the implementation status of logical network segments from a global manager 420. In some embodiments, interface 2900 is presented on a user client 440 accessed by an administrator of the logical network. User interface 2900 is, in some embodiments, a full user interface for network management and control systems, and is used to define the desired state of the logical network (e.g., by issuing events for the creation, update, and deletion of logical network elements) and to query the implemented status of one or more logical network elements. The following discussion focuses on using interface 2900 to query the implemented status of logical switches (e.g., overlapping segments or VLAN segments). Although Figure 29A The examples in the text involve overlapping segments for illustrative purposes, but the discussion below also applies to VLAN segments.

[0244] Figure 29AThe side panel 2905 is, in some embodiments, a menu that allows administrators to select different categories of logical network routers to query the implementation status. In this example, a segment has been selected in the side panel, causing the main panel 2910 to display a list of all currently active logical switches for the logical network definition. Multiple logical switches are active, and the logical switch labeled LS-103 is highlighted (either by default or through user interaction). Each entry provides a brief summary 2915 of the segment, indicating which router the segment is connected to, the type of segment, the associated subnet (if overlapping), and the implementation status. In this example, the summary indicates that logical switch LS-103 is an overlapping segment associated with IP subnet 192.168.1.1 / 24, connected to router Tier0Gateway1 (described above with reference to Figure 28), and successfully implemented.

[0245] The main panel 2910 provides various configuration information sections for this segment, including a segment profile and DHCP static binding. In some embodiments, one or more of these sections cannot be configured to the desired state via the global manager, but are instead configured to a discovered state based on the configuration at the physical site. Other available information includes statistics and associated groups defined for the segment.

[0246] In some embodiments, the main panel also displays information related to the desired state of logical network elements, such as the span of the elements. In the example, the "Location" section 2920 shows a list of all locations currently spanned by the LS-103 switch (e.g., London, Paris, and New York). Other available desired configuration information 2925 includes VLAN-specific settings, such as domain name and address bindings, which do not apply to this overlapping network but are used to populate VLAN segments. Administrators can modify some or all of these settings to define the desired state of the logical network.

[0247] In some embodiments, the implementation status of the LS-103 switch is visually represented by a status indicator 2930. In different embodiments, the status indicator may display different text, symbols, colors, or some combination thereof. In some embodiments, this status indicator 2930 represents a global status, indicating the global implementation status at all sites across the component span. For example, if the switch has been successfully implemented at all physical sites, the global status indicator 2820 may display a colored icon (e.g., green) and / or the word "Success". If the switch has not been successfully implemented at one or more physical sites in the span, the global status indicator 2820 will display a different color (e.g., yellow or red) and / or different text (such as "Pending" or "Error"). In some embodiments, if the global status is not "Success", there may be one or more alarms. In some embodiments, the number of alarms is also displayed next to the global status indicator, but in this example, the number of alarms is zero because the switch has been successfully implemented.

[0248] In some embodiments, the global state indicator 2930 implements states on demand. In some embodiments, selecting a logical network element from the interface is the action of initiating a state implementation request. Figure 29A In the example, selecting the logical switch LS-103 from the list of segments initiates a global implementation status request. In some embodiments, the global status indicator does not display any implementation status even after segment selection, and additional user interaction is required to initiate a request for global implementation status. For example, in such embodiments, instead of displaying a global status indicator, text labeled "Check Status" or a user interface element (button, etc.) can be displayed in the same part of the interface. Interaction with this element then initiates a global implementation status request.

[0249] In some embodiments, the global status indicator 2930 is also a user interface element. Interaction with the global indicator (i.e., clicking, mouse hovering, or touching (if the interface is on a touchscreen) the global status indicator) creates pop-ups in such embodiments, such as… Figure 29B As in the example. Pop-up window 2932 displays the local implementation status of the logical network element at each individual physical site that the element traverses.

[0250] In this example, in response to an implementation status query from the global manager, each local manager at each spanned site has reported "success" to the global manager (as indicated by status indicator 2935 next to each listed site). If the switch is not successfully implemented at one or more physical sites in the span, the local status indicator 2825 for that site will display a different color (e.g., yellow or red) and / or different text (such as "pending" or "error"). In some embodiments, the status can also be refreshed by selecting a refresh icon, either for a single physical site or for a global refresh, which initiates a new request for implementation status at one or more specified sites. If a refresh is selected for a particular site, the request includes a site identifier in some embodiments, which is used as described in operation 2607 in reference procedure 2600 above.

[0251] In some embodiments, the Global Manager also receives notifications from the Local Manager. Unlike on-demand requests for implemented states (e.g., pulled from the Local Manager), these notifications are automatically pushed from the Local Manager. In some embodiments, the notifications relate to site-specific implementation issues or situations. In some embodiments, notifications are provided to the GM when the events they describe occur, e.g., in real-time or near real-time rather than on-demand. In some embodiments, these notifications are also displayed as reports, e.g., for review by administrators of the logical network.

[0252] In some embodiments, one type of notification is an alert from the local manager indicating that an implementation of a logical network element (e.g., a creation event) has failed at a specific site. For example, the failure could be due to an IP address conflict with another network element defined locally by the local manager, which the global manager is unaware of. Figure 30 The diagram illustrates an example of an alert caused by a failure to create a requested logical network element at the local manager.

[0253] exist Figure 30 In the first phase (top left), the global manager 420 has a global policy tree 1700, and the local manager 430 at site B410 has a site-specific version 2100 of the global policy tree. For clarity, only a portion of these trees is illustrated. Both trees have a Layer 0 router 3005 defined at the global manager 420 and successfully implemented by the local manager 430. The tree at the local manager also has a Layer 1 router (T1-B) defined at the local manager, which references the globally created T0 router 3005.

[0254] exist Figure 30In the second phase (top right corner), the global manager 420 receives a command 3015 for creating a Layer 1 router 3020 (T1-G). Router T1-G 3010 is created under Layer 0 router 3005 in response to the creation command 3015. The span of router T1-G 3020 includes site B 410, therefore the global manager 420 performs a span calculation (as described above) and instructs the local manager 430 at site B to create the same router. In some embodiments, this instruction is provided to the local manager 430 using the process 1300 described above.

[0255] Local Manager 430 receives the instruction and attempts to complete the creation command by creating router T1-G under the Layer 0 router. In some embodiments, the local manager attempts to create the router using process 1500 as described above. However, in this example, the IP address range of router T1-G 3020 overlaps at least partially with the IP address range of the locally created router T1-B 3010. Therefore, the local manager fails to create the requested router because the requested IP range is invalid.

[0256] exist Figure 30 In the third stage (bottom), the local manager 430 sends an alert 3030 to the global manager 420, notifying that the requested router T1-G 3020 failed to be created at site B 410. In some embodiments, the alert 3030 also includes relevant details for diagnosing the cause of the failure (e.g., the IP address range of router T1-G 3010). In some embodiments, the local manager 430 still creates the node for the requested router but does not attempt to implement the router in the control plane. The unimplemented state of router T1-G 3020 at site B 410 is determined by... Figure 30 The dashed outline is visually represented in the text.

[0257] In some embodiments, update commands from the global manager 420 of existing logical network elements can also trigger fault alarms. For example, local managers 425-435 in some embodiments have emergency security policies that can override any other configurations in certain anomalous circumstances (e.g., security breaches, physical emergencies, or data loss). In such embodiments, the global manager does not support or recognize emergency policies; therefore, any attempt to specify policy changes at the local manager that affect or contradict emergency policies will also be subject to the above reference. Figure 30 The alarm is triggered in the same way as described.

[0258] In some embodiments, another type of notification is an alert from the local manager indicating that the local administrator of the network has attempted to override the desired state of a logical network element at that particular site. Figure 31The diagram illustrates an example of an alert caused by an attempt to modify global configuration.

[0259] exist Figure 31 In the first phase (top left), global manager 420 has global policy tree 1700, and local manager 430 at site B410 has site-specific version 2100 of the global policy tree. Layer 0 router 3105 is defined under global policy root 1702, and the corresponding Layer 0 router 3110 is defined at local manager 430 under site-specific global policy root 2102. For clarity, both trees are... Figure 17 and Figure 21 A simplified version of the peer example.

[0260] exist Figure 31 In the second phase (top right corner), the local manager 430 receives command 3115 to update layer 0 router 3110. In some embodiments, this instruction is provided to the local manager 430 using process 1300 as described above. In some embodiments, this occurs when an administrator modifies the logical network using the local manager 430 instead of using the global manager 420. In some embodiments, the same logical network elements can be modified by different administrators who may be located in different geographical regions or even countries.

[0261] After receiving the update command, the local manager 430 in this example failed to fulfill the requested update to the Layer 0 router 3110. One example of why the update might fail is that some embodiments limit the types of configurations that can be overridden at the local manager 430. For example, in some embodiments, only networking-related configurations are allowed to be modified by the local manager 430. If the update is a security policy-related update, then the local manager is not allowed to overridden the global manager 420. Another example occurs when the update, prior to extending the security policy governing the router, is extending the router's span to a new site (i.e., extending the router's span). In some embodiments, the security policy must be extended before extending logical network elements to avoid security vulnerabilities after the span is extended.

[0262] exist Figure 31 In the third stage (bottom), the local manager 430 creates a shadow copy 3120 of the Layer 0 router. This shadow copy (denoted by an asterisk, Layer 0*) is initially identical to the actual Layer 0 router 3110, but since it was created by the local manager 430, it can be modified according to the received update command 3115. The original Layer 0 router 3110 from the global manager 420 is saved as a read-only node. The local manager 430 uses the shadow copy 3120 of the router in place of the original router 3110 for further network configuration at physical site B 410.

[0263] In some embodiments, the local manager 430 sends an alert 3125 to the global manager 420 to notify it of an attempt to overwrite the configuration of the layer 0 router 3110 and to replace the shadow copy 3120 in use. This is necessary because without this information, the global manager 420 would assume that the original copy of the layer 0 router 3110 is in use, and any user intent based on this assumption might fail or be inconsistent.

[0264] In embodiments where an attempt to modify the configuration of logical network elements is successful (e.g., because the update is networking-related rather than security-related), then in some embodiments the local manager sends an update notification to the global manager. Figure 31 (Not shown in the image). This is necessary to notify the global manager 420 that due to site-specific updates, the implemented state of logical network elements at this physical site will not match the implemented state of elements at other sites.

[0265] In some embodiments, the third type of notification is a message from the local manager regarding the failure of deleting a logical network element at that site. Figures 32A-32B The diagram illustrates an example of an alarm caused by the failure to remove a logical network element.

[0266] In the first stage of the example ( Figure 32A (Above), Global Manager 420 has a global policy tree 1700, and Local Managers 425 and 430 at sites A 405 and B 410 respectively have site-specific versions 2100 of the global policy tree. At Global Manager 420, Layer 0 router 3205 is defined under global policy root 1702, and at Local Managers 425 and 430, corresponding Layer 0 routers 3210 and 3215 are defined under their respective site-specific global policy roots 3220 and 3225. For clarity, all three trees are... Figure 17 and Figure 21 A simplified version of the peer example.

[0267] In this example, at site A 405, the local manager 425 also has a local policy tree with a locally defined local root 3230 (e.g., by a local administrator, which may be different from the global administrator who defines the global policy tree at the global manager 420). Below the local root 3230, a local Layer 1 router 3235 is also defined. This local Layer 1 router has a reference from the global manager to a globally defined Layer 0 router 3210 (depicted as a dashed arrow from Layer 1 router 3235 to Layer 0 router 3210). In some embodiments, the global manager 420 is completely unaware of the existence of the Layer 1 router 3235.

[0268] At site B 410, the local manager 430 also has a local policy tree, which has a locally defined local root 3240 (e.g., by a local administrator, which may be different from the global administrator who defines the global policy tree at the global manager 420). Below the local root 3240, a local layer 1 router 3245 is also defined. This local layer 1 router 3245 is a completely different router from the local layer 1 router 3235 defined for site A 405 under the local root 3230 at the local manager 425. The local layer 1 router 3245 has no references from the global manager to the globally defined layer 0 router 3210.

[0269] In the second stage of the example ( Figure 32A At the bottom of the layer 0 router (3205), the global manager 420 receives a command 3250 to delete the layer 0 router 3205. However, in some embodiments, in response to command 3250, the layer 0 router 3205 is not immediately deleted at the global manager. The global manager 420 performs a span calculation to identify sites A and B as spans of router 3205 and sends instructions to local managers 425 and 430 to delete the implemented layer 0 routers 3210 and 3215. In some embodiments, this instruction is provided to the local manager 430 using the process 1300 described above.

[0270] In the third stage of the example ( Figure 32B At the top of the list, local manager 425 at site A 405 received the instruction but failed to remove layer 0 router 3210 because it was being referenced by local layer 1 router 3235. Local manager 430 at site B 410 also received the instruction and successfully removed layer 0 router 3215 because there was no local reference to the router at this site.

[0271] In the fourth stage of the example ( Figure 32B At the bottom, Local Manager 425 sends Alert 3255 to Global Manager 420, notifying that the requested Layer 0 router could not be removed at site A 405. In some embodiments, Alert 3255 also includes detailed information for diagnosing the cause of the failure. Layer 0 router 3260 was not removed, but in some embodiments it was marked as pending removal by the Global Manager, as shown by the dashed outline. In some embodiments, additional notifications are generated for the local administrator of Local Manager 425 so that they can resolve the issue (e.g., by changing the reference to T-1 router 3235 to a different T0 router, such as the one created at Local Manager 425).

[0272] Local manager 430 at site B 410 sends notification 3265 to global manager to notify that layer 0 router 3215 has been successfully removed from local root 3225. Global manager 420 receives notification 3265, but in some embodiments, layer 0 router 3205 is not removed from global root 1702 because of the presence of alert 3255 instead of a corresponding success notification from local manager 425.

[0273] In some embodiments, an additional example of the notification type is a message from the local manager regarding the migration of a logical network endpoint (e.g., a virtual machine) from one site to another. Such endpoints are typically attached to logical ports of logical network elements (e.g., logical switches) spanning two sites. In some embodiments, when an endpoint is migrated to a different site, it maintains its association with the same logical port and seamlessly inherits the same security policies associated with that port, even at the new site.

[0274] In some embodiments, a machine can be migrated from one physical site to another while retaining the state of the network policies applied to that machine. For example, for a logical segment (logical switch) spanning at least two physical sites, in some embodiments, a machine attached to a logical port of the segment at one of the spanned sites is migrated to a new logical port of the same segment at another of the spanned sites. In some embodiments, migrating a machine includes creating a new machine on a new host computer (at the new site), transferring configuration data from the original machine to the new machine, and removing the original machine from its host computer. In some embodiments, the configuration data includes physical state data and logical network policies.

[0275] In some embodiments, machine state (e.g., runtime data) is transferred from a computing device executing the original machine at a source site (i.e., the source host) to a computing device executing the new machine at a destination site (i.e., the destination host). In some embodiments, the state includes a local port identifier assigned by a local manager (e.g., local manager 425) at the source physical site and a global port identifier assigned by a global manager 420. In some embodiments, the global port identifier is associated with a logical network policy defined by a user client 440 (e.g., by a network administrator) as part of the desired configuration of the logical network. Upon receiving the transferred state data, the destination host overwrites at least a portion of the state of the new machine with the received state of the old machine.

[0276] In some embodiments, the destination host extracts a global port identifier from the received state data and provides it to the local manager of the destination site (e.g., local manager 430). In some embodiments, the destination local manager 430 uses the global port identifier to retrieve logical network policies associated with a global port identifier from global manager 420 and applies these policies to new machines. In some embodiments, the local manager 430 generates configuration data from the policies and provides the configuration data to the control plane of the logical network at the destination site (e.g., a set of controllers 1420-1425), and the control plane then distributes the configuration data to the destination host.

[0277] Figure 33 The illustration shows examples of some embodiments for migrating virtual machines between sites within a region while preserving the machine's group membership and associated policies. In this example, virtual machine (VM) 3305 is a member of group D 1825 (e.g., a machine running OS2 in a European domain), as referenced above. Figure 18 The example illustrates a simplified version of the global policy tree 3310 at global manager 420, and corresponding (simplified) global policy trees 3315-3320 at the local managers 425-430 of the source and destination sites (e.g., Paris and London, respectively). The policy trees have a Layer 1 router 3325 spanning both sites, and a segment 3330 that references (dashed arrow) the Layer 1 router 3325 as a parent and thus inherits the same span. Therefore, router 3325 and segment 3330 are defined in the policy trees at both sites. Segment 3330 also has a logical port 3327, which is assigned the global port identifier LP-1 at global manager 420.

[0278] Figure 33 The initial phase (upper half) illustrates the policy tree prior to the migration. Here, VM C 3305 has been created and attached to logical port 3336 of segment 3330 at the first physical site A (e.g., Paris 405). VM C 3305 is a member of group D 1805 and runs on one of the host computers 1805 at site A. Port 3336 is assigned a local port identifier LP-A (e.g., by the local manager at site A), which in some embodiments is specific to the source site. A set of policies (not shown) is configured to apply to group D 1825, and thus to VM C 3305 at site A 405, since the domain 1770 under which group D is defined spans site A. The migration (dashed line) is then initiated to move VM C 3305 from the source server 1805 at site A (405) to the destination server 1810 at site B (410).

[0279] Figure 33 The second phase (lower half) illustrates the policy tree after the migration. Here, VM C 3305 has been successfully migrated to the second physical site B (e.g., London 410). VM C 3305 is now attached to logical port 3337 of segment 3330 located at the second physical site B (e.g., London 410). VM C 3305 remains a member of group D 1805 and now runs on one of the host computers 1810 at site B. Port 3337 is assigned a local port identifier LP-B (e.g., by the local manager at site B), which in some embodiments is specific to the destination site. The set of policies still applies to VM C 3305 at site B 410 because the domain 1770 under which group D is defined spans site B.

[0280] In this example, the global manager is unaware of the local port identifier LP-A or LP-B. In some embodiments, each of the local managers 425-430 maintains a mapping between global port identifiers and local port identifiers for logical ports at the corresponding site. This mapping allows the local manager to retrieve relevant policy information applicable to the machine from the global manager 420, thereby seamlessly applying these policies before and after migration.

[0281] exist Figure 33 In some examples, machines are moved between sites within the same region, retaining their group membership, and the policies associated with that group membership are applied automatically. Some embodiments also allow machines to be migrated from a site in one region to a site in another region. However, in some such embodiments, the same policies may not necessarily apply to the machines (e.g., if the policies applicable to the machines at the source site do not cross the second site). In such cases, in some embodiments, a new security policy will need to be applied manually (e.g., by the administrator of the logical network). Furthermore, if the fragment does not cross the destination site, then in some embodiments, the machine will lose connectivity with the rest of the logical network after the migration.

[0282] Figure 34A -C conceptually illustrates the information exchange during VM creation and migration between the global manager, local manager, and source and destination hosts in some embodiments. Refer to Figure 34, Figure 35 A conceptual illustration shows a process 3500 performed by a source host 3415 at the source site during the creation and migration of a virtual machine (VM). In an embodiment illustrated by this process, a VM is migrated from the source site by powering on the new VM at the destination site and transferring the VM state from the source host to the destination host.

[0283] Process 3500 begins at 3505 with receiving instructions from the compute manager 3405 to register a new VM for attachment to a logical segment 3330 spanning the source site. The registration command is in... Figure 34A This is illustrated as instruction 3422. In some embodiments, the compute manager 3405 directs the allocation of compute resources across one or more physical sites spanned by a logical network. In some embodiments, the administrator of the logical network determines (e.g., via user interface 440) where new VMs should be registered.

[0284] At 3510, process 3500 requests port attachment for the new VM from the local manager at the source site at 425. The port attachment request is in... Figure 34A This is represented as instruction 3424. In response to this request, the local manager 425 creates logical port 3336 for logical segment 3330 and returns a source-site-specific local port identifier (e.g., LP-A), such as... Figure 34A As shown in instruction 3426. In some embodiments, the local port identifier is a universally unique identifier (UUID).

[0285] At 3515, process 3500 powers on the requested VM, as indicated by event 3427. Process 3500 uses a local port identifier (e.g., LP-A) received from local manager 425 to attach the new VM to port 3336 of logical segment 3330. In some embodiments, process 3500 sends a success message to local manager 425 after attaching to port 3336. The success message is in... Figure 34A The instruction is shown as 3428.

[0286] Upon receiving a success message, in some embodiments, the local manager 425 will assign the local port identifier (e.g., ...) to the local port identifier. Figure 34A The global manager 420 is provided with the global port identifier (as shown in instruction 3432) (e.g., the discovered state as described above). The global manager updates fragment 3330 to include logical port 3327 and assigns a global port identifier (e.g., LP-1) to the logical port. In some embodiments, the global manager 420 uses the same local port identifier assigned by the source local manager 425. In other embodiments, the global manager generates a new port identifier, for example, a path relative to the global root 1702 indicating the location of the parent fragment in the hierarchical policy tree. The global manager 3425 then provides the global port identifier back to the local manager 425, as... Figure 34A As shown in instruction 3434.

[0287] At 3520, process 3500 receives the global port identifier from local manager 425, such as Figure 34AAs shown in instruction 3436. In some embodiments, local manager 425 provides a global identifier to the central control plane of the logical network at the source site (e.g., a set of controllers 1420-1425). The controller then distributes the global identifier to process 3500, in some embodiments, via an agent executed on the same host next to the VM (e.g., the source host 3415 executing process 3500).

[0288] In some embodiments, a set of tags is also specified at global manager 420 for attaching to newly attached VMs. In some embodiments, these tags are used to define group memberships for VMs. Security rules can also be defined dynamically using these tags. In some embodiments, these dynamic references to tags can include regular expressions. Thus, for example, a set of tags, `dev_paris`, can be defined to indicate that a VM is in development mode, and security rules can be defined for the Paris site by referencing the full tag, or security rules can be defined using only a portion of the tag (e.g., "dev_*") to apply to all VMs marked as being in development mode, regardless of location. In some embodiments, tags are applied directly to VMs and internally to logical ports by source host 3415.

[0289] In some embodiments, the global manager 420 receives configuration data (e.g., desired states, such as security policies) to apply to logical ports. In some embodiments, the desired states are received via a user interface 440 (e.g., from an administrator of the logical network). The desired states are then provided from the global manager 420 to the local manager 425, as referenced above. Figure 13 The local manager 425 receives the desired state as described above. Figure 34A The instruction is shown as 3438.

[0290] At 3525, process 3500 receives the desired configuration (e.g., security policies, some of which may reference labels) and applies it to logical port 3336, such as... Figure 34A As shown in instruction 3442. In other words, process 3500 implements the desired state. In some embodiments, local manager 425 generates configuration data from the received desired state and provides the configuration data to the central control plane of the logical network at the source site (e.g., a set of controllers 1420-1425). The controllers then distribute the configuration data to process 3500, in some embodiments via an agent executed on the same host next to the VM (e.g., the source host 3415 executing process 3500).

[0291] At point 3530, process 3500 receives instructions from compute manager 3405 to prepare the VM for migration. The migration command is... Figure 34AThis is shown as instruction 3444. In some embodiments, this instruction includes the address of the destination host 3410 to which the VM will be migrated. In some embodiments, the administrator of the logical network determines (e.g., via user interface 440) where the VM will be migrated.

[0292] At 3535, process 3500 saves implemented state data for the VM, as indicated by event 3445. In some embodiments, the saved state data includes a global port identifier, which is received as described above at 3520. Process 3500 sends the saved data to destination host 3410, as referenced below. Figure 36 The process described in 3600, and as Figure 34B As shown in instruction 3458.

[0293] At point 3540, process 3500 receives a command to deregister the VM from the compute manager 3405. The deregistration command is... Figure 34C This is shown as instruction 3446. In some embodiments, compute manager 3405 sends this instruction based on one or more notifications (not shown) that state data has been successfully saved (e.g., from local manager 425) and / or that the VM at the destination site has been successfully updated using the saved state data, as referenced below. Figure 36 The process described in 3600.

[0294] At 3545, process 3500 requests local manager 425 to detach logical port 3336, as follows: Figure 34C As indicated by instruction 3448. After successfully detaching the port, process 3500 powers down the VM, as indicated by event 3450. Then process 3500 terminates.

[0295] Figure 36 A conceptual illustration, referring to Figure 34, shows a corresponding process 3600 performed by the destination host 3410 at the destination site in some embodiments during the migration of a VM from the source host 3415. In the embodiment illustrated by this process, the VM is migrated from the source site by powering on the new VM at the destination site and transferring the VM state from the source host to the destination host.

[0296] Process 3600 begins at 3605 with receiving instructions from the compute manager 3405 to register a new VM for attachment to a logical segment 3330 spanning the source and destination sites. The registration command is in... Figure 34B This is illustrated as instruction 3452. As described above, in some embodiments, compute manager 3405 directs the allocation of compute resources across one or more physical sites spanned by the logical network. In some embodiments, the administrator of the logical network determines (e.g., via user interface 440) which site a VM should be migrated to.

[0297] At 3610, process 3600 requests port attachment for the new VM from the local manager 430 at the destination site. The port attachment request is... Figure 34B This is represented as instruction 3454. In response to this request, the local manager 430 creates a logical port 3337 for logical segment 3330 and returns a source-site-specific local port identifier (e.g., LP-B), such as... Figure 34B As shown in instruction 3456. In some embodiments, the local manager 430 also includes a flag for creating port 3337 to indicate that the port has a pending state.

[0298] At position 3615, procedure 3600 powers on the requested VM, as follows: Figure 34B As indicated by event 3457, process 3600 uses a local port identifier (e.g., LP-B) received from local manager 430 to attach the new VM to port 3337 of logical segment 3330.

[0299] At 3620, process 3600 receives saved state data from source host 3415. In some embodiments, the saved state data includes a global port identifier (e.g., LP-1), which is saved by the original VM on source host 3415, as referenced above. Figure 35 The process described in 3500. The reception of state data and global identifiers is... Figure 34B The instruction is shown as 3458.

[0300] In some embodiments, the transfer of state data from source host 3415 3458 is a lengthy process involving the replication of disks, storage, and other data. In some embodiments, this process can take several hours, during which time the source VM remains active on source host 3415. Packets received by the source host are also included in the state transfer, so it doesn't matter if no policies are applied to the destination host during the state transfer, as the source VM is processing traffic correctly. The destination VM does not receive any direct data traffic during the state transfer. Therefore, in some embodiments, a pending flag is used to ensure that the destination host does not send a success message to the local manager before the state transfer is complete. Otherwise, the local manager will report success to the global manager, and traffic will begin arriving at the destination VM ahead of schedule. In some embodiments, the pending state is tracked by both the destination host 3410 and the destination local manager 430.

[0301] At 3625, process 3600 overwrites the configuration of logic port 3337 with the received status data, such as... Figure 34BAs indicated by event 3459. Specifically, process 3600 restores the saved data from the original VM to the new VM. In some embodiments, the state data also includes the local port identifier (e.g., LP-A) assigned to the original VM from the source local manager 430, as referenced above. Figure 35 As described. In such embodiments, the process does not overwrite the local port identifier (e.g., LP-B) received from the local manager 430 with the local port identifier (e.g., LP-A) from the saved data. In some embodiments, the state data also includes metadata associated with the original VM, some of which is also overwritten onto the new VM. The state data also includes any tags assigned to the source VM.

[0302] In some embodiments, when the destination local manager 430 recognizes that a complete copy (of storage, disk, etc.) is complete, it performs a Fast Suspension and Resume (FSR) operation at the destination VM. In some embodiments, this is coordinated with the source local manager 425 (e.g., via compute manager 3405) to also perform the same FSR operation at the source VM. The FSR operation shuts down the activity state of the source VM and enables the activity state for the destination VM so that the destination VM can begin receiving traffic. In some embodiments at this stage, the pending state of the destination port is also removed.

[0303] At 3630, process 3600 sends a success message to local manager 430 after attaching the port. In some embodiments, process 3600 also reports the global port identifier (e.g., LP-1) and any associated tags to local manager 430. Providing a success message, global port identifier, and tags to local manager 430... Figure 34B The instruction is shown as 3462.

[0304] Upon receiving the global port identifier, in some embodiments, the local manager 430 updates fragment 3330 to associate the global port identifier (e.g., LP-1) with a local port identifier (e.g., LP-B) for port 3337. The local manager 430 then uses the global port identifier and the tag to request the desired state for port 3337 from the global manager 420, such as... Figure 34B As indicated by instruction 3464, the desired state is then transmitted to the global manager 420. This state is then provided from the global manager 420 to the local manager 430, as referenced above. Figure 13 As described. The expected status is received by the local manager. Figure 34B The instruction is shown as 3466.

[0305] At 3635, process 3600 receives the desired state and applies it to logic port 3337, such as... Figure 34BAs shown in instruction 3468. In other words, at 3635, process 3600 achieves the desired state. In some embodiments, local manager 430 generates configuration data from the received desired state and provides the configuration data to the central control plane of the logical network at the source site (e.g., the set of controllers 1420-1425). The controllers then distribute the configuration data to process 3600. Then process 3600 terminates.

[0306] Many of the aforementioned features and applications are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also known as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard disk drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted via wireless or wired connections.

[0307] In this specification, the term "software" means including firmware residing in read-only memory or an application stored in magnetic storage, which can be read into memory for processor processing. Furthermore, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that collectively implement the software inventions described herein is within the scope of this invention. In some embodiments, when a software program is installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that run and execute the operations of the software program.

[0308] Figure 37 An electronic system 3700, conceptually illustrated, is shown that implements some embodiments of the present invention. The electronic system 3700 may be a computer (e.g., a desktop computer, personal computer, tablet computer, server computer, mainframe, blade computer, etc.), a telephone, a PDA, or any other type of electronic device. This electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 3700 includes a bus 3705, one or more processing units 3710, system memory 3725, read-only memory 3730, permanent storage device 3735, input device 3740, and output device 3745.

[0309] Bus 3705 collectively represents all system, peripheral, and chipset buses that communicatively connect numerous internal devices of electronic system 3700. For example, bus 3705 communicatively connects processing unit(s) 3710 to read-only memory 3730, system memory 3725, and permanent storage device 3735.

[0310] From these various memory units, processing unit(s) 3710 retrieves instructions to be executed and data to be processed in order to perform the process of the present invention. In different embodiments, processing unit(s) may be a single processor or a multi-core processor.

[0311] Read-only memory (ROM) 3730 stores static data and instructions required by processing unit (one or more) 3710 and other modules of the electronic system. On the other hand, permanent storage device 3735 is a read-write memory device. This device is a non-volatile memory unit that can store instructions and data even when the electronic system 3700 is powered off. Some embodiments of the invention use mass storage devices (such as magnetic disks or optical disks and their corresponding disk drives) as permanent storage device 3735.

[0312] Other embodiments use removable storage devices (such as floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 3735, system memory 3725 is a read-write memory device. However, unlike storage device 3735, system memory is volatile read-write memory, such as random access memory. System memory stores some of the instructions and data required by the processor during runtime. In some embodiments, the processes of the present invention are stored in system memory 3725, permanent storage device 3735, and / or read-only memory 3730. From these various memory units, processing unit(s) 3710 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

[0313] Bus 3705 is also connected to input and output devices 3740 and 3745. Input devices enable users to communicate information and select commands to the electronic system. Input device 3740 includes an alphanumeric keypad and a pointing device (also known as a "cursor control device"). Output device 3745 displays images generated by the electronic system. Output devices include printers and display devices such as cathode ray tube (CRT) or liquid crystal displays (LCDs). Some embodiments include devices that serve as both input and output devices, such as touchscreens.

[0314] Finally, as Figure 37As shown, bus 3705 also couples electronic system 3700 to network 3765 via a network adapter (not shown). In this way, the computer can be part of a computer network (such as a local area network (“LAN”), wide area network (“WAN”), or intranet) or a network of networks (such as the Internet). Any or all components of electronic system 3700 can be used in conjunction with this invention.

[0315] Some embodiments include electronic components, such as microprocessors, storage devices, and memories, that store computer program instructions in machine-readable or computer-readable media (which may alternatively be referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only digital multifunction discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid-state hard disk drives, read-only and recordable... Disks, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store computer programs that can be executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as machine code generated by a compiler, and files that include high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0316] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, some embodiments are executed by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0317] As used in this specification, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the term "display" or "displaying" refers to something displayed on an electronic device. As used in this specification, the terms "computer-readable medium," "machine-readable medium," and "computer-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0318] This specification deals with computing and networking environments including virtual machines (VMs) throughout. However, a virtual machine is just one example of a data compute node (DCN) or data compute endpoint (also known as an addressable node). A DCN may include non-virtualized physical hosts, virtual machines, containers that run on top of a host operating system without a hypervisor or a separate operating system, and hypervisor kernel network interface modules.

[0319] In some embodiments, VMs operate on the host machine alongside their own guest operating system, using resources virtualized by virtualization software (e.g., hypervisors, virtual machine monitors, etc.). Tenants (i.e., the VM owners) can choose which applications run on top of the guest operating system. On the other hand, some containers are constructed to run on top of the host operating system without requiring a hypervisor or a separate guest operating system. In some embodiments, the host operating system uses namespaces to isolate containers from each other, thus providing operating system-level isolation for different groups of applications operating within different containers. This isolation is similar to the VM isolation provided in a hypervisor virtualization environment of virtualization system hardware and can therefore be viewed as a form of virtualization that isolates different groups of applications operating in different containers. Such containers are more lightweight than VMs.

[0320] In some embodiments, the hypervisor kernel network interface module is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive / transmit threads. An example of a hypervisor kernel network interface module is the vmknic module, which is used by VMware's ESXi. TM Part of the management process.

[0321] It should be understood that although the specification mentions VMs, the examples given can be of any type of DCN, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, in some embodiments, the example network may include a combination of different types of DCNs.

[0322] Although the invention has been described with reference to many specific details, those skilled in the art will recognize that the invention can be embodied in other specific forms without departing from its spirit. Furthermore, several figures (including...) Figure 10 , 12The processes are conceptually illustrated in (13, 15, 16, 19, 20, 24, 26, 35, and 36). The specific operations of these processes may not be performed in the exact order shown and described. Specific operations may not be performed in a continuous sequence of operations, and different specific operations may be performed in different embodiments. Furthermore, the process may be implemented using several sub-processes or as part of a larger macro-process. Therefore, those skilled in the art will understand that the invention is not limited to the foregoing illustrative details but is defined by the appended claims.

Claims

1. A method for defining a logical network, the logical network spanning multiple sites and managed by a local manager at each site, the method comprising: In the global manager that manages the logical network, A tree is created from the definition of a logical network spanning the multiple sites, the tree including nodes representing elements in the logical network and connections between nodes expressing relationships between elements, each node in at least a subset of the nodes having a span attribute that identifies the set of sites spanned by the logical network element corresponding to that node. Use the span attribute to create a parsed tree for each site; Based on the parsed tree for each corresponding site, identify a portion of the relevant logical network definition for that site; as well as Provide the local manager of each corresponding site with the portion of the logical network definition identified for that corresponding site.

2. The method of claim 1, wherein a specific local manager at a specific site uses a portion of the logical network definition identified for that specific site to generate a set of configuration data and provides it to a cluster of controllers managed by the specific local manager at that specific site.

3. The method of claim 2, wherein the cluster of controllers (i) receives configuration data from a specific local manager, (ii) identifies a set of computing devices at a specific site to implement logical network elements across the specific site, and (iii) distributes the configuration data to the identified set of computing devices.

4. The method of claim 3, wherein the configuration data received by the particular computing device is received by a local controller executing on the particular computing device, wherein the local controller uses the received configuration data to configure at least one forwarding element also executing on the particular computing device to implement at least one of the logical network elements spanning the particular site.

5. The method of claim 3, wherein for each identified computing device at a particular site, the cluster of controllers (i) determines a set of mappings between logical addresses of logical network endpoints executed on the identified computing device and physical addresses of tunnel endpoints associated with the identified computing device, (ii) distributes the set of mappings for the identified computing devices to other computing devices at that site, and (iii) distributes the set of mappings to other clusters of controllers at other sites.

6. The method of claim 1, wherein the nodes of the tree include logical network elements and logical network strategies.

7. The method of claim 6, wherein the logical network policy includes a forwarding policy, a security policy, and a service policy.

8. The method of claim 6, wherein the logical network element includes a logical forwarding element that forwards data in the logical network.

9. The method of claim 8, wherein each logical forwarding element is implemented by one or more physical forwarding elements, the physical forwarding elements being executed on a computing device at a site traversed by the logical forwarding element.

10. The method of claim 1, wherein a specific local manager at a specific site creates and stores a local tree based on a portion of the logical network definition received from the global manager.

11. The method of claim 10, wherein the identified portion of the logical network definition is a first logical network definition, wherein a specific local manager directly receives a second logical network definition that is not received from a global manager and updates the local tree based on the received second logical network definition.

12. The method of claim 11, wherein for a specific network element spanning a specific site, when a second logical network definition conflicts with a first logical network definition, a specific local manager uses a set of priority rules to resolve the conflict.

13. The method of claim 12, wherein the set of priority rules includes a rule specifying that, when a conflict involves a forwarding rule associated with a particular logical network element, the local tree is updated using a second logical network definition received directly at the local manager.

14. The method of claim 13, wherein a particular local manager stores a first logical network definition received from a global manager as a read-only definition that is not modified when the local tree is updated.

15. The method of claim 12, wherein the set of priority rules includes a rule specifying that when a conflict involves a security rule associated with a particular logical network element, the local tree is updated using a first logical network definition received from the global manager.

16. The method of claim 11, wherein the first logical network definition specifies the deletion of a particular logical network element, wherein the particular local manager deletes the node associated with the particular logical network element from the local tree when the particular local manager determines that other logical network elements in use at a particular site do not require the particular logical network element.

17. The method of claim 16, wherein when a particular local manager determines that other logical network elements in use at a particular site require the particular logical network element, the particular local manager automatically notifies the global manager that the particular logical network element has not yet been deleted.

18. The method of claim 1, further comprising determining the span attribute of a specific node by performing span calculation based on a logical network definition.

19. The method of claim 18, wherein span calculation includes identifying the parent node of a particular node and inheriting the span attribute of the particular node from the identified parent node.

20. The method of claim 1, wherein the logical network definition specifies a span attribute for a particular node, the span attribute comprising a set of site identifiers associated with each of the sites.

21. The method of claim 1, wherein providing a specific identified portion of the logical network definition for a particular local manager includes writing the specific identified portion of the logical network definition to a file and transmitting the file to the particular local manager over a network.

22. A machine-readable medium storing a program that, when implemented by at least one processing unit, implements the method according to any one of claims 1-21.

23. A computing device, comprising: A set of processing units; as well as A machine-readable medium storing a program that, when implemented by at least one of the processing units, implements the method according to any one of claims 1-21.

24. A system comprising components for implementing the method according to any one of claims 1-21.

25. A computer program product storing instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1-21.

Images