Logical processing of containers

Abstract

The present disclosure relates to logical processing of containers. Some embodiments provide a method for a first managed forwarding element (MFE). The method receives a data message, the data message including a logical context tag identifying a logical port of a specific logical forwarding element. Based on the logical context tag, the method adds a local tag to the data message. The local tag is associated with a specific logical forwarding element, which is one of several logical forwarding elements to which one or more containers running on a container virtual machine (VM) belong. The container VM is connected to a first MFE. The method delivers the data message to the container VM without any logical context. A second MFE running on the container VM forwards the data message to a correct container among several containers running on the container VM using the local tag.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 16, 2016, application number 201680027855.0, and name “Logical Processing of Containers”. Background Art

[0002] More and more applications are being deployed into virtual machines (VMs), many of which consume network and security services (e.g., firewalls, access control lists (ACLs), quality of service (QoS), etc.). In a virtualized network, the virtualization system may further virtualize other systems, thereby increasing the complexity and depth of the system creating a layer of virtual interfaces behind the virtual interfaces. For example, a Linux container running on a VM may create several interfaces that share a single interface of the VM (also referred to as a container VM). The container interface that shares a single interface of the container VM may make it difficult to provide a generally applicable network virtualization platform that provides network and security services for non-container VMs as well as for containers executing within container VMs in the system. Summary of the invention

[0003] Some embodiments of the present invention provide a network control system for defining a logical network connecting virtual machines (VMs) running on host machines in a network and containers (e.g., Linux containers, VMs, etc.) running within another VM (i.e., container VM) running on one of the host machines of the network. The network control system of some embodiments defines a logical data path of a logical network that logically connects the VMs on the host machines and the containers running on the container VMs. In some embodiments, the network control system includes a set of controllers that manage forwarding elements on the host machines and within the container VMs to implement the logical network.

[0004] In some embodiments, each host machine in the network includes virtualization software (e.g., a hypervisor) for virtualizing the physical resources of the host machine and a host managed forwarding element (MFE) for forwarding network traffic (e.g., data messages) to and from the virtual machines. In some embodiments, the host MFE runs within the virtualization software. In addition, some host machines include one or more VMs connected to the host MFE, some of which may be container VMs hosting a collection of containers. In some embodiments, a local MFE runs within each container VM to forward data messages to and from containers hosted within the container VM.

[0005] The network control system of some embodiments includes a controller set for managing host MFE and local MFE. In some embodiments, the controller set configures the host MFE and the local MFE to logically forward data messages of containers and VMs according to the logical forwarding elements (e.g., logical switches, logical routers) configured by the administrator of the network. The controller set of some embodiments includes a local VM controller (LVC) set for managing the local MFE of the container VM, a local host controller (LHC) set for managing the host MFE of the host machine, and a centralized network controller (CNC) set for managing the LHC and / or LVC to implement the logical forwarding element (LFE) of the logical network.

[0006] Different controllers can be distributed across different machines in different ways, running on the same machine as the elements they manage or on separate machines. For example, the LHC of some embodiments runs on a host machine with a host MFE (e.g., within virtualization software), while in other embodiments, the LHC runs on a machine separate from the host machine and communicates with the host MFE over a network. In some embodiments, the network control system configures the MFEs (both the host MFE and the local MFE) to attach containers of container VMs as well as non-container VMs to one or more LFEs.

[0007] To attach a container to a specific LFE, the LHC of some embodiments receives container information about a container of the container VM from the LVC running on the container VM. In some embodiments, the container information includes address information (e.g., MAC address, IP address) and application state data of the application running in the container. The container information of some embodiments identifies a mapping of a local tag value (e.g., VLAN ID) of the container to logical data (e.g., logical port, LFE, etc.).

[0008] In some embodiments, the mapping provides a different local label value to each LFE implemented by the MFE of the container VM (i.e., each LFE to which one of the containers running in the container VM is connected). In other embodiments, rather than assigning a local label value to each LFE, the local controller assigns a different local label value to each container on the container VM, regardless of the LFE to which the container is associated.

[0009] Based on the received container information, the LHC of some embodiments maps each container to a logical port of a specific LFE. The LHC uses the mapping of local tag values ​​to the logical ports of the LFE to configure the host MFE to process network traffic to and from the container and apply logical policies to network traffic sent to and from the container, thereby removing the responsibility of applying such policies from the local MFE.

[0010] For data messages sent to the container, the host MFE marks the network traffic with a local tag value based on the logical port associated with the network traffic. For example, when the host MFE receives a data message destined for a specific logical port, the host MFE determines that the specific logical port is associated with a specific local tag value based on the mapping data from the LHC, and marks the data message with the specific local tag value before sending the data message to the container VM. The local MFE on the container VM is configured to forward the data message to the correct destination container based on the local tag value.

[0011] For data messages received from the container, the local MFE of some embodiments (i.e., the MFE running on the container VM) is configured to tag the data message with a local tag value and forward the data message to the host MFE. The host MFE receives the data message and identifies the logical port associated with the data message based on the local tag value and / or the unique address (e.g., the source MAC address). The host MFE then applies a set of logical network policies (e.g., policies received from the LHC via the CNC) to the data message before forwarding the data message through the logical network. The logical network policies of some embodiments (e.g., firewall policies, quality of service (QoS) policies, load balancing, etc.) are defined at multiple levels of the logical network (e.g., logical switch ports, logical switches, logical router ports, etc.).

[0012] In some embodiments, rather than applying all logical network policies at the host MFE, the network control system distributes some logical processing between the host MFE and the local MFE. For example, the local MFE of some embodiments is configured to apply logical network policies specifically for local traffic between containers on a container VM.

[0013] In some embodiments, the LHC also instantiates a new container to be added to the logical network. The LHC of some embodiments determines whether a suitable container VM is available on the host machine, and if not, creates a new container VM and initializes the local MFE for the container VM. In some embodiments, the LHC configures the local MFE to forward all traffic from the container to the host MFE.

[0014] The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. This is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and the drawings mentioned in the detailed description will further describe the embodiments described in the summary of the invention as well as other embodiments. Therefore, in order to understand all embodiments described by this document, a comprehensive review of the summary of the invention, the detailed description, the drawings and the claims is required. Moreover, the claimed subject matter is not limited by the illustrative details in the summary of the invention, the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following figures.

[0016] Figure 1 An example of a logical network with containers implemented on a physical network is illustrated.

[0017] Figure 2 An example of a controller in a logical network with containers implemented on a physical network is illustrated.

[0018] Figure 3 Conceptually illustrates the process for adding a container to a logical network at the container VM level.

[0019] Figure 4 An example of adding a container to a logical forwarding element of a logical network is illustrated.

[0020] Figure 5 Conceptually illustrates the process for adding containers to a logical network at the host machine level.

[0021] Figure 6 An example of adding a container to a logical forwarding element of a logical network is illustrated.

[0022] Figure 7 The figure shows an example of adding containers for different logical forwarding elements.

[0023] Figure 8 An example of mapping between logical networks and physical networks is illustrated.

[0024] Fig. 9 A process for processing data messages from a container over a logical network is conceptually illustrated.

[0025] Fig.10 An example of forwarding a data message from a container through a logical network is illustrated.

[0026] Fig.11 Another example of forwarding a data message from a container through a logical network is illustrated.

[0027] Fig.12 A process for handling data messages destined for a container over a logical network is conceptually illustrated.

[0028] Fig.13 An example of receiving a data message for a container over a logical network is illustrated.

[0029] Fig.14 A computer system is conceptually illustrated on which some embodiments of the invention are implemented. DETAILED DESCRIPTION

[0030] In the following detailed description of the present invention, many details, examples and embodiments of the present invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the present invention is not limited to the embodiments set forth, and the present invention can be practiced without some of the specific details and examples discussed.

[0031] When containers run on a guest system inside a container VM, the network virtualization system must treat the container VM the same as a physical container host. All network forwarding and services must be performed by the guest system inside the container VM. The network virtualization system installs and controls data plane components, control plane components, and management plane components in each container VM in order to implement virtualized network services, perform L2 / L3 traffic forwarding, apply configured policies to container interfaces, and even set up tunnels (such as VXLAN tunnels) for virtualized networks.

[0032] In many cases, configuring a client system to perform network forwarding and services is inefficient and complicates the implementation of a unified networking solution for all applications in a virtual environment, because the networking solution needs to replicate and manage the same functions in both the client system and the hypervisor. Managing services and components in both the VM and the hypervisor unnecessarily complicates the network virtualization system, and in some cases requires special support for forwarding elements in the container VM (e.g., support for VXLAN tunnels terminated in the container VM). When a large number of container VMs are deployed on a single hypervisor, the same functions may be replicated in each of these VMs (e.g., setting up the same VXLAN tunnel), and the network virtualization system cannot take advantage of the optimization of the hypervisor in the case of hardware offload, resulting in unnecessary consumption of more physical (computing, memory, network bandwidth, etc.) resources.

[0033] In order to provide a generally applicable network virtualization platform for a virtual machine (VM) and a container (e.g., Linux container, VM, etc.) executed within another VM, some embodiments of the present invention provide a network control system for defining a logical network connecting a virtual machine (VM) running on a host machine in a network and a container (e.g., Linux container, VM, etc.) running within another VM (i.e., container VM) running on one of the host machines of the network. The network control system of some embodiments defines a logical data path of a logical network that logically connects the VM on the host machine and the container running on the container VM. In some embodiments, the network control system includes a controller set that manages forwarding elements on the host machine and within the container VM to implement a logical network.

[0034] In some embodiments, each host machine in the network includes virtualization software (e.g., a hypervisor) for virtualizing the physical resources of the host machine and a host managed forwarding element (MFE) for forwarding network traffic (e.g., data messages) to and from the virtual machines. In some embodiments, the host MFE runs within the virtualization software. In addition, some host machines include one or more VMs connected to the host MFE, some of which may be container VMs hosting a collection of containers. In some embodiments, a local MFE runs within each container VM to forward data messages to and from containers hosted within the container VM.

[0035] The network control system of some embodiments includes a controller set for managing host MFE and local MFE. In some embodiments, the controller set configures the host MFE and the local MFE to logically forward data messages of containers and VMs according to the logical forwarding elements (e.g., logical switches, logical routers) configured by the administrator of the network. The controller set of some embodiments includes a local VM controller (LVC) set for managing the local MFE of the container VM, a local host controller (LHC) set for managing the host MFE of the host machine, and a centralized network controller (CNC) set for managing the LHC and / or LVC to implement the logical forwarding element (LFE) of the logical network.

[0036] Different controllers can be distributed across different machines in different ways, running on the same machine as the elements they manage or on separate machines. For example, the LHC of some embodiments runs on a host machine with a host MFE (e.g., within virtualization software), while in other embodiments, the LHC runs on a machine separate from the host machine and communicates with the host MFE over a network. In some embodiments, the network control system configures the MFEs (both the host MFE and the local MFE) to attach containers of container VMs as well as non-container VMs to one or more LFEs.

[0037] To attach a container to a specific LFE, the LHC of some embodiments receives container information about a container of the container VM from the LVC running on the container VM. In some embodiments, the container information includes address information (e.g., MAC address, IP address) and application state data of the application running in the container. The container information of some embodiments identifies a mapping of a local tag value (e.g., VLAN ID) of the container to logical data (e.g., logical port, LFE, etc.).

[0038] In some embodiments, the mapping provides a different local label value to each LFE implemented by the MFE of the container VM (i.e., each LFE to which one of the containers running in the container VM is connected). In other embodiments, rather than assigning a local label value to each LFE, the local controller assigns a different local label value to each container on the container VM, regardless of the LFE to which the container is associated.

[0039] Based on the received container information, the LHC of some embodiments maps each container to a logical port of a specific LFE. The LHC uses the mapping of local tag values ​​to the logical ports of the LFE to configure the host MFE to process network traffic to and from the container and apply logical policies to network traffic sent to and from the container, thereby removing the responsibility of applying such policies from the local MFE.

[0040] For data messages sent to the container, the host MFE marks the network traffic with a local tag value based on the logical port associated with the network traffic. For example, when the host MFE receives a data message destined for a specific logical port, the host MFE determines that the specific logical port is associated with a specific local tag value based on the mapping data from the LHC, and marks the data message with the specific local tag value before sending the data message to the container VM. The local MFE on the container VM is configured to forward the data message to the correct destination container based on the local tag value.

[0041] For data messages received from the container, the local MFE of some embodiments (i.e., the MFE running on the container VM) is configured to tag the data message with a local tag value and forward the data message to the host MFE. The host MFE receives the data message and identifies the logical port associated with the data message based on the local tag value and / or the unique address (e.g., the source MAC address). The host MFE then applies a set of logical network policies (e.g., policies received from the LHC via the CNC) to the data message before forwarding the data message through the logical network. The logical network policies of some embodiments (e.g., firewall policies, quality of service (QoS) policies, load balancing, etc.) are defined at multiple levels of the logical network (e.g., logical switch ports, logical switches, logical router ports, etc.). It should be understood that the term "data message" as used herein may refer to a collection of various formatted bits that can be sent across a network, such as Ethernet frames, IP packets, TCP segments, UDP datagrams, etc.

[0042] In some embodiments, rather than applying all logical network policies at the host MFE, the network control system distributes some logical processing between the host MFE and the local MFE. For example, the local MFE of some embodiments is configured to apply logical network policies specifically for local traffic between containers on a container VM.

[0043] In some embodiments, the LHC also instantiates a new container to be added to the logical network. The LHC of some embodiments determines whether a suitable container VM is available on the host machine, and if not, creates a new container VM and initializes the local MFE for the container VM. In some embodiments, the LHC configures the local MFE to forward all traffic from the container to the host MFE.

[0044] The above description introduces a system for adding containers to a logical network. Several more detailed embodiments are described below. Section I describes an example of a network control system for implementing containers and adding containers to a logical network. Section II describes an example of adding containers to a logical network. Section III describes an example of forwarding network traffic for containers in a network. Finally, Section IV describes an electronic system that implements some embodiments of the present invention.

[0045] I. Network Control System

[0046] Figure 1 An example of a logical forwarding element including both VMs and containers running as end machines and a physical implementation of the logical network is illustrated. In particular, the figure conceptually illustrates two logical forwarding elements 105 and 110 with an example of a physical implementation 102 of the logical forwarding element. These logical forwarding elements can be part of the same logical network (e.g., two logical switches connected by a logical router not shown in the figure) or completely different logical networks (e.g., logical networks owned by two different tenants),

[0047] The logical configuration 100 shows a logical forwarding element (LFE) A 105 coupled to virtual machines VM1, VM2, and VM7 and to containers C1-C3. The logical configuration 100 also shows an LFE B 110 coupled to virtual machines VM3, VM4, and VM8 and containers C4-C5. LFEs A and B of some embodiments belong to different tenants of a data center that houses the physical network 102. In other embodiments, LFEs A and B may be logically separate forwarding elements for a single tenant. Each LFE allows different VMs and containers to run as if they were all attached to a single forwarding element, regardless of the actual connections and topology of the underlying physical network 102.

[0048] The physical implementation 102 includes three host machines 120, 123, and 126 that provide various containers and VMs to implement LFEs 105 and 110. The host machines 120, 123, and 126 include virtualization software (not shown) and host managed forwarding elements (MFEs) 140, 143, and 146, respectively, which in some embodiments run within the virtualization software of the respective hosts. Each of the host MFEs is connected to the MFEs of the other host machines (e.g., via tunnels through the physical network infrastructure) and to the VMs hosted on the respective host machines.

[0049] The host MFEs 140, 143, and 146 of some embodiments may include several different types of managed forwarding elements (e.g., flow-based software forwarding elements such as Open vSwitch (OVS)), feature-based software forwarding elements such as VMWare TM ESX Server) and the like). In addition, some embodiments include a hardware managed forwarding element that runs outside the host machine but performs the operations of the host MFE (in a flow-based or feature-based manner). The flow entries of some embodiments are stored in the forwarding table of the MFE to define rules for forwarding data messages (e.g., Ethernet frames, IP packets, TCP segments, UDP datagrams, etc.) through the MFE. The flow entry includes a set of conditions to be matched by the data message header and a set of actions (e.g., discard, forward, modify, etc.) to be performed on the data message that matches the set of conditions. The host MFE of some embodiments may also be connected to a gateway (not shown) and other network elements for connecting the logical network to other external physical networks (e.g., the Internet, an intranet, etc.).

[0050] Host machines 120, 123, and 126 also include several VMs running on top of their respective host machine's virtualization software. VM1 and VM5 run on host machine 120, VM2 and VM6 run on host machine 123, and VM7 and VM8 run on host machine 126. As shown, some VMs are directly connected to both physical and logical forwarding elements (e.g., VM1, VM2, VM7, and VM8, which are end machines of the logical forwarding element), while other VMs (e.g., VM5 and VM6) are attached to the host MFE of physical network 102 but are not directly represented in logical forwarding elements 105 and 110.

[0051] VM5 and VM6 are examples of container VMs that virtualize a collection of containers (e.g., virtual machines, applications, etc.). For example, VM5 hosts containers C1-C5, which provide environments that run on the kernel of VM5 but are otherwise isolated from each other. VM6 hosts VM3 and VM4, which provide environments that can each run on their own operating systems virtualized within VM6. In this application, these virtualized environments will be referred to as containers, but it should be clear to those skilled in the art that containers can refer to any of many different types of virtualized environments. The host machines of different embodiments may host container VMs, non-container VMs (i.e., VMs that do not virtualize containers and typically run as terminal machines of a logical network), or some combination of the two.

[0052] In some embodiments, the container is configured in the logical forwarding element without referencing its hosting container VM. That is, the interface of the container VM has no counterpart in the logical network, but is instead part of the physical implementation of the container VM. Instead, the container is mapped to the physical (e.g., virtual) interface of the physical container to its local MFE via the configured logical interface of its logical forwarding element connected to it. Each VM hosted directly on its host's virtualization software (e.g., VM1, container VM6) has a primary interface to its host MFE (e.g., VM6's connection to host MFE 143), while containers hosted within a container VM (e.g., VM3, C1) have a secondary interface to the local MFE within its container VM. In the case of containers VM5 and VM6, their primary interfaces are also connected to ports of local MFEs 150 and 155, respectively.

[0053] The local MFEs 150 and 155 of the containers VM5 and VM6 work with the host MFE to perform logical processing of network traffic sent to and from the containers. In different embodiments, the local MFE may be a bridge, a software virtual switch (e.g., Open vSwitch (OVS)), or other simple forwarding elements that can mark and forward network data. Some embodiments distribute logical processing between the host MFE and the local MFE, allowing the local MFE to handle some processing and forwarding of data messages. Due to potential problems with isolation and security in containers, in some cases, the local MFE on the container VM cannot be fully trusted. In particular, it may be difficult to protect the security of containers running on the container VM, so the local MFE is likely to be damaged. However, since the security of the VM has been established and tested for a long time, it is unlikely that a damaged container in the container VM will affect other VMs on the host machine. In order to improve security in the network control system, some embodiments of the network control system treat each different VM or host as a separate security domain, so that only containers belonging to a single tenant (e.g., a single logical network) run within a given VM.

[0054] By identifying a separate security domain for each container VM, the system is able to isolate problems caused by a compromised container to the container VM and a single tenant. In some such embodiments, the host MFE enforces all network policies for isolating different tenants, while the local MFE is primarily used to logically divide a single tenant's network (e.g., between terminal machines of a tenant logical network, between different logical forwarding elements of a logical network) to provide isolation to a single tenant.

[0055] One of the advantages of distributing logical processing between the host MFE and the local MFE is the ability to avoid hairpinning, which occurs when a data message where both the source container and the destination container are on the local MFE is sent to the host MFE, processed, and then sent back down to the local MFE to be sent to the destination container. Hairpinning can create additional congestion on the network between the local MFE and the host MFE. However, distributing logical processing between the host MFE and the local MFE may require the installation of a more powerful and configurable MFE on each container VM (e.g., an MFE that supports VXLAN tunnels that will terminate in the container VM).

[0056] In some embodiments (e.g., VEPA, VN-Tag, etc.), traffic forwarding is not performed inside the container VM, and logical processing is primarily maintained at the host MFE. The local MFE is only responsible for marking all data messages from the container with a local tag value to identify the LFE for the source container, and passing the data messages to the host MFE for logical processing. The local MFE of some such embodiments is a simple bridge that simply marks all network traffic from the container and forwards it to the host MFE.

[0057] Another advantage of maintaining logical processing in the host MFE is that it reduces redundancy and resource waste. As processing power and virtualization usage increase, a single host may have several container VMs, each with its own local MFE. When performing logical processing in each local MFE, the network control system of some such embodiments needs to install and control data plane components, control plane components, and management plane components in each container VM, thereby unnecessarily consuming more physical (computing, memory, network bandwidth, etc.) resources.

[0058] As mentioned, in some embodiments, a set of network controllers manages the MFEs such that the MFEs implement logical forwarding elements of a logical network. Figure 2An example of different network controllers in a network control system 202 of some embodiments is illustrated. As shown, the network control system 202 manages a set of MFEs of a physical network implementation 102. The network control system 202 includes a centralized network controller (CNC) 210, local host controllers (LHCs) 220, 223, and 226, and local VM controllers (LVCs) 230 and 235.

[0059] CNC 210 maintains high-level abstractions of one or more logical networks and calculates policies and forwarding tables in high-level expressions. In some embodiments, the responsibilities of CNC can be distributed on a controller cluster. The high-level abstraction is then distributed to local controllers (e.g., LHC 220, 223 and 226, and LVC 230 and 235), thereby offloading the low-level data plane programmed to the local controllers running on the host with managed forwarding elements. The separation of high-level and low-level computing improves scalability in some embodiments by distributing complexity to local controllers. In some embodiments, CNC 210 manages LHC 220, 223 and 226 of hosts 120, 123 and 126 to provide logical network data to LHC.

[0060] The LHCs 220, 223, and 226 of some embodiments manage the host MFEs 140, 143, and 146 to implement logical network policies (e.g., L2 / L3 traffic forwarding, tunnel establishment, ACL policies, etc.) based on the logical network data received from the CNC 210. For example, in some embodiments, the LHC or host MFE sets up a tunnel (e.g., VXLAN or STT) with a remote hypervisor that hosts a host VM or container connected to the same LFE (e.g., a tunnel connecting the host MFEs 140, 143, and 146). The LHCs 220, 223, and 226 of some embodiments also communicate with the LVCs 230 and 235 of the containers VM5 and VM6 to manage the local MFEs and receive information about the containers hosted on the containers VM5 and VM6.

[0061] In some embodiments, the LVC is a lightweight local controller that works with the LHC to enable network virtualization for containers in the VM. The LVCs 230 and 235 of some embodiments configure the local MFEs 150 and 155 to mark and forward data messages of containers running on the container VM. In some embodiments, the LVCs 230 and 235 also configure the local MFEs 150 and 155 based on the configuration data received from the LHCs 220 and 223 to perform some logical processing on data messages between containers of the container VM.

[0062] In different embodiments, the various controllers of the network control system can communicate with each other in different ways. In some embodiments, the CNC communicates directly with both the LHC and the LBC, while in other embodiments, the CNC communicates only with the LHC, which then communicates with its local LVC. In some embodiments, the LVC communicates with the LHC using a dedicated communication interface between the virtualization software and the container VM (e.g., a virtual machine communication interface (VMCI)). Alternatively or in combination, the different controllers of the network control system can communicate using a control plane protocol to send control plane data messages through the MFE or through other channels between the controllers.

[0063] Controllers may also need to pass different types of information between themselves. For example, the LVC of some embodiments passes mapping information from the container VM to the LHC so that the LHC can configure the host MFE to interpret the context in the data message tag from the container on the container VM. In some embodiments, the LVC provides application runtime state (e.g., user ID, application type, etc.) or container runtime state (e.g., container interface link state, MAC, etc.) to the CNC, LHC, or to the LHC of other host machines in order to calculate network policies (e.g., firewall rules) and forwarding tables (e.g., VXLAN MAC-VTEP mapping) that should be updated to the MFE and other elements of the network (e.g., gateways). In some embodiments, the CNC populates the forwarding table of the LFE on the host MFE and the forwarding tables of other host MFEs on the remote hypervisor with the local container runtime state.

[0064] exist Figure 1 and Figure 2 In the example of , each of the LHCs is used to manage a software host MFE coupled to a collection of VMs (where each VM executes on a specific host). However, in some embodiments, some or all of these elements are hardware elements or software elements executed on separate computing devices. For example, in some embodiments, the LHC runs as an application on the same computing device as the host MFE and VM, while in other embodiments, the LHC runs on a computing device separate from the MFE and VM. In some embodiments, the MFE is coupled to a collection of physical hosts rather than a collection of VMs, where each physical host runs on a separate computing device. The MFE of some embodiments is a dedicated hardware forwarding element (e.g., a top of rack (ToR) switch), or a combination of a hardware MFE and a software MFE.

[0065] II. Adding containers to a logical network

[0066] When containers configured for logical networks are added to the physical network, the MFE also needs to be configured to handle logical network traffic to and from these containers. Figure 3 Conceptually illustrates a process of some embodiments for adding a container to a logical network at the container VM level. In some embodiments, process 300 is performed by an LVC on a container VM hosting a set of containers to be added to a logical network.

[0067] As shown, process 300 begins by identifying (at 305) an LFE associated with a new container. In some embodiments, the LFE is identified based on a networking configuration for the container received from a CNC or LHC (acting as an intermediary between the CNC and the LVC). In some embodiments, the new container is instantiated on a container VM by a separate compute controller process or by the LVC itself. A new container may be created during the initial startup of a new logical network implemented in a physical network (e.g., a physical data center), or in different embodiments, the new container may be a new addition to an already running logical network.

[0068] The process 300 then sends (at 310) a request for a tag value for the LFE of the identified new container, and receives (at 320) the tag value for the LFE in response. In some embodiments, the LVC sends a request for a tag value for the LFE to an external tag allocation controller. In different embodiments, the tag allocation controller can be a separate controller that maintains a pool of available tags (e.g., VLAN tags) and allocates tags on demand, can run in the same machine as the CNC, or can run on a host machine (each host machine having a separate tag allocation controller). In some cases, the tag allocation controller maintains a separate set of tag values ​​for each container VM; that is, different LFEs can receive different tag values ​​on different container VMs because the use of tags is localized to the connection between the local MFE on the container VM and the host MFE. In addition, the same LFE may be mapped to different tag values ​​on different container VMs.

[0069] In some embodiments, upon receiving a request for a label value, the label allocation controller determines whether the LFE has already been assigned a label value, and if necessary, allocates a new label. In some embodiments, if the LVC has already processed a data message for a container on the same LFE, then no request is sent to the label allocation controller. Instead, the LVC reuses the same label value for the existing LFE.

[0070] After identifying the tag value of the LFE for the newly added container, the process 300 then sends (at 325) the mapping between the LFE and the local tag value to the LHC. In some embodiments, the LVC communicates directly with the CNC and sends the mapping to the CNC instead of, or in addition to, the LHC. The CNC and LHC use the mapping to perform logical processing and network forwarding of data messages for the container over the network, as described below.

[0071] Finally, process 300 configures (at 330) the local MFE of the container VM to use the identified tag. Specifically, as described in more detail in the following sections, the LVC configures the local MFE to (1) tag data messages received from the newly added container with the identified tag value before sending the data messages to the host MFE, and (2) distribute incoming data messages received from the host MFE with the identified tag value (and the destination address of the newly added container) to the newly added container.

[0072] Although shown as a single local MFE 150 in some embodiments, the container VM of some embodiments will include a separate bridge for each LFE of the container VM. Each bridge is connected to the network interface of the container VM (or an interface bound to support load balancing on multiple MFE ports of the container VM) and the container interface of the container for the corresponding LFE. For example, in some embodiments, the LVC configures the local MFE to mark data messages by creating a bridge for each LFE of the container VM, so that the network traffic for the container attached to each bridge is marked with the appropriate label value for the LFE. In some such embodiments, all traffic from the container is sent to the host MFE for processing.

[0073] In some embodiments, the LVC also configures the local MFE to forward all data messages received from the container to the host MFE for further processing. In other embodiments, the LVC configures the local MFE to handle at least a portion of the logical processing and network forwarding for network traffic that remains local to the container VM (i.e., both the source container and the destination container are within the container VM), while forwarding all other traffic to the host MFE for processing.

[0074] Figure 4An example of adding a container logically connected to a first logical forwarding element to a first container VM in four stages 401-404 is illustrated. The first stage 401 shows a host machine 120 including a local host controller (LHC) 220, a host MFE 140, and a container VM 460. The container VM 460 includes a local MFE 150, a local VM controller (LVC) 230, and containers C1-C3. Containers C1-C3 are newly added to the container VM 460 and connected to LFE A of the logical network.

[0075] The second stage 402 shows that the LVC 230 sends a request 410 for a tag value to the tag allocation controller 405. In some embodiments, the LVC detects the addition of a new container C1-C3, or is notified of the new addition by the LHC, CNC, or compute controller. In some embodiments, the request 410 includes the LFE (LFE A) associated with the container C1-C3 (i.e., the UUID representing the LFE). The second stage 402 also shows that the tag allocation controller 405 responds to the request 410 with a new mapping 415, which maps LFE A to the tag value T1.

[0076] The tag allocation controller 405 of some embodiments allocates a different tag value (e.g., VLAN ID) to each of the different LFEs running in the container VM. In some embodiments, the tag allocation controller 405 allocates a different tag value to each container regardless of which LFE the container is connected to. For example, instead of allocating a single tag (i.e., tag 1) to all containers C1-C3, the tag allocation module of some embodiments allocates a different tag to each container C1-C3, even if they are on the same LFE. In some such embodiments, the LVC 230 sends a separate tag request to the tag allocation controller 405 for each of the three containers C1-C3.

[0077] The label distribution controller 405 of some embodiments is an external controller running on a machine separate from the host machine, while in other embodiments, the label distribution controller is a module executing on the host machine 120 within the container VM 460. For example, in some embodiments, the label distribution controller is part of the CNC, or runs on the same physical machine as the CNC (or one of the CNCs in a distributed system).

[0078] The third stage 403 shows that once the LVC 230 receives the label value T1 for LFE A, the LVC sends a mapping 420 of LFE A to label 1 to its LHC 220. The LHC uses the mapping 420 to configure the host MFE to perform network forwarding and logical processing for network traffic to and from containers C1-C3. In some embodiments, the LVC also transmits other information about the container (e.g., MAC address, name of the parent (or shared) VIF (or any unique ID (e.g., UUID)), etc.). In some embodiments, the LHC transmits the mapping and other container information to the CNC (not shown) so that the CNC can calculate the state of the logical network integrated with the container information. The CNC then distributes the logical network state data to the LHC to manage the host MFE on the host machine.

[0079] Finally, the fourth stage 404 shows that the LVC 230 then configures the local MFE 150 running in its container VM. In some embodiments, the LVC receives configuration information from the LHC or CNC for configuring the local MFE to perform tagging and logic processing. The configuration of the local MFE 150 by the LVC 230 includes configuring the local MFE 150 to: tag data messages received from any of the containers C1-C3 with a tag T1 before sending the data messages to the host MFE 140, and to use the tag to identify that the packets received from the host MFE 140 belong to LFE A.

[0080] Figure 3 and Figure 4 Describes the process of adding a container from the perspective of the LVC on the container VM. Figure 5 and Figure 6 Then, from the host machine (for example, Figure 4 These same operations are illustrated from the perspective of an LHC running within virtualized software on a host machine 120 in FIG.

[0081] Figure 5 The process 500 conceptually illustrates some embodiments of adding a container to a logical network at the host controller level. In some embodiments, the process 500 is performed by an LHC running on a host machine to manage the host MFE to which the container VM containing the newly added container is connected. Figure 6 Describing process 500, Figure 6 An example of adding a container to a first logical forwarding element in three stages 601 - 603 is shown.

[0082] Process 500 begins by receiving (at 505) a network configuration for a container that has been instantiated on a particular container VM (on a local host machine) having a local MFE running on the VM. In some embodiments, process 500 receives the network configuration for the container from a CNC that provides logical network data to an LHC in a physical network to implement a logical network. Figure 6 In the first stage 601, LHC 220 (controller on the host machine) receives network configuration 620 for containers C1-C3 from CNC 210. The network configuration data of some embodiments includes logical network policies (e.g., firewall, QoS, ACL, etc.), logical forwarding rules, etc. for containers C1-C3 in the context of a logical network.

[0083] In some embodiments, the received network configuration is the result of an instruction received from a tenant in the data center to add a new container to the tenant's logical network. Based on the container information received from the LVC, the network configuration may also come from changes in the physical network (for example, when VMs are added to and removed from the physical network). The network configuration of some embodiments includes LFE information (for example, logical ports and LFEs associated with the container) and configuration information for the container. If container C1-C3 is the first terminal machine of a logical network running on host machine 120, then the network configuration data may include all necessary information for configuring host MFE 140 to implement the logical network (for example, forwarding information for other terminal machines in the logical network, etc.).

[0084] Return to reference Figure 5 , process 500 then receives (at 510) an LFE label mapping that associates logical elements of the logical network (e.g., logical switches, logical ports, LFEs, etc.) with label values ​​assigned to the logical elements at the container VM hosting the newly added container. Figure 6 The second stage 602 illustrates that the LHC 220 receives a mapping 620 from the LVC 230. The mapping 620 identifies the label value T1 associated with the LFE A to which the containers C1-C3 are connected. In some embodiments, the LHC receives the LFE label mapping for the container from the LVC that manages the local MFE of the container VM, while in other embodiments, the LHC receives the LFE label mapping from the CNC along with the network configuration (e.g., after the CNC receives the mapping from the LHC).

[0085] Then, process 500 configures (at 515) the host MFE to perform network forwarding and other logical processing for data messages sent to and from the container based on the received network configuration and mapping. In some embodiments, the LHC configures the host MFE by generating flow entries stored in a forwarding table of the host MFE based on the received network configuration and LFE label mapping to modify the forwarding behavior of the host MFE. In other embodiments, the LHC configures the host MFE by configuring various modules of the host MFE.

[0086] More specifically, the LHC configures the host MFE to tag data messages destined for the container with an appropriate label value before sending the data message to the local MFE of the container VM. The LHC also configures the host MFE to perform logical processing on the data message from the container based on the logical element associated with the container. In some embodiments, the logical element associated with the container is identified based on the LFE label mapping received from the LVC. This processing of the data message is described in more detail in the following section.

[0087] Figure 6 The third stage 603 shows the LHC 220 configuring the host MFE 140. The LHC configures the host MFE to tag data messages destined for the container with appropriate tag values ​​and to perform logical processing on data messages from the container based on logical elements associated with the container, as described in the previous paragraphs.

[0088] In some embodiments, LHC 220 generates local MFE data 625 based on the LFE label mapping and logical network information received from LVC 230 and CNC 210 to be sent to LVC 230. LVC 230 of some embodiments uses the local MFE data to configure local MFE 150. In some embodiments, the generated local MFE data 625 includes a set of flow entries stored by LVC 230 in a forwarding table of local MFE 150 to control the forwarding behavior of local MFE 150.

[0089] Figure 7 An example of adding a container that is logically connected to a second logical forwarding element to the same container VM 460 in four stages 701-704 is illustrated. The second logical forwarding element may be part of the same logical network as the first logical forwarding element (e.g., if all containers on a particular container VM are required to be part of the same logical network for isolation reasons), or may be part of a different logical network. The first stage 701 is similar to Figure 61 and shows the host machine 120 with the LHC 220, the host MFE 140 and the container VM 460. The first stage 701 also shows that new containers C4 and C5 have been added to the local MFE. However, unlike the containers C1-C3 attached to LFE A, the containers C4 and C5 are attached to a different LFE B.

[0090] In the second stage 702, the LVC 230 of container VM5 communicates with the label distribution controller 605 to learn the associated labels for the LFEs of containers C4 and C5. Since containers 4 and 5 are attached to LFE B, the label distribution controller identifies a new label T2 to be associated with containers C4 and C5 of LFE B. The third stage 703 shows that the LVC 230 then communicates the mapping of LFE B with label 2 to the LHC 220. The LHC 220 (and / or CNC (not shown)) uses the mapping to enable the host and local MFE to create and implement logical network policies.

[0091] In the fourth stage 704, the LHC 220 modifies the host MFE 140 and provides the local MFE rules 730 to the LVC 230 based on the received mapping. The LVC 230 modifies the local MFE based on the local MFE rules 730. Once the host MFE and the local MFE have been configured based on the LFE label mapping, the host MFE identifies the associated logical element for the data message received from the container VM 460 based on the label value and performs logical processing on the data message accordingly.

[0092] In addition to adding existing containers to the logical network, the LHC of some embodiments is also responsible for instantiating new containers in the physical network and associating the containers with the logical elements of the logical network. The LHC of some of these embodiments determines whether the container VM is available on the host machine. When the container VM is not available, the LHC of some embodiments communicates with the VM generation module (e.g., Nova) to create a new container VM. The LHC of some embodiments then communicates with the container orchestration system (e.g., Docker) for creating the new container and connects the new container to the local MFE.

[0093] In some embodiments, the LHC also communicates with a network port module (e.g., Neutron) to identify a new logical port for the container. The LHC of some embodiments sends a create port request to the network port module with a label name, a virtual interface (VIF) name (e.g., a VIF UUID), and a logical forwarding element to which the container is to be added. The network port module then assigns the logical port to the logical forwarding element. Once the container is instantiated and associated with the logical element, the LHC adds the container to the logical network as described above.

[0094] As described above, the LHC (or CNC) modifies the host MFE and the local MFE using a mapping of logical elements (e.g., logical ports, logical forwarding elements, etc.) to label values. The host MFE uses the mapping to identify the source LFE and logical port of the ingress data message based on the associated label value and the physical port of the host MFE that received the data message. In some embodiments, the host MFE uses the physical port of the host MFE and the associated label value to identify the LFE, and uses the unique address of the source container (e.g., MAC address) to identify the logical port within the LFE.

[0095] Figure 8 An example of a mapping between a logical network and a physical network is shown in FIG. Figure 1 802 and a pair of logical forwarding elements (e.g., logical switches) 105 and 110 and a physical network 802 that implements the two logical forwarding elements. The LFEs also show various logical ports A1-A6 of the first LFE 105 and various logical ports B1-B5 of the second LFE 110. Similarly, the representation of the physical network 802 shows physical ports (e.g., virtual NICs, tunnel ports) 1-4 for each of the host MFEs 140, 143, and 146, physical ports 1-6 for the local MFE 150 on the container VM5, and physical ports 1-4 for the local MFE 155 on the container VM6. The figure also illustrates mappings 850 and 855 between the LFEs and the physical networks.

[0096] Specifically, mapping table 850 shows the mapping between logical ports and physical ports for LFE A. LFE A has six ports A1-A6, which are connected to VM1, VM2, containers C1-C3, and VM7, respectively. Mapping table 850 shows that VM1, VM2, and VM7 are directly connected to host MFE 140, 143, and 146 through physical ports MFE1:1, MFE2:2, and MFE3:1, respectively. Since these do not have a local MFE, there are no label values ​​associated with these VMs. However, containers C1-C3 are all connected to local MFE 150 connected through port 2 of host MFE 140. Traffic for each of these containers is associated with label value T1.

[0097] Similarly, mapping table 855 shows the mapping between logical ports and physical ports of LFE B. LFE B has five ports B1-B5 connected to VM3, VM4, containers C4 and C5, and VM8, respectively. Mapping table 855 shows that VM8 is directly connected to host MFE 146 through physical port 1 of MFE3. Containers C4 and C5 are connected to local MFE 150 like containers C1-C3, and local MFE 150 is connected to host MFE 140 through port 2. However, when containers C4 and C5 are associated with LFE B, the traffic of each of these containers is associated with a different label value T2. In addition, VM3 and VM4 are virtualized within container VM6. VM3 and VM4 are connected to local MFE 155, and local MFE 155 is connected to host MFE 143 through port 3. Like containers C4 and C5, the traffic of VM3 and VM4 is associated with label value T2. That is, in some embodiments, the same label value is used for a specific logical forwarding element on different container VMs. However, in other embodiments, different label values ​​than those used for containers C4 and C5 may be used for traffic to and from VM3 and VM4.

[0098] III. Forwarding container network traffic

[0099] Once the container has been mapped to a logical port in the logical network, the CNC of some embodiments generates logical network state data for implementing the logical network through the managed forwarding elements. The local controllers (e.g., LHC and LVC) receive the logical network state data and modify the forwarding behavior of their managed forwarding elements. At this point, the MFE can process logical network traffic sent to and from the container.

[0100] Fig. 9 The process 900 of some embodiments is performed by a host MFE that has been configured by LHC to forward container traffic according to a logical network policy. Fig.10 Describing process 900, Fig.10 An example of forwarding data messages from a container through a logical network is illustrated. As shown in the first stage 1001, each of these stages illustrates a host MFE 140 connected to a network 1050 (i.e., the physical infrastructure of a data center through which tunnels are maintained between the host MFE 140 and other MFEs) and a container VM 460. The container VM 460 includes a local MFE 150 and containers C1-C5. In this example, C1-C3 are attached to LFE A, while C4 and C5 are attached to LFE B (as in the example in the previous section).

[0101] Return to reference Fig. 9 , process 900 receives (at 905) a data message with a local tag from a local MFE. Fig.10 The first stage 1001 illustrates the local MFE 150 receiving a data message 1010 from a container C1. The data message includes a payload, a destination MAC address, and a source MAC address (potentially among other header fields). The payload represents the data sent to the destination and may include higher layer (e.g., IP, TCP, application layer, etc.) headers. The source MAC address (C1) is the address of the sender container C1, while the destination MAC address (99) identifies the machine to which the data message 1010 is addressed (e.g., a logical router port, another machine on the same logical switch). The local MFE 150 tags the data message 1010 with a local tag value T1 based on, for example, the port of the local MFE 150 to which the container C1 is attached.

[0102] The second stage 1002 shows that the local MFE 150 sends the data message, which now includes the tag value T1 (which may be, for example, a VLAN tag in a layer 2 header), to the host MFE 140. The host MFE 140 receives the data message 1010. The third stage 1003 shows that the host MFE 140 has received the data message 1010. In some embodiments, the host MFE 140 processes the received data message by storing information (e.g., logical port, MAC address, other header data, etc.) in metadata and / or register fields of the data message object 1020.

[0103] Process 900 next identifies (at 910) the LFE to which the data message belongs based on the local tag. In the third stage 1003, the host MFE 140 has identified the source logical port (A3) of LFE A based on the tag (T1) of the data message. In some embodiments, the host MFE identifies the LFE for the data message based on the LFE tag mapping, and identifies the logical port for the data message based on the {LFE, ingress port, source MAC} and logical port mapping. In some embodiments, the mappings are each specific to a specific port of the MFE. For example, the host MFE identifies the logical port for the data message based on (1) the physical port of the host MFE that received the data message, (2) the logical forwarding element (determined based on the local tag value), and (3) the source MAC address of the data message. The host MFE 140 of some embodiments identifies logical data based on a set of mapping tables that include various mappings between local tag values ​​and logical entities. As shown in the third stage, the host MFE 140 stores the logical ingress port (A3) in a field of the data message object 1020. Although the data message object 1020 only displays the logical ingress port for the data message 1010, the register / metadata fields of the data message object 1020 of some embodiments store various other information about the data message 1010 (e.g., the current stage of packet processing, intermediate logical processing context information, etc.).

[0104] After process 900 identifies the LFE for the data message, process 900 applies (at 915) a policy based on the identified LFE and forwards (at 920) the data message through the logical network. In some embodiments, the policy and logical forwarding performed at host MFE 140 are generated by the LHC based on the logical network data received from the CNC. In some embodiments, logical processing is defined per logical entity (e.g., logical switch port, logical switch, logical router port, etc.) so that the host MFE processes a packet through the logical network (e.g., identifies the ingress port of the LFE and then forwards the packet to the logical egress port (which may correspond to other logical forwarding elements)), and the host MFE applies the appropriate logical network policy. Fig.10 The third stage 1003 illustrates the host MFE applying a logical policy at the ingress port level based on identifying the logical ingress port A3 for the message.

[0105] In addition to logically forwarding the data message, the host MFE of some embodiments also physically forwards the data message to its destination based on the logical processing. Unless the destination is located on the same host machine, the host MFE sends the data message to the physical infrastructure (e.g., via a tunnel). This may require sending the data message to other host machines or gateways (which are connected to an external physical network outside the physical infrastructure on which the logical network is implemented). In the fourth stage 1004, the host MFE 140 has removed the local tag and forwarded the data message 1010 to the network. In place of the local tag, the MFE has added logical context information (specifically, the logical egress port determined based on the logical processing).

[0106] In some cases, data messages from one container are destined for other containers on the same container VM. Fig.11 An example of forwarding a data message from a container to other containers on the same container VM over a logical network in four stages 1101-1104 is illustrated. The first stage 1101 is similar to the first stage 1102 except that the data message 1110 is addressed to other containers (C3) on the same container VM 460, rather than other machines implemented outside the host. Fig.10 1001. In this example, data message 1110 is forwarded to the host MFE even though the destination of the data message is on container VM 460. In some other embodiments, the data message is not forwarded to the host MFE at all, but rather the network control system distributes portions of the logic processing to the local MFEs of the container VMs so that the local MFEs can forward all local traffic between the containers of each container VM.

[0107] The second stage 1102 shows that the host MFE 140 has received the tagged data message (tagged with LFE A's T1) and stores the data message information (source logical port A3) for the data message 1110 in the data message object 1120. The host MFE 140 identifies the logical ingress port (A3) based on the port through which the data message was received, the local tag attached to the data message, and the source MAC address of the data message. The second stage also shows that the host MFE 140 performs logical processing (e.g., logical forwarding) and applies policies to the data message 1110.

[0108] The third stage 1103 illustrates that the host MFE has identified the logical egress port (A5) for the data message by performing logical forwarding on the data message. That is, the destination MAC address C3 is mapped to the logical port A5 to which the data message is logically forwarded. Thus, at the third stage 1103, the host MFE 140 applies an egress policy to the data message based on the logical port information.

[0109] The fourth stage 1104 shows that the host MFE 140 has re-tagged the data message 1110 with the label value T1 because the logical egress port A5 is also on the same logical forwarding element. On the other hand, if the data message is processed through a logical router to another logical forwarding element (e.g., LFE B), then the data message will be tagged with a different label value. Since the destination container is on container VM 460, the host MFE 140 sends the data message 1110 back down to the local MFE 150. The local MFE 150 then identifies the container to which to forward the data message 1110 based on the label T2 and the destination MAC, and delivers the data message to the appropriate container C3. Referring to FIG. Fig.12 and Fig.13 The logical processing and forwarding of data messages down to the container is described in more detail.

[0110] Fig.12 The process 1200 of some embodiments for performing logical processing on a data message destined for a container is conceptually illustrated. The process 1200 of some embodiments is performed by a host MFE of a host machine having a container running on a container VM. Fig.13 Describing process 1200, Fig.13 An example of a data message received at a host and addressed to a container running on the host is illustrated.

[0111] As shown, process 1200 begins by receiving (at 1205) a data message with a logical context tag. In some embodiments, the logical context tag identifies a logical egress port of a (logical forwarding element) corresponding to a container running on a container VM connected to a host MFE. In some embodiments, the logical egress port may have been determined by a first host MFE to process the data message (e.g., Fig.10 Host MFE in the . Fig.13 The first stage 1301 shows that the host MFE 140 receives a data message 1310 from its connection to the physical infrastructure 1050 (e.g., from a gateway or other host MFE via a tunnel). The data message 1310 includes a payload (PLD), a source MAC address, and a destination MAC address. In addition, the data message includes logical context data that specifies a logical egress port determined by the first-hop MFE (not counting the local MFE if the packet originates from a container).

[0112] Process 1200 applies (at 1210) a policy to a data message based on a logical context stored in the data message. Fig.13The second stage 1302 of FIG. 1 shows that the host MFE 140 has received the data message 1310 with logical context information and stored some of the logical context information (i.e., the destination logical port A3) in a data message object 1320 that is created for use during the processing of the data message. The second stage also shows that the host MFE 140 performs logical processing and applies policies to the data message 1310. In this case, the logical processing primarily entails identifying that the specified logical egress port maps to the physical port of the host MFE that is connected to the local MFE.

[0113] Process 1200 removes (at 1215) the logical context tag from the data message, thereby removing all logical network information from the data message. This allows the logical network to be transparent to the container and local MFE receiving the packet.

[0114] The process 1200 also adds (at 1220) a local tag to the data message. The host MFE of some embodiments identifies the local tag based on the logical context stored in the data message object 1320 (e.g., by mapping the logical forwarding element to which the logical egress port belongs to a particular local tag). Finally, the process 1200 delivers (at 1225) the data message to the container VM. In some embodiments, the local MFE on the container VM then forwards the data message to the destination container.

[0115] Fig.13 The third stage 1303 shows that the host MFE 140 has removed the logical context from the packet and added a local tag T1 (which maps to the LFE A to which the logical egress port belongs) before sending the packet to the local MFE 150 on the container VM 460. In the fourth stage 1304, the local MFE 150 receives the data message 1310, removes the local tag T1, and forwards the packet to the destination container C1 based on the local tag T1 and the destination MAC address C1.

[0116] IV. Electronic Systems

[0117] Many of the above features and applications are implemented as a software process, which is specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of processors, or other processing units), they cause the (one or more) processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0118] In this specification, the term "software" refers to applications stored in magnetic storage including firmware residing in a read-only memory or which can be read into a memory to be processed by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement the software inventions described herein is within the scope of the present invention. In some embodiments, when a software program is installed to run on one or more electronic systems, the software program defines one or more specific machine implementations that perform the operations of the software program.

[0119] Fig.14 A computer system 1400 is conceptually illustrated that implements some embodiments of the present invention. The computer system 1400 can be used to implement any of the above-described hosts, controllers, and managers. Thus, it can be used to perform any of the above-described processes. The computer system includes various types of non-transient machine-readable media and interfaces for various other types of machine-readable media. The computer system 1400 includes a bus 1405, (one or more) processing units 1410, a system memory 1425, a read-only memory 1430, a permanent storage machine 1435, an input machine 1440, and an output machine 1445.

[0120] Bus 1405 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal machines of computer system 1400. For example, bus 1405 communicatively connects processing unit(s) 1410 with read-only memory 1430, system memory 1425, and permanent storage machine 1435.

[0121] From these various memory units, the processing unit(s) 1410 retrieves instructions to be executed and data to be processed in order to perform the processes of the present invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. The read-only memory (ROM) 1430 stores static data and instructions required by the processing unit(s) 1410 and other modules of the computer system. On the other hand, the permanent storage machine 1435 is a read and write memory machine. Such a machine is a non-volatile memory unit that stores instructions and data even when the computer system 1400 is turned off. Some embodiments of the present invention use a mass storage machine (such as a magnetic disk or optical disk and its corresponding disk drive) as the permanent storage machine 1435.

[0122] Other embodiments use a removable storage machine (such as a floppy disk, flash drive, etc.) as a permanent storage machine. Like the permanent storage machine 1435, the system memory 1425 is a read and write memory machine. However, unlike the storage machine 1435, the system memory is a volatile read and write memory, such as a random access memory. The system memory stores some instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in the system memory 1425, the permanent storage machine 1435, and / or the read-only memory 1430. From these various memory units, the processing unit(s) 1410 retrieves instructions to be executed and data to be processed in order to perform the processes of some embodiments.

[0123] The bus 1405 is also connected to input and output machines 1440 and 1445. The input machine enables a user to transmit information and select commands to the computer system. The input machine 1440 includes an alphanumeric keyboard and a pointing machine (also known as a "cursor control machine"). The output machine 1445 displays images generated by the computer system. The output machine includes a printer and a display machine, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include machines such as a touch screen that serves as both an input machine and an output machine.

[0124] Finally, if Fig.14 As shown, bus 1405 also couples computer system 1400 to a network 1465 via a network adapter (not shown). In this manner, the computer may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all of the components of computer system 1400 may be used in conjunction with the present invention.

[0125] Some embodiments include electronic components such as microprocessors, storage and memory that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code such as produced by a compiler, and files including higher-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0126] Although the above discussion mainly refers to a microprocessor or multi-core processor that executes software, some embodiments are performed by one or more integrated circuits, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such an integrated circuit executes instructions stored on the circuit itself.

[0127] As used in this specification, the terms "computer", "server", "processor", and "memory" all refer to electronic or other technical machines. These terms do not include people or groups of people. For the purposes of this specification, the terms "display" or "displaying" mean displaying on an electronic machine. As used in this specification, the terms "computer-readable medium", "multiple computer-readable media" and "machine-readable medium" are completely limited to tangible, physical objects that store information in a form that is readable by a computer. These terms do not include any wireless signals, wired download signals, and any other short-term or transient signals.

[0128] Although the present invention has been described with reference to many specific details, it will be appreciated by those of ordinary skill in the art that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. For example, throughout this specification, reference is made to computing and network environments including virtual machines (VMs). However, a virtual machine is merely an example of a data computing node (DCN) or a data computing terminal node (also referred to as an addressable node). A DCN may include a non-virtualized physical host, a virtual machine, a container that runs on top of a host operating system without the need for a hypervisor or a separate operating system, and a hypervisor kernel network interface module.

[0129] In some embodiments, the VM uses the resources of the host virtualized by virtualization software (e.g., a hypervisor, a virtual machine monitor, etc.) to run with its own client operating system on the host. The tenant (i.e., the owner of the VM) can choose which applications to run on top of the client operating system. On the other hand, some containers are structures that run on top of the host operating system without the need for a hypervisor or a separate client operating system. In some embodiments, the host operating system uses namespaces to isolate containers from each other, and thus provides operating system-level isolation of different application groups running in different containers. This isolation is similar to the VM isolation provided in a hypervisor virtualization environment that virtualizes the system hardware, and can therefore be viewed as a form of virtualization that isolates different application groups running in different containers. This container is lighter than a VM.

[0130] In some embodiments, the hypervisor kernel network interface module is a non-VM DCN that includes a network stack with a hypervisor kernel network interface and receive / transmit threads. An example of a hypervisor kernel network interface module is an ESXi host as part of VMware Inc. TM The vmknic module is part of the hypervisor.

[0131] Those skilled in the art will recognize that although this specification refers to VMs, the examples given may be any type of DCN, including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. In fact, in some embodiments, the example network may include a combination of different types of DCNs.

[0132] Many graphs (e.g. Figure 3 , Figure 5 , Fig. 9 and Fig.12 ) conceptually illustrates the processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in a series of continuous operations, and different specific operations may be performed in different embodiments. In addition, the process can be implemented using several sub-processes, or as part of a larger macro process. In view of the above, it will be understood by those skilled in the art that the present invention is not limited by the above illustrative details, but is defined by the appended claims.

Claims

1. A method for a local network controller running on a host computer to manage a first managed forwarding element, the first managed forwarding element forwarding traffic on the host computer for a set of logical networks, the method include: receiving, from a centralized network controller, logical network configuration information for a logical network to which a set of containers are logically connected, the set of containers running within a virtual machine (VM) on the host computer connected to the first managed forwarding element; receiving, from the VM, a mapping of a label value to a logical forwarding element LFE of the logical network to which the set of containers is connected, the label value being used by a second managed forwarding element running within the VM to identify the LFE for a data message sent from the second managed forwarding element to the first managed forwarding element; and The first managed forwarding element is configured to apply the logical network configuration information to data messages received from the VM and tagged with the label value.

2. The method of claim 1, wherein the tag value is a VLAN tag associated with the LFE.

3. The method of claim 1, wherein the logical network is a first logical network and the label value is a first label value, the method further comprising: include: receiving logical network configuration information for a second logical network to which a second set of containers running in the VM is logically connected; receiving a mapping of a second label value used by the second managed forwarding element to an LFE of the second logical network to which the second set of containers is connected; as well as The first managed forwarding element is configured to apply the logical network configuration information to data messages received from the VM and tagged with the second label value.

4. The method of claim 1, wherein the local network controller is a first local network controller, the method further comprising: include: Configuration information is sent to a second local network controller to manage the second managed forwarding element, the configuration information being used to configure the second managed forwarding element to apply a portion of logical network configuration information related to forwarding data messages between containers on the VM.

5. The method of claim 1, wherein receiving the mapping include: A media access control (MAC) address is received for each container in the set of containers.

6. The method of claim 1, wherein the containers in the container set are used to isolate services running in each container.

7. A method for configuring a first managed forwarding element running on a virtual machine VM to process data messages from a plurality of containers running on the VM, the method include: identifying a logical forwarding element LFE to which a specific container executing on the VM is logically connected; identifying a label value mapped to the LFE; as well as The first managed forwarding element running on the VM is configured to associate a data message received from the particular container with the label value before sending the data message along an egress path out of the VM for processing based on the label value to forward the data message to a destination of the data message.

8. The method according to claim 7, in: The method is performed by a network controller executed on the VM; The VM runs on a host computer; The network controller is a first network controller; as well as A second network controller runs external to the VM on the host computer.

9. The method of claim 8, wherein identifying the LFE include: Configuration information identifying the LFE is received from the second network controller.

10. The method of claim 8, wherein identifying the LFE include: Configuration information identifying the LFE is received from a third network controller that operates external to the host computer and manages a plurality of host computers.

11. The method of claim 8, further comprising: include: Sending a mapping of the label value and the LFE to the second network controller; as well as A container information set is sent to the second network controller, the container information set including application context data for a set of applications executed in the set of containers.

12. The method of claim 7, wherein identifying the tag value include: Sending a request to a label allocation controller, the label allocation controller maintaining a pool of available label values ​​specified for the VM; as well as A response is received including the tag value.

13. The method of claim 7, wherein a second managed forwarding element external to the VM and executing on the same host computer as the VM processes the data message from the specific container using the label value for forwarding in a logical network including the LFE.

14. The method of claim 13, wherein the LFE is a first LFE, the specific container is a first container, and the tag value is a first tag value, the method further comprising: include: identifying a second LFE to which a second container executing on the VM is logically connected; identifying a second label value mapped to the second LFE; as well as The first managed forwarding element running on the VM is configured to associate a data message received from the second container with the second label value before sending the data message along an egress path out of the VM for processing based on the second label value to forward the data message to a destination of the data message.

15. A method for forwarding a data message to a container on a specific virtual machine executing on a host computer, the method include: At a first managed forwarding element executing on the particular virtual machine: receiving a data message from a second managed forwarding element executing on the host computer, the second managed forwarding element forwarding data messages between different VMs executing on the host computer; identifying a particular container as a destination for a received data message; as well as Forwards the received data message to the specific container identified.

16. The method of claim 15, in: The data message is received with a tag; and Identifying the particular container includes using the label to identify the particular container.

17. The method of claim 16, wherein the tag uniquely identifies the particular container because the tag is associated with only one container executing on the particular virtual machine.

18. The method of claim 16, wherein the label is used to identify the specific container include: The particular container that is a destination for the received data message is identified using the tag and at least one header value of the received data message.

19. The method of claim 18, in: The first managed forwarding element implements a plurality of logical forwarding elements (LFEs), each logical forwarding element spanning more than one host computer; and The tag is associated with the LFE to which the identified container belongs.

20. The method of claim 16, wherein the second managed forwarding element associates the tag with the data message when the second managed forwarding element receives the data message and identifies the particular container as a destination for the data message.

21. The method of claim 20, in: The second managed forwarding element implements multiple LFEs with other managed forwarding elements on other host computers, each LFE spanning more than one host computer; and The data message is associated with the label after the second managed forwarding element determines that the data message is intended for a particular LFE associated with the label based on a set of logical attributes received by the second managed forwarding element with the data message.

22. The method of claim 20, wherein the first managed forwarding element and the second managed forwarding element are configured by a set of controllers to associate the label with the particular container when the particular container is instantiated on the particular virtual machine.

23. A machine-readable medium storing a program, which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 22.

24. An electronic device, include: A collection of processing units; as well as A machine-readable medium storing a program, which, when the program is implemented by at least one of the processing units, implements the method according to any one of claims 1 to 22.

25. A system comprising means for implementing the method according to any one of claims 1-22.

26. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 22.

Images