Controller and network configuration method

Abstract

The present disclosure relates to a controller and network configuration method. The controller comprises an orchestrator of a virtualized computing infrastructure, wherein the orchestrator is configured to: receive namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; receive virtual execution element specification data specifying a virtual execution element, wherein the virtual execution element specification data comprises a namespace object specifying a namespace of the virtual execution element; and send, based on the namespace specification data and the namespace object, one or more requests to a network controller of the virtualized computing infrastructure to create respective virtual network interfaces of the first virtual network and the second virtual network for the virtual execution element.

Description

[0001] This application is a divisional application of Chinese patent application No. 201910809461.6, filed on August 29, 2019, entitled "Multiple Networks for Virtual Execution Elements". Technical Field

[0002] This disclosure relates to a virtualized computing infrastructure, and more specifically, to configuring network connectivity for virtual execution elements of a virtualized computing infrastructure deployed within a network. Background Technology

[0003] In a typical cloud data center environment, a large number of interconnected servers provide computing and / or storage capacity to run a variety of applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., the data center's customers). For instance, a data center may host all infrastructure units, such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, storage systems and application server clusters are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide user support facilities located in various physical hosting facilities for infrastructures spread across the globe.

[0004] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. In particular, modern data centers have widely utilized virtualized environments, in which virtual machines, such as virtual machines or containers, are deployed and run on the underlying computing platform of physical computing devices.

[0005] Virtualization within a data center offers several advantages. One advantage is its significant efficiency gains. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become increasingly powerful, making virtualization easier and more efficient. A second advantage is the significant control virtualization provides over the computing infrastructure. As physical computing resources become alternatives, for example, in cloud-based computing environments, the provisioning and management of computing infrastructure becomes easier. Therefore, in addition to the efficiency and higher return on investment (ROI) offered by virtualization, enterprise IT professionals often prefer virtualized computing clusters in data centers due to their management advantages.

[0006] Containerization is a virtualization solution based on operating system-level virtualization. A container is a lightweight, portable execution element used to isolate applications from each other and from the host machine. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to a container image and run as a single lightweight package on any host or virtual machine that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work across different computing environments. Containers offer the promise of continuous operation from one computing environment to another virtual or physical computing environment.

[0007] Due to the inherent lightweight nature of containers, a single host can support more container instances than a traditional virtual machine (VM). Containers are typically short-lived, can be created and moved more efficiently than VMs, and can also be managed as groups of logically related components (sometimes called "pods" on some orchestration platforms, such as Kubernetes). These container characteristics influence the requirements for container networking solutions: the network should be agile and scalable. Virtual machines, containers, and bare-metal servers may need to coexist in the same compute environment and be able to communicate between different deployments of the application. Container networks should also be unaware of how to use the various orchestration platforms used to deploy containerized applications.

[0008] Managing the deployment and infrastructure of application execution in computing infrastructure may involve two main roles: (1) orchestration—for automating the deployment, scaling, and operation of applications across host clusters and providing computing infrastructure, which may include container-centric computing infrastructure; and (2) network management—for creating virtual networks within the network infrastructure to enable packet communication between applications running on virtual execution elements (e.g., containers or virtual machines) and between applications running on traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0009] Typically, this describes a list of virtual networks used to specify network infrastructure for virtual execution elements and to create multiple virtual network interfaces that virtual execution elements can use to communicate via these virtual networks. For example, namespace specification data for a virtualized computing infrastructure namespace can specify multiple virtual networks. Based on the namespace specification data, the orchestrator of the virtualized computing infrastructure can deploy virtual execution elements labeled with the namespace to the namespace. Based on the multiple virtual networks specified in the namespace specification data, the orchestrator can enable the network controller of the virtualized computing infrastructure to configure virtual execution elements with corresponding virtual network interfaces for the multiple virtual networks.

[0010] In some examples, multiple virtual networks specified in the namespace specification data constitute the primary virtual network group for a virtual execution element deployed to the namespace. The configuration data for the virtual execution element to be deployed to the namespace may also include network annotations specifying secondary virtual network groups. In some cases, at least one of the secondary virtual network groups is not represented by the namespace specification data (i.e., it is not in the primary virtual network group). In some examples, the corresponding virtual execution element is configured with virtual network interfaces for each virtual network in the primary virtual network group specified in the namespace specification data and for each virtual network in the secondary virtual network group specified in the configuration data of the virtual execution element.

[0011] These technologies can provide one or more technical advantages. For example, by specifying multiple virtual networks in the namespace specification data, all virtual execution elements deployed to the corresponding namespace will be configured to communicate via any of the multiple virtual networks. As another example, by simply including annotations specifying virtual networks in the namespace of a virtual execution element, virtual execution elements can be efficiently distributed across a primary virtual network group. Previous methods of creating namespaces limited each namespace configuration to specifying only one virtual network, which restricted the types of applications that could run in a virtual execution element. As yet another example, because the configuration data of a virtual execution element can specify secondary virtual network groups, virtual execution elements deployed as described herein can be configured with virtual network interfaces for each virtual network in the primary virtual network group specified for the namespace and for each virtual network in the secondary virtual network groups specified for the virtual execution element. This provides users with a high degree of versatility and coarse-grained and fine-grained control over network policies for communication between virtual execution elements.

[0012] In one example, a controller includes one or more computing devices, each of which includes processing circuitry coupled to a storage device. The controller further includes: an orchestrator for a virtualized computing infrastructure, configured to be executed by the processing circuitry, wherein the orchestrator is configured to: receive namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; and send at least one request to a network controller of the virtualized computing infrastructure based on the namespace specification data to create corresponding virtual network interfaces for the first and second virtual networks for virtual execution elements to be deployed to the namespace and instantiated in the computing devices of the virtualized computing infrastructure; and a network controller, configured to be executed by the processing circuitry, wherein the network controller is configured to: send interface configuration data to the computing devices to configure the first virtual network interface for the first virtual network and the second virtual network interface for the second virtual network.

[0013] In another example, one method includes: receiving namespace specification data from an orchestrator of a virtualized computing infrastructure, which specifies a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; and the orchestrator sending at least one request to a network controller of the virtualized computing infrastructure based on the namespace specification data to create corresponding virtual network interfaces for the first and second virtual networks for virtual execution elements to be deployed to the namespace and instantiated in a computing device of the virtualized computing infrastructure; and the network controller sending interface configuration data to the computing device to configure the first virtual network interface for the first virtual network and the second virtual network interface for the second virtual network.

[0014] In another example, a non-transitory computer-readable medium includes instructions for causing one or more processors to: execute an orchestrator of a virtualized computing infrastructure to receive namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; execute the orchestrator to send at least one request to a network controller of the virtualized computing infrastructure based on the namespace specification data to create corresponding virtual network interfaces for the first and second virtual networks for virtual execution elements to be deployed to the namespace and instantiated in a computing device of the virtualized computing infrastructure; and execute the network controller to send interface configuration data to the computing device to configure the first virtual network interface for the first virtual network and the second virtual network interface for the second virtual network.

[0015] In another example, a controller includes an orchestrator for a virtualized computing infrastructure, wherein the orchestrator is configured to: receive namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; receive virtual execution element specification data specifying a virtual execution element, wherein the virtual execution element specification data includes a namespace object specifying the namespace of the virtual execution element; and send one or more requests to a network controller of the virtualized computing infrastructure based on the namespace specification data and the namespace object to create corresponding virtual network interfaces for the first and second virtual networks for the virtual execution elements.

[0016] In another example, a network configuration method includes: receiving namespace specification data by an orchestrator of a virtualized computing infrastructure, the namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace; receiving virtual execution element specification data specifying a virtual execution element by the orchestrator, wherein the virtual execution element specification data includes a namespace object specifying the namespace of the virtual execution element; and sending one or more requests by the orchestrator to the network controller of the virtualized computing infrastructure based on the namespace specification data and the namespace object to create corresponding virtual network interfaces for the first and second virtual networks for the virtual execution elements.

[0017] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will become apparent from the specification, drawings, and claims. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating an example computing infrastructure that can implement the technologies described herein;

[0019] Figure 2 This is a block diagram of an example computing device according to the technology described in this disclosure, the example computing device including a network module for configuring multiple virtual network interfaces for a group of one or more virtual execution elements sharing at least one virtual network interface;

[0020] Figure 3 This is a block diagram of an example computing device based on the technology described in this disclosure;

[0021] Figure 4 This is a flowchart illustrating the creation of multiple network virtual interfaces for a virtual execution element using a single network module according to the techniques described in this disclosure;

[0022] Figure 5 This is a flowchart illustrating the creation of multiple network virtual interfaces for virtual execution elements based on namespaces according to the techniques described in this disclosure.

[0023] Throughout the specification and drawings, the same reference numerals denote the same elements. Detailed Implementation

[0024] Figure 1This is a block diagram illustrating example computing infrastructure 8, where examples of the technologies described herein can be implemented. Typically, data center 10 provides an operating environment for applications and services at customer site 11 (illustrated as "Customer 11"), which has one or more customer networks coupled to the data center via service provider network 7. For example, data center 10 may be host infrastructure units, such as network and storage systems, redundant power supplies, and environmental controls. Service provider network 7 is coupled to public network 15, which may represent one or more networks managed by other providers and thus may form part of a large-scale public network infrastructure (e.g., the Internet). Public network 15 may represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider operating service provider network 7, an enterprise IP network, or some combination thereof.

[0025] Although customer site 11 and public network 15 are primarily illustrated and described as edge networks of service provider network 7, in some examples, one or more of customer site 11 and public network 15 may be tenant networks within data center 10 or another data center. For example, data center 10 may host multiple tenants (customers), each associated with one or more virtual private networks (VPNs), each VPN implementing a customer site 11.

[0026] Service provider network 7 provides packet-based connectivity to connected customer sites 11, data centers 10, and public networks 15. Service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. Service provider network 7 may implement Multiprotocol Label Switching (MPLS) forwarding, and in this case, may be referred to as an MPLS network or MPLS backbone. In some cases, service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, which provides services from one or more service providers.

[0027] In some examples, data center 10 can represent one of many geographically distributed network data centers. For example... Figure 1As illustrated, data center 10 may be a facility providing network services to customer 11. The service provider's customers may be collective entities, such as businesses and governments or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although illustrated as a separate edge network of service provider network 7, elements of data center 10 (e.g., one or more physical network functions (PNFs) or virtual network functions (VNFs)) may be included in the core of service provider network 7.

[0028] In this example, data center 10 includes storage and / or computing servers interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers. Servers 12A-12X (referred to herein as "server 12") are depicted as coupled to top-of-rack switches 16A-16N. Server 12 is a computing device and may also be referred to herein as a "host" or "host device." Although... Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10 may include many additional servers coupled to other TOR switches 16 of data center 10.

[0029] The switch architecture 14 in the illustrated example includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively referred to as "TOR switches 16"), which are coupled to a distribution layer of chassis (or "backbone" or "core") switches 18A-18M (collectively referred to as "rack switches 18"). Although not shown, data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Data center 10 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), evolved packet cores or other cellular network elements, and other PNFs.

[0030] In this example, TOR switch 16 and rack switch 18 provide redundant (multi-homed) connectivity to server 12 to IP fabric 20 and service provider network 7. Rack switch 18 aggregates traffic flows and provides connectivity between TOR switches 16. TOR switch 16 may be a network device providing Layer 2 (e.g., MAC) and / or Layer 3 (e.g., IP) routing and / or switching capabilities. Both TOR switch 16 and rack switch 18 may include one or more processors and memory, and may execute one or more software processes. Rack switch 18 is coupled to IP fabric 20, which may perform Layer 3 routing to route network traffic between data center 10 and customer site 11 via service provider network 7. The switching architecture of data center 10 is merely an example. Other switching architectures may have more or fewer switching layers, for example.

[0031] The terms "packet flow," "traffic flow," or simply "flow" refer to a group of packets originating from a specific source device or endpoint and destined for a specific destination device or endpoint. A single packet flow can be identified by a 5-tuple, such as: <source network address, destination network address, source port, destination port, protocol>. This 5-tuple typically identifies the packet flow corresponding to a received packet. An n-tuple refers to any n items extracted from the 5-tuple. For example, a packet's binary tuple can refer to a combination of <source network address, destination network address> or <source network address, source port>.

[0032] Server 12 may each represent a compute server, switch, or storage server. For example, each server 12 may represent a computing device, such as an x86 processor-based server, configured to operate according to the techniques described herein. Server 12 may provide Network Functions Virtualization Infrastructure (NFVI) for an NFV architecture.

[0033] Any server in Server 12 can configure virtual execution elements using the resources of the virtualization server to provide isolation between one or more processes (applications) running on the server. "Hypervisor-based," "hardware-level," or "platform" virtualization refers to the creation of virtual machines, each containing a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualization / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is virtualized from the host server's physical hardware, the executing application is isolated from the host hardware and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communication on the corresponding virtual network.

[0034] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used as an alternative to VLAN-based isolation and to provide multi-tenancy in a virtualized data center (e.g., Data Center 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly permitted by security policies.

[0035] Virtual networks can use the edge routers of data center 10 ( Figure 1 (Not shown) Connects to and extends on top of Physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3VPNs) and Ethernet Virtual Private Networks (EVPNs). Virtual networks can also be used to implement Network Functions Virtualization (NFV) and service links.

[0036] Virtual networks can be implemented using various mechanisms. For example, each virtual network can be implemented as a Virtual Local Area Network (VLAN), a Virtual Private Network (VPN), etc. Virtual networks can also be implemented using two networks—a physical underlying network consisting of IP structure 20 and switching structure 14, and a virtual overlay network. The role of the physical underlying network is to provide the "IP structure," which provides unicast IP connections from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified, low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0037] As further described below regarding the virtual router (vROUTER) 21A, the virtual router running in the kernel or hypervisor of the virtual server 12 creates a virtual overlay network over the physical underlying network using a mesh of dynamic "tunnels" between them. For example, these overlay tunnels could be GRE / UDP tunnels, VXLAN tunnels, or MPLS over NVGRE tunnels. The underlying physical routers and switches may not contain any per-tenant state, such as any Media Access Control (MAC) addresses, IP addresses, or policies for virtual machines or other virtual execution elements. The forwarding tables of the underlying physical routers and switches may only contain the IP prefix or MAC address of the physical server 12. (The gateway router or switch connecting the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0038] The virtual router 21 of server 12 does indeed contain the state of each tenant. The virtual router contains a separate forwarding table (routing instance) for each virtual network. This forwarding table contains the IP prefixes (in the case of Layer 3 overlay) or MAC addresses (in the case of Layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., container bays). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses of all virtual machines throughout the entire data center. A given virtual router 21 only needs to contain those routing instances that are locally present on server 12 (i.e., at least one virtual execution element exists on server 12).

[0039] The control plane protocol between the control plane nodes of network controller 24 or the physical gateway router (or switch) can be BGP (and can also be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3VPN and MPLS EVPN. For example, the protocol between network controller 24 and virtual router 21 can be based on XMPP.

[0040] —Container-based “or” operating system virtualization refers to the virtualization of the operating system to run multiple isolated systems on a single machine (virtual or physical). These isolated systems are containers, such as those provided by the open-source DOCKER container application or CoreOSRkt (“Rocket”). Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit a separate operating system, providing only an application suite and dedicated libraries. Typically, containers are executed by the host as isolated user-space instances and can share the operating system and common libraries with other containers running on the host. Therefore, containers may require fewer processing power, storage, and network resources compared to virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communication on the corresponding virtual network.

[0041] In some examples, containers are managed by their host kernel to allow for the throttling and prioritization of resources (CPU, memory, block I / O, network, etc.) without requiring any virtual machines to be started. In some cases, namespace isolation is used, which allows for a completely isolated view of the operating environment (e.g., a given container), including the process tree, network, user identifier, and mounted filesystems. In some examples, containers can be deployed using Linux Containers (LXC), an operating system-level virtualization approach for running multiple isolated Linux systems (containers) on a single controlling host using a single Linux kernel. LXC is an operating system-level virtualization approach for running multiple isolated Linux systems (containers) on a single controlling host (LXC host). LXC does not use virtual machines (although LXC may be hosted by virtual machines). Instead, LXC uses virtual environments with their own CPU, memory, block I / O, network, and / or other resource spaces. LXC resource control mechanisms are provided by namespaces and cgroups in the Linux kernel on the LXC host. Linux namespaces are different from Kubernetes or other orchestration system namespaces. Further information about containers can be found in Docker's "Docker Overview" at docs.docker.com / engine / understanding-docker, last accessed on July 9, 2016. Other examples of containerization approaches include OpenVZ, FreeBSD jails, AIX workload partitioning, and Solaris containers. Therefore, as used in this article, the term "container" can include not only LXC-style containers but also any one or more virtualization engines, virtual private servers, silos, or jails.

[0042] Server 12 hosts virtual network endpoints for one or more virtual networks that operate on the physical network represented here by IP structure 20 and switching structure 14. Although primarily described in the context of a data center-based switching network, other physical networks, such as service provider network 7, may serve as the underlying layer for one or more virtual networks.

[0043] Each server 12 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint for configuring one or more virtual networks in a physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements sharing a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a group or group of one or more containers (e.g., a bay), or another one or more other virtual execution elements, such as a Layer 3 endpoint of a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide at least a partially independent execution environment for applications. The term "virtual execution element" can also include one or more container bays. Figure 1 As shown, server 12A hosts a virtual network endpoint in the form of a cabin 22A with one or more containers. However, given the hardware resource limitations of server 12, server 12 can execute as many virtual execution elements as possible. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / transmit packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0044] Each server 12 includes at least one network interface card (NIC) 13, and each NIC includes at least one interface for exchanging packets with a TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 may provide one or more virtual hardware components for virtualized I / O (I / O), such as a virtual router 21. The virtual hardware components for I / O may be virtualizations (“physical functions”) of the physical NIC 13. For example, in Single Root I / O Virtualization (SR-IOV) as described in the Peripheral Component Interface Special Interest Group (SR-IOV) specification, the PCIe physical functions of a network interface card (or “network adapter”) are virtualized to present one or more virtual network interfaces as “virtual functions” for use by corresponding endpoints executing on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are examples of virtual hardware components. As another example, one or more servers 12 may implement the paravirtualization framework Virtio, which, for example, is available for the Linux operating system, provides emulated NIC functionality as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement Open vSwitch to perform distributed virtual multi-tier switching between one or more virtual NICs (vNICs) of hosted virtual machines, where such vNICs may also represent a type of virtual hardware component that provides virtual network interfaces to virtual network endpoints. In some cases, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, the virtual hardware component is an SR-IOV virtual function. In some examples, any server in server 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For a Docker implementation of containers hosted by the server, the Linux bridge or other operating system bridge that exchanges packets between containers and is executed on the server may be referred to as a "Docker bridge". The term "virtual router" as used herein may include Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software located on a host device that perform the exchange, bridging, or routing of packets between virtual network endpoints in one or more virtual networks, wherein the virtual network endpoints are hosted by one or more servers 12.

[0045] Any NIC 13 may include an internal device switch to switch data between virtual hardware components associated with the NIC. For example, for an SR-IOV-enabled NIC, the internal device switch may be a Virtual Ethernet Bridge (VEB) to switch between SR-IOV virtual functions, and correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. The internal device switch may also be referred to as a network interface card switch, or, for an SR-IOV implementation, an SR-IOV NIC switch. Virtual hardware components associated with the NIC 13A may be associated with a Layer 2 destination address, which may be assigned by the NIC 13A or the software process responsible for configuring the NIC 13A. Physical hardware components (or the “physical functions” of the SR-IOV implementation) are also associated with a Layer 2 destination address.

[0046] To exchange data between virtual hardware components associated with the NIC 13A, the internal device switch can perform Layer 2 forwarding to exchange or bridge Layer 2 packets between the virtual and physical hardware components of the NIC 13A. Each virtual hardware component can reside on a virtual local area network (VLAN) of a virtual network, serving as a virtual network endpoint that uses the virtual hardware component for I / O.

[0047] One or more servers 12 may each include a virtual router 21, which performs one or more routing instances for a corresponding virtual network within data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a Virtual Router and Forward Instance (VRF) for Internet Protocol-Virtual Private Network (IP-VPN). Packets received by the virtual router 21A of server 12A (illustrated as "vROUTER21A") from the underlying physical network structure of data center 10 (i.e., IP structure 20 and switching structure 14) may include an external header to allow the physical network structure to tunnel the payload or "internal packet" to the physical network address of the network interface card 13A of server 12A performing the virtual router. The external header may include not only the physical network address of the server's network interface card 13A, but may also include a virtual network identifier, such as a VxLAN label or Multiprotocol Label Switching (MPLS) label identifying a virtual network, and the corresponding routing instance performed by the virtual router 21A. Internal packets include an internal header with a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0048] Virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network receiving the packet based on the packet's tunnel encapsulation header, then forwards the packet to the appropriate destination virtual network endpoint. For server 12A, for example, for each packet output from a virtual network endpoint hosted by server 12A (e.g., cabin 22A), virtual router 21A attaches a tunnel encapsulation header indicating the virtual network of the packet to generate an encapsulated or "tunneled" packet, and virtual router 21A outputs the encapsulated packet via the virtual network overlay tunnel to a physical destination computing device, such as another of servers 12. As used herein, virtual router 21 can perform the operations of a tunnel endpoint to encapsulate internal packets initiated by a virtual network endpoint to generate tunnel packets and decapsulate tunnel packets to obtain internal packets for routing to other virtual network endpoints.

[0049] The compute infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements on server 12 to provide virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration platform that provides container-centric infrastructure for automating the deployment, scaling, and operation of containers. In the context of virtualized compute infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements by host servers available to the orchestration platform. Specifically, container orchestration allows for container coordination and refers to, for example, deploying, managing, scaling, and configuring containers to host servers via a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker Swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0050] The components of the automated platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. Virtual execution elements can be deployed to a virtualized computing environment using a cluster-based framework. In this framework, the cluster master node manages the deployment and operation of containers to one or more cluster slave nodes. The terms "master node" and "slave node" used herein include different orchestration platform terms used for similar devices, distinguishing between the primary management element of a cluster and the primary virtual execution element hosting device. For example, the Kubernetes platform uses the terms "cluster master node" and "slave node," while the Docker Swarm platform refers to both the cluster manager and the cluster nodes.

[0051] Orchestrator 23 and network controller 24 together implement controller 5 of computing infrastructure 8. Orchestrator 23 and network controller 24 can execute on separate computing devices or on the same computing device. Each of orchestrator 23 and network controller 24 can be a distributed application executing on one or more computing devices. Orchestrator 23 and network controller 24 can implement corresponding master nodes for one or more clusters, each cluster having one or more slave nodes implemented by corresponding servers 12. Typically, network controller 24 controls the network configuration of the data center 10 structure to, for example, establish one or more virtual networks for packet communication between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10. In some examples, network controller 24 can operate in response to configuration input received from orchestrator 23 and / or administrator / operator. Additional information regarding network controller 24 operating with other devices or other software-defined networks in data center 10 can be found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled "Physical Path Determination for Virtual Network Packet Flows," and U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled "Tunneled Packet Aggregation for Virtual Networks," both of which are incorporated herein by reference as if fully set forth herein. U.S. Patent Application No. 14 / 226,509 also includes further description of virtual routers, such as virtual router 21A.

[0052] Typically, orchestrator 23 controls the deployment, scaling, and operation of virtual execution elements across the server cluster 12 and provides the computing infrastructure, which may include container-centric computing infrastructure. Orchestrator 23, and in some cases network controller 24, can implement corresponding cluster master nodes for one or more Kubernetes clusters. Kubernetes is a container management platform that provides portability across public and private clouds, each of which can provide virtualization infrastructure for the container management platform.

[0053] Kubernetes operates using various "objects"—entities that represent a state of a Kubernetes cluster. Kubernetes objects can include any combination of names, namespaces, tags, annotations, field selectors, and recommended tags. For example, a Kubernetes cluster can include one or more "namespace" objects. In some cases, each namespace in a Kubernetes cluster is isolated from the other namespaces in the cluster. Namespace objects can improve at least one of the organization, security, and performance of a Kubernetes cluster. As an example, a cabin can be associated with a namespace, thus associating the cabin with characteristics of the namespace (e.g., virtual networks). This characteristic allows multiple newly created cabins to be organized by associating them with a common set of characteristics. Namespaces can be created based on namespace specification data that defines the characteristics of the namespace, including the namespace name. In one example, the namespace is named "My Namespace," and each newly created cabin can be associated with a set of characteristics represented by "My Namespace." Additionally, Kubernetes includes a "default" namespace. If a newly created cabin does not specify a namespace, it may be associated with characteristics of the "default" namespace.

[0054] Namespaces allow a Kubernetes cluster to be used by multiple users, user teams, or a single user with multiple applications. Furthermore, each user, user team, or application can be isolated from every other user in the cluster within a namespace. Therefore, each user of the Kubernetes cluster within a namespace operates as if they were the only user of the Kubernetes cluster. The techniques disclosed herein include the ability to associate multiple virtual networks with a single namespace. This allows users within a given namespace to access each virtual network associated with that namespace, including virtual execution elements that act as virtual network endpoints of a group of virtual networks.

[0055] The virtualized computing infrastructure managed by orchestrator 23 and network controller 24 can communicate according to one of several modes. In the default mode, all modules can communicate with all other modules without using Network Address Translation (NAT). Network controller 23 creates a virtual network shared by all namespaces, from which service and module IP addresses are assigned. All modules in all namespaces generated in the cluster are able to communicate with each other. The module subnets configured from orchestrator 23 are assigned IP addresses to all modules.

[0056] In namespace isolation mode, cluster administrators can configure namespace annotations to enable isolation. Therefore, services within that namespace cannot access it from other namespaces unless explicitly allowed by a security group or network policy. Namespaces can be configured to be isolated by annotating namespace canonical data with directives. For example:

[0057] Isolation: Real

[0058] Namespaces labeled "Isolation" and "Service Isolation" exhibit the following network behaviors:

[0059] • All pods created in isolated namespaces are network reachable from each other;

[0060] • Modules in other namespaces within a cluster cannot reach modules in an isolated namespace;

[0061] • Containers from other namespaces cannot reach services in isolated namespaces.

[0062] In custom isolation mode, administrators and application developers can add annotations to specify a virtual network within which a namespace is provided, or all of the cabins. An example annotation for specifying this custom virtual network is:

[0063] network:<fq_network_name>

[0064] If this annotation is configured on the cabin specification (cabin configuration data), the cabin will launch in the named network. If the annotation is configured on the namespace specification (namespace specification data), all cabins in the namespace will launch in the provided network. In some examples, network controller 24 must configure virtual networks in compute infrastructure 8 before virtual network names are available for annotating cabins or namespaces. In some examples, network controller 24 may configure additional virtual networks in compute infrastructure 8 as part of the deployment of virtual execution elements that will use the additional virtual networks.

[0065] The network model of the cabin with the above pattern can be implemented using different systems and combinations of systems, such as Contrail and Tungsten Fabric.

[0066] exist Figure 1In the example shown, orchestrator 23 accepts namespace specification data 27 as input. Namespace specification data 27 can be encoded using a human-readable data serialization language, such as YAML (Yin't Markup Language), JavaScript Object Notation (JSON), etc. Namespace specification data 27 can represent either "namespace specification" or "namespace spec."

[0067] According to the techniques described in this disclosure, the namespace specification data 27 for specifying a namespace includes a list of multiple virtual networks for the namespace, and more specifically, a list of multiple virtual networks for communication between virtual execution elements deployed to the namespace.

[0068] In some examples, namespace specification data 27 specifies a list of multiple virtual networks corresponding to virtual networks configured by network controller 24 in network system 2. Virtual networks can be specified using any virtual network identifier, such as a name, UUID, or code. Below is an example of namespace specification data 27 for the namespace "admin":

[0069]

[0070] In the example above, network controller 24 specifies the mode "Provider / Network" used to designate each virtual network. The four virtual networks are designated by virtual network names: "Management Network," "Development Network," "Quality Assurance Network," and "Human Resources Network."

[0071] Other examples of namespace canonical data 27 can use different patterns to specify virtual networks. For example, here is another example from namespace canonical data 27:

[0072]

[0073] The example above specifies four virtual networks using Kubernetes Network (CustomResourceDefinition) objects to make the virtual network specification conform to the Kubernetes Custom Resource Definition De-facto Standard, which specifies the requirements and procedures for connecting Kubernetes cabins to one or more virtual or physical networks, including plugin requirements for connecting cabin networks using the Container Network Interface (CNI).

[0074] Orchestrator 23 can process namespace specification data 27 to generate configuration objects corresponding to the respective namespaces. These configuration objects include corresponding virtual network configuration objects for multiple virtual networks specified for the namespace. In the example where orchestrator 23 implements a corresponding cluster master node for one or more Kubernetes clusters, the namespace represents a virtual cluster within the Kubernetes cluster.

[0075] The virtual networks specified in namespace specification data 27 can represent a primary virtual network group, where each virtual network is within the primary virtual network group. In request 29, orchestrator 23 passes a virtual network configuration object to network controller 24 to request the namespace to deploy the virtual network interfaces of the virtual execution elements in server 12.

[0076] Multiple virtual networks can be configured by network controller 24 independently of orchestrator 23. In this way, orchestrator 23 can operate as a "through" station for namespace specification data 27; for example, orchestrator 23 generates records of namespaces in namespace specification data 27 and then transmits virtual network configuration objects or other virtual network configuration data collected from namespace specification data 27 to network controller 24 for processing. Although network controller 24 can configure multiple virtual networks specified by namespace specification data 27, orchestrator 23 is the architect of virtual execution elements (e.g., cabins, containers, virtual machines, etc.) instantiated as virtual network endpoints of multiple virtual networks.

[0077] Cabin 22A is a Kubernetes cabin, an example of a virtual network endpoint. A cabin is a group of one or more logically related containers. Figure 1 (Not shown in the diagram) Container shared storage and options for how containers run. When instantiated for execution, a container can also be referred to as a "container copy". Each container in container 22A is an example of a virtual execution element. Containers within a container always reside on the same server, are co-scheduled, and run in a shared context. The shared context of a container can be a set of Linux namespaces, cgroups, and other aspects of isolation. Within the container's context, individual applications may apply further sub-isolation. Typically, containers within a container have a common IP address and port space and are able to detect each other via the localhost. Because containers have a shared context, containers within a container can also communicate with each other using inter-process communication (IPC). Examples of IPC include System V semaphores or POSIX shared memory. Typically, containers belonging to different containers have different IP addresses and cannot communicate via IPC without a configuration that enables this feature. Containers belonging to different containers typically communicate with each other via container IP addresses.

[0078] Orchestrator 23 manages the deployment of virtual execution elements to server 12. Virtual execution element specification data ( Figure 1 (Not shown) may include specifications for corresponding virtual execution elements, such as modules. For example, virtual execution element specification data may be included in a human-readable file containing information for deploying module 22A within computing infrastructure 8. More specifically, the human-readable file corresponding to module 22A may be encoded in YAML or JSON. The virtual execution element specification data may include namespace objects specifying the namespace associated with module 22A, such as the namespace indicated in namespace specification data 27. The namespace represents the primary virtual network group of module 22A. Additionally or alternatively, the virtual execution element specification data of module 22A may include network annotations specifying secondary virtual network groups, wherein at least one virtual network of the secondary virtual network group is not specified by the namespace of module 22A. The primary and secondary virtual network groups together constitute a group of virtual networks, wherein module 22A will be configured with corresponding virtual network interfaces.

[0079] Orchestrator 23 obtains virtual execution element specification data corresponding to the virtual execution element, wherein the virtual execution element specification data specifies a set of virtual networks (e.g., a secondary virtual network group). Orchestrator 23 can process the virtual execution element specification data to generate configuration objects corresponding to the respective namespaces, wherein the configuration objects include corresponding virtual network configuration objects for the multiple virtual networks specified for the virtual execution element.

[0080] In some examples, to specify the virtual network that a cabin should connect to, the virtual execution element specification data corresponding to the cabin may include network annotations to specify a particular virtual network. Virtual execution element specification data can be included within the cabin specification. The virtual execution element specification data may specify the cabin name, at least one namespace, or other objects. Network annotations may include cabin object annotations to indicate the virtual network to which the cabin will be instantiated. Annotations may include one or more "NetworkAttachmentDefinition" objects that specify one or more virtual networks to which the cabin will be instantiated. The following JSON data provides an example human-readable file corresponding to a cabin:

[0081] kind:Pod

[0082] metadata:

[0083] name:my-pod

[0084] namespace:my-namespace

[0085] annotations:

[0086] k8s.v1.cni.cncf.io / networks:net-a,net-b,other-ns / net-c

[0087] As shown in the data above, the data specifies an object: a pod. This pod is named "my-pod" and its namespace is "my-namespace". The annotation specifies three virtual networks: "net-a", "net-b", and "net-c". In this example, "net-a" and "net-b" are located within "my-namespace", while "net-c" is located within another namespace, "other-ns". Thus, the virtual execution element specification data corresponding to the container can annotate a list of virtual networks that the pod will place within, and this list of virtual networks can include virtual networks within the pod's namespace or another namespace.

[0088] The virtual actuator specification data corresponding to a cabin may also include cabin-specific requirements for connecting to one or more virtual networks, such as specific IP addresses and Media Access Control (MAC) addresses associated with at least one virtual network specified in the comments. For example, the virtual actuator specification data may be given as the following JSON data:

[0089]

[0090] JSON keys can be defined for network connection mappings. For example, "name", "namespace", and "ips" represent JSON keys. The "name" key is the name of the NetworkAttachmentDefinition object, located in the cabin's namespace (if the "namespace" key is missing or empty) or another namespace specified by the "namespace" key. In some examples, the "namespace" key defines a value of type string and gives the namespace of the NetworkAttachmentDefinition object named by the "name" key. The "ips" key is a value of type string array. Additionally, the "IP" key instructs the plugin to handle network connections to assign a given IP address to the cabin. The value of the IPS key must contain at least one array element, and each element must be a valid IPv4 or IPv6 address. If this value is invalid, the network connection selection annotation will be invalid and ignored by the CNI plugin during the cabin's implementation. The following JSON snippet gives an example of an "ips" key:

[0091]

[0092] CNI delegation plugins are implementations that delegate container network connection / deconnection operations to other plugins (e.g., network module 17A) that conform to the CNI specification. In other words, CNI delegation plugins enable the transfer of information between nodes in a virtual network, allowing CNI plugins to connect and deconnect containers from the compute infrastructure. In some examples, CNI delegation plugins must add an "ips" key to the CNI "args" mapping and set its value to a transformation of the "ips" key value. In container implementations, "ips" may be favored, but CNI "args" may be ignored by the plugin. Therefore, this implementation ensures that the requested IP address is assigned to the network connection interface in the returned CNI request structure. If the requested IP address is not assigned, this implementation may result in network connection failure. Given the example annotation above, which includes the "ips" key for calling "network-b", the CNI delegation plugin is configured to transform data into the following JSON fragment for each plugin (e.g., each slave node of the virtual network named "network-b") in the CNI call passed to "network-b":

[0093]

[0094] The pod's JSON comment may include a "mac" key, which represents the MAC address to be assigned to the pod. The MAC key value may contain a valid 6-byte Ethernet MAC address or a valid 20-byte IP-over-InfiniBand hardware address. If the "mac" key value is invalid, the network connection selection comment may be determined to be invalid and ignored during the pod's implementation. An example network connection selection comment is given for the pod, specifying the virtual network "network-b" and additionally specifying the "mac" key:

[0095]

[0096] In some examples, the "mac" key is added to the CNI "args" mapping. The CNI specification may require implementations to adhere to "mac". However, the CNI specification may state that the CNI "args" might be ignored by CNI plugins. Therefore, this implementation ensures that the requested MAC address is assigned to the network connection interface in the returned CNI request structure; if it is not assigned, this implementation may cause the network connection to fail. The JSON snippet below represents the "mac" key added to the "args" mapping.

[0097]

[0098] In some examples, a secondary virtual network group includes at least one virtual network that is also included in a primary virtual network group, resulting in overlap between the virtual network groups. Since the primary and secondary virtual network groups define a combined virtual network group, where orchestrator 23 deploys cabin 22A, in some examples, orchestrator 23 can deploy cabin 22A to each virtual network in the combined virtual network group, including virtual networks overlapping between the primary and secondary virtual network groups. Alternatively, in other examples, the network annotation for cabin 22A can specify at least one overlapping virtual network that is also included in the primary virtual network group, and the network annotation can include instructions to abandon the deployment of cabin 22A to at least one overlapping virtual network. In other words, the combined virtual network group can exclude the deployment of cabin 22A from at least one virtual network in the primary virtual network group by including instructions in the network annotation for cabin 22A.

[0099] In other words, the virtual execution element specification data may include a namespace object specifying the namespace representing the primary virtual network group and network annotations specifying the secondary virtual network groups. Network controller 24 is configured to parse the virtual execution element specification data to create secondary virtual network groups in compute infrastructure 8. Additionally, orchestrator 23 is configured to deploy virtual execution elements to each virtual network within a secondary virtual network group. The secondary virtual network groups specified by the network annotations may include at least one of a first group of virtual networks not included in the primary virtual network group and a second group of virtual networks included in the primary virtual network group. The network annotations may instruct orchestrator 32 to avoid deploying virtual execution elements to at least one virtual network in the second group of virtual networks.

[0100] In some examples, when orchestrator 32 deploys bay 22A, a secondary virtual network group is established within computing infrastructure 8. Alternatively, in other examples, at least one of the secondary virtual network groups is not established within computing infrastructure 8 at the start of bay 22A. Thus, when bay 22A is deployed, network controller 24 can identify at least one of the secondary virtual network groups that was not established within computing infrastructure 8 at the start of bay 22A, and subsequently create the virtual network that has not yet been established. Network controller 24 creates the secondary virtual network group within computing infrastructure 8 based on network configuration information obtained from parsing virtual execution element specification data.

[0101] Server 12A includes a container platform 19A for running containerized applications, such as the application in container 22A. Container platform 19A receives requests from orchestrator 23 to obtain and host containers on server 12A. Container platform 19A obtains and executes the containers.

[0102] Container platform 19A includes a network module 17A that configures virtual network interfaces for virtual network endpoints. Container platform 19A uses network module 17A to manage networking of cabins (including cabin 22A). For example, network module 17A creates virtual network interfaces to connect cabins to virtual router 21A and enables containers in such cabins to communicate with other virtual network endpoints via the virtual network interfaces. Network module 17A can, for example, insert the virtual network interfaces of a virtual network into the network namespace of containers in cabin 22A, and configure (or request configuration) the virtual network interfaces of the virtual network in virtual router 21A such that virtual router 21A is configured to send packets received from the virtual network via the virtual network interfaces to containers in cabin 22A, and to send packets received from cabin 22A via the virtual network interfaces. Network module 17A can assign network addresses (e.g., virtual IP addresses for the virtual network) and can set routes for the virtual network interfaces. In Kubernetes, all cabins can communicate with all other cabins without the need for Network Address Translation (NAT). In some cases, orchestrator 23 and network controller 24 create a virtual network shared by all namespaces, from which service and container network addresses are allocated. This shared virtual network may be referred to as the service virtual network. In some cases, all containers in all namespaces generated in a Kubernetes cluster may be able to communicate with each other, and network addresses for all containers can be allocated from the container subnets specified by orchestrator 23. In addition to container container addressing, network module 17A can also support configuring network isolation, policy-based security, gateways, static network address translation (SNAT), load balancers, and service link capabilities for orchestration.

[0103] Network module 17A can represent libraries, plugins, modules, or other runtime executable code of server 12A. Network module 17A can at least partially conform to the Container Network Interface (CNI) specification or the rkt network proposal. Network module 17A can represent a Contrail, OpenContrail, or Tungsten architecture network plugin. Network module 17A can also be called a network plugin or a CNI plugin. For the purposes of the CNI specification, a container can be considered a synonym for the Linux network namespace. Which unit corresponds depends on the specific implementation of the container runtime: for example, in an implementation of an application container specification (e.g., rkt), each container runs in a unique network namespace. However, in Docker, the network namespace typically exists within each individual Docker container. For the purposes of the CNI specification, a network refers to a set of uniquely addressable entities that can communicate with each other. This can be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can conceptually be added to or removed from one or more networks.

[0104] The CNI specification outlines several considerations for conforming plugins (“CNI plugins”). These include the following:

[0105] Before invoking any CNI plugins, the container runtime must create a new network namespace for the container;

[0106] The container runtime must determine which network the container should belong to, and for each network, which plugins must be executed;

[0107] The container runtime must add the container to each network by executing the corresponding plugin for each network in turn.

[0108] In some examples, a single network module 17A configures multiple virtual network interfaces 26A-26N (“virtual network interfaces”) for compartment 22A for corresponding virtual networks configured in switching structure 14. Thus, as further described below, network module 17A addresses certain limitations of CNI plugins that strictly conform to the CNI specification. Virtual network interfaces 26 can be interfaces to multiple virtual networks specified in namespace specification data 27 (major group), interfaces to one or more virtual networks specified in the virtual execution element specification data of compartment 22A (minor group), or interfaces to combinations of such virtual networks (combination groups).

[0109] Each virtual network interface 26 may represent a virtual Ethernet (“veth”) pair, wherein each end of the pair is a separate device (e.g., a Linux / Unix device), one end of the pair is assigned to cabin 22A, and the other end is assigned to virtual router 21A. A veth pair, or the ends of a veth pair, is sometimes referred to as a “port”. Each virtual network interface 26 may alternatively represent a macvlan network with Media Access Control (MAC) addresses assigned to cabin 22A and virtual router 21A for communication between cabin 22A and virtual router 21A. Each virtual network interface 26 may alternatively represent a different type of interface between virtual router 21A or other network virtual entities and virtual network endpoints. For example, virtual network interface 26 may alternatively be referred to as a Virtual Machine Interface (VMI), cabin interface, container network interface, tap interface, veth interface, or simple network interface (in a specific context).

[0110] exist Figure 1 In example server 12A, container 22A is a virtual network endpoint in multiple different virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment that specifies multiple virtual networks and designates container 22A (or one or more containers therein) as a virtual network endpoint in each of the multiple virtual networks. For example, orchestrator 23 may receive configuration data from users, operators / administrators, or other machine systems.

[0111] As part of the process of creating Module 22A, orchestrator 23 sends request 29 to request network controller 24 to create corresponding virtual network interfaces (indicated in the configuration data) for multiple virtual networks. Orchestrator 23 may store, send, or otherwise notify network controller 24 of the virtual network configuration objects among the multiple virtual networks specified for Module 22A. Network controller 24 may configure any virtual networks that have not yet been configured in computing infrastructure 8.

[0112] Network controller 24 processes request 29 to generate interface configuration data 25 for multiple virtual network interfaces 26 of cabin 22A, for communication via virtual networks indicated in namespace specification data 27, virtual execution element specification data, or both. Interface configuration data 25 may include a container or cabin unique identifier and a list or other data structure specifying network configuration data for configuring each virtual network interface 26. The network configuration data for a virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. An example of JSON-formatted network configuration data is shown below. Multiple virtual network interfaces 26 correspond to multiple virtual networks. Network controller 24 sends interface configuration data 25 to server 12A, and more specifically, in some cases, to virtual router 21A. To configure one or more virtual network interfaces for cabin 22A, container platform 19A invokes a single instance of network module 17A. Network module 17A obtains and processes interface configuration data 25. For each virtual network interface specified in interface configuration data 25, network module 17A creates a virtual network interface 26. For example, network module 17A can connect one end of a veth pair implementing virtual network interface 26A to virtual router 21A, and the other end of the same veth pair to cabin 22A. Similarly, network module 17A can connect one end of a veth pair implementing virtual network interface 26N to virtual router 21A, and the other end of the same veth pair to cabin 22A. In this way, a single instance of network module 17A configures multiple virtual network interfaces 26 for one or more virtual execution elements (cabin 22A in this case) sharing at least one virtual network interface.

[0113] The following is sample network configuration data for cabin 22A with a single virtual network interface 26A:

[0114]

[0115] The network configuration data for cabin 22A may include multiple annotations similar to those provided above for multiple different virtual network interfaces 26.

[0116] Traditional CNI plugins are invoked by the container platform / runtime, receiving add commands from the container platform to add containers to a single virtual network. These plugins are then maintained in the server's runtime memory to subsequently receive delete commands from the container / runtime and remove containers from the virtual network. This requires a separate CNI plugin for each virtual network. A single network module 17A, invoked by container platform 19A, extends the functionality of traditional CNI plugins by obtaining interface configuration data 25 and adding multiple different virtual network interfaces 26. The term "invocation" can refer to the instantiation of a software component or module in memory (e.g., user space 245) executed by microprocessor 210 as executable code. However, in some cases, multiple instances of network module 17A can be invoked to configure the virtual network interfaces 26.

[0117] These technologies can provide one or more technical advantages. For example, by specifying multiple virtual networks in the namespace specification data 27, a cabin 22A deployed to the corresponding namespace will be configured to communicate via any of the multiple virtual networks. As another example, by simply including annotations specifying virtual networks in the namespace of the virtual execution element, the virtual execution element can be effectively distributed across a primary virtual network group. Previous methods of creating namespaces limited each namespace configuration to specifying only one virtual network, which restricted the types of applications that could be executed in the virtual execution element. As another example, because the configuration data of the virtual execution element can specify secondary virtual network groups, virtual execution elements deployed as described herein can be configured with virtual network interfaces for each virtual network in the primary virtual network group specified for the namespace and in the secondary virtual network groups specified for the virtual execution element. This provides users with a high degree of versatility and coarse-grained and fine-grained control over network policies for communication between virtual execution elements deployed to one or more namespaces.

[0118] Figure 2 This is a block diagram of an example computing device according to the technology described in this disclosure, the example computing device including a network module for configuring multiple virtual network interfaces for a group of one or more virtual execution elements sharing at least one virtual network interface. Figure 2 The computing device 200 can represent a real or virtual server, and can represent Figure 1This is an example instance of any server 12. In this example, computing device 200 includes a bus 242 that couples hardware components of the computing device 200 hardware environment. Bus 242 couples a network interface card (NIC) 230, a storage disk 246, and one or more microprocessors 210 (hereinafter referred to as "microprocessor 210"). NIC 230 may support IOV. In some cases, the front-side bus may couple microprocessor 210 and memory 244. In some examples, bus 242 may couple memory 244, microprocessor 210, and NIC 230. Bus 242 may represent a Peripheral Component Interface (PCI) Fast (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 242 control DMA transfers between components coupled to bus 242.

[0119] Microprocessor 210 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, which are stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or “multi-core” processors), each multi-core processor being implemented using a single IC (i.e., a chip multiprocessor).

[0120] Disk 246 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by microprocessor 210.

[0121] Main memory 244 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM) or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media, which may be used to carry or store desired program code and program data in the form of instructions or data structures and may be accessible to a computer. Main memory 244 provides a physical address space consisting of addressable storage locations.

[0122] Network interface card (NIC) 230 includes one or more interfaces 232 configured to switch packets using links of the underlying physical network. Interface 232 may include a port interface card with one or more network ports. NIC 230 may also include on-card memory for, for example, storing packet data. Direct memory access transfers between NIC 230 and other devices coupled to bus 242 can read / write to NIC memory.

[0123] Memory 244, NIC 230, storage disk 246, and microprocessor 210 provide an operating environment for a software stack, which includes an operating system kernel 214 executing in kernel space. Kernel 214 may represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, available from Microsoft. In some cases, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). The operating system including kernel 214 provides an execution environment for one or more processes in user space 245.

[0124] Kernel 214 includes a physical driver 225 that uses network interface card 230. Network interface card 230 can also implement SR-IOV to enable the sharing of physical network functions (I / O) between one or more virtual execution elements, such as containers 229A-229B or one or more virtual machines. Figure 2 (Not shown in the image). Shared virtual devices (e.g., virtual functions) can provide dedicated resources, allowing each virtual execution element to access the dedicated resources of NIC 230. Therefore, NIC 230 appears as a dedicated NIC for each virtual execution element. A virtual function may represent a lightweight PCIe function that shares physical resources with the physical functions used by physical drive 225 and other virtual functions. For an SR-IOV-enabled NIC 230, NIC 230 can have thousands of available virtual functions according to the SR-IOV standard; however, for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0125] Computing device 200 can be coupled to a physical network switching infrastructure, which includes an overlay network that extends the switching infrastructure from physical switches to software or "virtual" routers, including virtual router 220, coupled to physical servers connected to the switching infrastructure. The virtual router can be a physical server (e.g., Figure 1 A server 12) executes a process or thread or component thereof that dynamically creates and manages one or more virtual networks that can be used to communicate between virtual network endpoints. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executing. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various technologies can be used to transmit packets within and over the physical network. As used herein, the term "virtual router" can include an Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other means and / or software residing on a host device and performing the exchange, bridging, or routing of packets between virtual network endpoints of one or more virtual networks hosted by one or more servers 12. Figure 2 In the example computing device 200, the virtual router 220 executes within the kernel 214, but in various implementations, the virtual router 220 may execute within a hypervisor, a host operating system, a host application, or a virtual machine.

[0126] Virtual router 220 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used in Kubernetes deployments like cabin 202. Virtual router 220 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 220 can perform network services such as application security policies, NAT, multicast, mirroring, and load balancing. Further details of IP-VPN are described in Request for Comments 4364 (Internet Engineering Task Force Networking Working Group, February 2006) (hereinafter referred to as "RFC 4364"), which is incorporated herein by reference in its entirety. Virtual router 220 can represent a PE router, and the virtual execution endpoint can be an example of the CE device described in RFC 4364.

[0127] Typically, each compartment 202A-202B (collectively referred to as "Compartment 202") can be assigned one or more virtual network addresses for use in a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by virtual router 220. Compartment 202B can be assigned its own virtual Layer 3 (L3) IP address, for example, for sending and receiving communications, but may not know the IP address of the computing device 200 in which Compartment 202B resides. The virtual network address can therefore be different from the logical address of the underlying physical computer system (e.g., computing device 200).

[0128] Computing device 200 includes a virtual router agent 216 that controls the coverage of virtual networks within computing device 200 and coordinates data packet routing within computing device 200. Typically, virtual router agent 216 communicates with network controller 24 of the virtualization infrastructure, which generates commands to control the creation of virtual networks and the configuration of network virtualization endpoints, such as computing device 200, and more specifically, virtual router 220 and virtual network interfaces 212, 213.

[0129] In one example, network packets (e.g., Layer 3 (L3) IP packets or Layer 2 (L2) Ethernet packets) generated or used by containers 229A-229B within a virtual network domain can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted over the physical network. Packets transmitted within the virtual network may be referred to herein as “internal packets,” while physical network packets may be referred to herein as “external packets” or “tunneled packets.” Virtual router 220 can perform encapsulation and / or decapsulation of virtual network packets within physical network packets. This function is referred to herein as tunneling and can be used to create one or more overlay networks. Other example tunneling protocols that can be used, besides IP-in-IP, include IP over Generic Routing Encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), etc. Virtual router 220 performs tunnel encapsulation / decapsulation on packets originating from / to container 202, and virtual router 220 exchanges packets with container 202 via bridges on bus 242 and / or NIC 230.

[0130] As described above, network controller 24 can provide a logically centralized controller to facilitate the operation of one or more virtual networks. Network controller 24 may, for example, maintain a routing information base, such as one or more routing tables storing routing information for the physical network and one or more overlay networks. Virtual router 220 implements one or more Virtual Router and Forwarding Instances (VRFs) 222A-222B for the corresponding virtual network, and operates on them as corresponding tunnel endpoints. Typically, each VRF 222 stores forwarding information for the corresponding virtual network and identifies where packets will be forwarded and whether packets will be encapsulated in a tunneling protocol, for example, with a tunnel header that may include one or more headers from different layers of the virtual network protocol stack. Each VRF 222 may include a network forwarding table storing routing and forwarding information for the virtual network.

[0131] NIC 230 can receive tunnel packets. Virtual router 220 processes tunnel packets to determine the virtual networks of the source and destination endpoints of the internal packets from the tunnel encapsulation header. Virtual router 220 can strip the Layer 2 header and tunnel encapsulation header to forward internal packets only internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN label or MPLS label, indicating the virtual network, for example, the virtual network corresponding to VRF 222A. VRF 222A may include forwarding information for the internal packets. For example, VRF 222A may map the destination Layer 3 address of the internal packets to virtual network interface 212A. In response, VRF 222A forwards the internal packets to compartment 202A via virtual network interface 212A.

[0132] Containers 229A–229B can also be sources of internal packets as source virtual network endpoints. For example, container 229A can generate Layer 3 internal packets destined for a destination virtual network endpoint executed by another computing device (i.e., non-computing device 200) or another of containers 229A–229B. Container 229A sends Layer 3 internal packets to virtual router 220 via virtual network interface 212A connected to VRF 222A.

[0133] Virtual router 220 receives internal packets and Layer 2 headers, and determines the virtual network for the internal packets. Virtual router 220 can use any of the aforementioned virtual network interface implementation technologies (e.g., macvlan, veth, etc.) to determine the virtual network. Virtual router 220 uses VRF 222A corresponding to the virtual network of the internal packet to generate an outer header for the internal packet. The outer header includes an outer IP header overlaying the tunnel and a tunnel encapsulation header identifying the virtual network. Virtual router 220 encapsulates the internal packet with the outer header. Virtual router 220 can encapsulate the tunnel packet with a new Layer 2 header having a destination Layer 2 address associated with a device outside computing device 200 (e.g., TOR switch 16 or a server 12). If outside computing device 200, virtual router 220 outputs the tunnel packet with the new Layer 2 header to NIC 230 using physical function 221. NIC 230 outputs the packet on its output interface. If the destination is another virtual network endpoint executed on computing device 200, then virtual router 220 will route the packet to the appropriate one of virtual network interfaces 212, 213.

[0134] In some examples, the controller of computing device 200 (e.g., Figure 1 The network controller 24) configures a default route in each compartment 202 such that the virtual machine 224 uses the virtual router 220 as the initial next hop for outgoing packets. In some examples, the NIC 230 is configured with one or more forwarding rules such that all packets received from the virtual machine 224 are switched to the virtual router 220.

[0135] Cabins 202A-202B can be described in more detail. Figure 1 Example instance of compartment 22A. Compartment 202A includes one or more containers 229A, and compartment 202B includes one or more containers 229B.

[0136] The container platform 204 can be described in more detail. Figure 1 An example instance of container platform 19A. Container platform 204 includes container engine 208, orchestration agent 209, service agent 211, and network modules 206A-206B. Each network module 206A-206B can represent... Figure 1 Example instance of network module 17A. In some cases, one network module 206 is invoked for each cabin 202. However, in some cases, additional instances of network module 206 are invoked for different virtual network interfaces 212, 213, and / or network module 206 can configure multiple virtual network interfaces 212, 213 for multiple cabins.

[0137] Container engine 208 includes code executable by microprocessor 210. Container engine 208 can be one or more computer processes. Container engine 208 runs containerized applications in the form of containers 229A–229B. Container engine 208 can represent Dockert, rkt, or other container engines used to manage containers. Typically, container engine 208 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions to create a container. A container is an executable instance of an image. Based on instructions from orchestration agent 209, container engine 208 can obtain an image and instantiate it into executable containers 229A–229B within containers 202A–202B.

[0138] Service agent 211 includes code executable by microprocessor 210. Service agent 211 can be one or more computer processes. Service agent 211 monitors the addition and removal of services and endpoint objects, and maintains the network configuration of computing device 200 to ensure communication between cabins and containers, for example, using services. Service agent 211 can also manage iptables to capture traffic to the virtual IP address and port of the service and redirect traffic to the proxy port of the proxy backup cabin. Service agent 211 can represent a Kube agent on a slave node of a Kubernetes cluster. In some examples, container platform 204 does not include service agent 211, or disables service agent 211, to support the configuration of virtual router 220 and cabin 202 by network module 206.

[0139] The orchestration agent 209 includes code executable by the microprocessor 210. The orchestration agent 209 can be one or more computer processes. The orchestration agent 209 can represent a kubelet of a slave node in a Kubernetes cluster. The orchestration agent 209 is an agent of the orchestrator, for example... Figure 1 The orchestrator 23 receives container specification data for a container and ensures that the container is executed by the computing device 200. The container specification data may be a manifest file sent from the orchestrator 23 to the orchestration agent 209 or received indirectly via a command-line interface, HTTP endpoint, or HTTP server. The container specification data may be a container specification (e.g., a PodSpec—a YAML (yet another markup language) or JSON object describing a container) for a compartment 202 of container 229. Based on the container specification data, the orchestration agent 209 instructs the container engine 208 to obtain and instantiate a container image of container 229 for execution by the computing device 200. The container specification data is an example of virtual execution element specification data.

[0140] In some examples, orchestrator 23 deploys cabin 202 to user space 245 via orchestration agent 209. More specifically, orchestrator 23 deploys cabin (e.g., cabin 202A) to user space 245 in response to determining that computing device 200 hosting user space 245 is part of at least one virtual network in a list of virtual networks specified by cabin 202A. In some examples, computing device 200 is part of more than one virtual network in a custom list of virtual networks.

[0141] According to the techniques described herein, orchestration agent 209 instantiates at least one of network modules 206 to configure one or more virtual network interfaces for each cabin 202. Each network module 206 can represent Figure 1 An example instance of network module 17A. For example, orchestration agent 209 receives container specification data for cabin 202A and instructs container engine 208 to create cabin 202A with container 229A based on the container specification data of cabin 202A. Orchestration agent 209 also invokes network module 206A to configure multiple virtual network interfaces 212A-212B for cabin 202A, corresponding to virtual networks of VRFs 222A-222B respectively. In a similar manner, orchestration agent 209 instructs container engine 208 to create cabin 202B with container 229B based on the container specification data of cabin 202B. Orchestration agent 209 also invokes a single network module 206B to configure virtual network interface 213 for cabin 202B corresponding to the virtual network of VRF 222B. In this example, cabin 202A and cabin 202B are both virtual network endpoints corresponding to the virtual network of VRF 222B. Either virtual network interface 212, 213 can represent Figure 1 An example instance of a virtual network interface 26 described in the document.

[0142] Network module 206A can obtain interface configuration data for configuring virtual network interfaces for cabin 202. Virtual router agent 216 operates as a virtual network control plane module, enabling network controller 24 to configure virtual router 220. Unlike the orchestration control plane (including container platform 204, e.g., orchestrator 23, for slave and master nodes) which manages the provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 for slave nodes and virtual router agent 216) manages the configuration of the virtual network partially implemented in the data plane by the virtual router 220 of the slave nodes. Virtual router agent 216 transmits interface configuration data of the virtual network interfaces to network module 206, enabling orchestration control plane elements (i.e., network module 206) to configure the virtual network interfaces according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this allows a single network module 206A to obtain interface configuration data for multiple virtual network interfaces of the cabin and configure multiple virtual network interfaces, which can reduce the communication and resource overhead inherent in calling a separate network module 206 to configure each virtual network interface.

[0143] Figure 3 This is a block diagram of an example computing device according to the technology described in this disclosure. The computing device 300 is an example instance of a controller 5 for a cloud computing infrastructure. Figure 3 The computing device 300 can represent one or more physical or virtual servers configured to perform operations on at least one of the network controller 24 and orchestrator 23. Thus, the computing device 300 can, in some cases, implement one or more master nodes for a corresponding cluster.

[0144] Although scheduler 322, API server 320, controller manager 326, network controller 324, network controller manager 325, and configuration storage 328 are shown and described as being executed by a single computing device 300, they can be distributed across multiple computing devices 300 that make up a computing system or hardware / server cluster. In other words, each of the multiple computing devices 300 can provide a hardware operating environment for one or more instances of any one or more of scheduler 322, API server 320, network controller manager 326, network controller 324, network controller manager 325, or configuration storage 328. Network controller 324 can represent... Figure 1Example instance of network controller 24. Scheduler 322, API server 320, controller manager 326, and network controller manager 325 can implement example instances of orchestrator 23. Network controller manager 325 can represent an example implementation of Kubernetes cloud controller manager. Network controller 324 can represent an example instance of network controller 24.

[0145] In this example, computing device 300 includes a bus 342 that couples hardware components of the computing device 300 hardware environment. Bus 342 couples a network interface card (NIC) 330, a storage disk 346, and one or more microprocessors 310 (hereinafter referred to as "microprocessor 310"). In some cases, the front-side bus may couple the microprocessor 310 and memory 344. In some examples, bus 342 may couple memory 344, microprocessor 310, and NIC 330. Bus 342 may represent a Peripheral Component Interface (PCI) Fast (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 342. In some examples, components coupled to bus 342 control DMA transfers between components coupled to bus 342.

[0146] Microprocessor 310 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, which are stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or “multi-core” processors), each multi-core processor being implemented using a single IC (i.e., a chip multiprocessor).

[0147] Disk 346 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technique for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by the microprocessor 310.

[0148] Main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM) or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media, which may be used to carry or store desired program code and program data in the form of instructions or data structures and may be accessible to a computer. Main memory 344 provides a physical address space consisting of addressable storage locations.

[0149] Network interface card (NIC) 330 includes one or more interfaces 332 configured to switch packets using links of the underlying physical network. Interface 332 may include a port interface card with one or more network ports. NIC 330 may also include on-card memory for, for example, storing packet data. Direct memory access transfers between NIC 330 and other devices coupled to bus 342 can read / write to NIC memory.

[0150] Memory 344, NIC 330, storage disk 346, and microprocessor 310 provide an operating environment for a software stack, which includes an operating system kernel 314 executing in kernel space. Kernel 314 may represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, available from Microsoft. In some cases, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). The operating system including kernel 314 provides an execution environment for one or more processes in user space 345. Kernel 314 includes a physical drive 327 using network interface card 230.

[0151] Computing device 300 can be coupled to a physical network switching infrastructure, which includes an overlay network that extends the switching infrastructure from physical switches to software or "virtual" routers coupled to the physical servers of the switching infrastructure, for example... Figure 2 The virtual router 220. The computing device 300 can use one or more dedicated virtual networks to configure the slave nodes of the cluster.

[0152] API server 320, scheduler 322, controller manager 326, and configuration storage can implement the cluster's master node and are alternatively referred to as the "master component". The cluster can be a Kubernetes cluster, and the master node can be a Kubernetes master node, in which case the master component is the Kubernetes master component.

[0153] API server 320 includes code executable by microprocessor 310. API server 320 can be one or more computer processes. API server 320 authenticates and configures data for objects, such as virtual execution elements (e.g., container cabins), services, and replication controllers. A service can be an abstraction that defines a set of logical cabins and policies for accessing those cabins. A set of cabins is selected to implement the service based on the service definition. A service can be partially implemented as a load balancer or otherwise include a load balancer. API server 320 can implement a Representational State Transfer (REST) ​​interface to handle REST operations and provide a front end to the shared state of the corresponding cluster stored in configuration storage 328. API server 320 can authenticate and authorize requests. API server 320 communicates with other components to instantiate virtual execution elements in compute infrastructure 8. API server 320 can represent a Kubernetes API server.

[0154] Configuration store 328 serves as the backup storage for all cluster data. This cluster data can include cluster state and configuration data. The configuration data can also provide a backend for service discovery and / or provide locking services. Configuration store 328 can be implemented as a key-value store. Configuration store 328 can be a central database or a distributed database. Configuration store 328 can represent etcd storage. Configuration store 328 can represent Kubernetes configuration storage.

[0155] Scheduler 322 includes code executable by microprocessor 310. Scheduler 322 can be one or more computer processes. Scheduler 322 monitors newly created or requested virtual execution elements (e.g., container bays) and selects slave nodes on which the virtual execution elements will run. In some examples, scheduler 322 may select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, location, etc. In other examples, scheduler 322 may select slave nodes based on a custom list of virtual networks specified by the respective virtual execution element. Scheduler 322 may represent a Kubernetes scheduler.

[0156] Typically, API server 320 can invoke scheduler 322 to schedule virtual execution elements. A virtual execution element can select a slave node and return the identifier of the selected slave node to API server 320. API server 320 can then write the identifier, associated with the virtual execution element, to configuration store 328. API server 320 can invoke orchestration agent 209 for the selected slave node, which enables the container engine 208 of the selected slave node to obtain the virtual execution element from the storage server and create the virtual execution element on the slave node. The orchestration agent 209 of the selected slave node can update the state of the virtual execution element to API server 320, and API server 320 persists this new state to configuration store 328. In this way, computing device 300 instantiates new virtual execution elements in computing infrastructure 8.

[0157] Controller Manager 326 includes code executable by microprocessor 310. Controller Manager 326 can be one or more computer processes. Controller Manager 326 can embed a core control loop to monitor the shared state of the cluster by receiving notifications from API server 320. Controller Manager 326 can attempt to move the cluster state towards a desired state. Example controllers (not shown) managed by Controller Manager 326 may include a replication controller, endpoint controller, namespace controller, and service account controller. Controller Manager 326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, termination bucket garbage collection, cascading delete garbage collection, node garbage collection, etc. Controller Manager 326 can represent the Kubernetes controller manager of a Kubernetes cluster.

[0158] Network controller 324 includes code executable by microprocessor 310. Network controller 324 may include one or more computer processes. Network controller 324 may represent... Figure 1 Example instance of network controller 24. Network controller 324 may be a logically centralized but physically distributed software-defined networking (SDN) controller responsible for providing management, control, and analysis functions for the virtualized network. In particular, network controller 324 may be a logically centralized control plane and management plane of computing infrastructure 8, and orchestrate vRouters for one or more slave nodes.

[0159] Network controller 324 can provide cloud networking for computing architectures running on network infrastructure. Cloud networks may include private clouds of enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) of cloud service providers. Private cloud, VPC, and IaaS use cases may involve multi-tenant virtualized data centers, such as... Figure 1In this scenario, multiple tenants in the data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is allocated its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; virtual networks). These logical resources are isolated from each other unless specifically permitted by security policies. Virtual networks in the data center may also be interconnected to physical IP VPNs or L2 VPNs.

[0160] The network controller 324 can provide network function virtualization (NFV) to networks such as business edge networks, broadband user management edge networks, and mobile edge networks. NFV involves the orchestration and management of network functions in virtual machines, containers, or other virtual execution elements, rather than on physical hardware devices, such as firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, wide area network (WAN) optimization, etc. The main drivers for network service virtualization in the market are time-to-market and cost optimization.

[0161] Network controller 324 programs network infrastructure components to create virtual networks and can create interface configurations for virtual network interfaces of the virtual networks. For example, network controller 324 can receive data indicating namespace specification data 27 from network controller manager 325. Based on namespace specification data 27, network controller manager 325 can create a namespace specifying a primary virtual network group. Network controller 324 can parse the data indicating namespace specification data 27, which is formatted as a human-readable text file. By parsing the human-readable text file, the network controller can obtain information for creating the primary virtual network group. In some examples, network controller 324 uses this information to configure any combination of TOR switch 16 and rack switch 18 to create a primary virtual network group in computing infrastructure 8. Network controller 324 can create the primary virtual network group independently of API server 320, scheduler 322, network controller manager 325, and controller manager 326.

[0162] Additionally, when API server 320 and scheduler 322 schedule new virtual execution elements, the new virtual execution element may include a namespace object specifying a namespace and a network annotation specifying a secondary virtual network group. The namespace specified in the namespace object may specify a primary virtual network group. In some examples, at least one virtual network in the secondary virtual network group is not included in the primary virtual network group. Network controller 324 may create each virtual network in the secondary virtual network group that has not yet been created when scheduling the virtual execution element.

[0163] Additional information regarding the network controller 324, which operates in conjunction with other devices or other software-defined networks of the data center 10, can be found in international application number PCT / US2013 / 044378 and U.S. patent application No. 14 / 226,509, both of which are incorporated herein by reference.

[0164] Network controller manager 325 includes code executable by microprocessor 310. Network controller manager 325 can be one or more computer processes. Network controller manager 325 acts as an interface between orchestration-oriented elements (e.g., scheduler 322, API server 320, controller manager 326, and configuration store 328) and network controller 324. Typically, network controller manager 325 monitors new objects (e.g., cabins and services) in the cluster. Network controller manager 325 can isolate cabins in a virtual network and connect cabins and services.

[0165] The Network Controller Manager 325 can be executed as a container of the cluster's master node. In some cases, using the Network Controller Manager 325 enables the disabling of service proxies on slave nodes (e.g., the Kubernetes kube agent), thereby enabling all cabin connectivity using a virtual router, as described in this article.

[0166] Network Controller Manager 325 can use the orchestration platform's controller framework to listen for (or otherwise monitor) changes to objects defined in the API and add annotations to some of these objects. Annotations can be labels or other identifiers specifying object attributes (e.g., "virtual network green"). Network Controller Manager 325 can use the interface to Network Controller 324 to create network solutions for applications, defining network objects such as virtual networks, virtual network interfaces, and access control policies. Network Controller 324 can implement network solutions in the computing infrastructure, for example, by configuring one or more virtual networks and virtual network interfaces in a virtual router.

[0167] The Network Controller Manager 325 can use a single network module 206A for configuring pod replica hosts to drive the creation of multiple virtual network interfaces for each pod replica.

[0168] Figure 1-3The various components, functional units, and / or modules shown and / or described elsewhere in this disclosure can perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and / or executing on one or more computing devices. For example, a computing device may execute one or more such modules using multiple processors or multiple devices. A computing device may execute one or more such modules as a virtual machine executing on underlying hardware. One or more such modules may execute as one or more services of an operating system or computing platform. One or more such modules may execute as one or more executable programs at the application layer of a computing platform. In other examples, the functionality provided by a module may be implemented by dedicated hardware devices. Although certain modules, data stores, components, programs, executable files, data items, functional units, and / or other items included in one or more storage devices may be shown individually, one or more such items may be combined and operate as a single module, component, program, executable file, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined such that they operate or provide functionality as a single module. Furthermore, one or more modules may operate in combination with each other such that, for example, one module acts as a service or extension of another module. Furthermore, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may include multiple components, subcomponents, modules, submodules, data stores, and / or other components or modules or data stores not shown. Additionally, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may be implemented in various ways. For example, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may be implemented as part of an operating system that executes on a computing device.

[0169] Figure 4 This is a flowchart illustrating the creation of multiple network virtual interfaces for a virtual execution element using a single network module, according to the techniques described in this disclosure. (Reference) Figures 2 to 3 The components of computing devices 200 and 300 are used to describe the operation. API server 320 receives a request to instantiate pod 202A and modifies the configuration storage 328 (402) with the configuration used to create pod 202A. Scheduler 322 can select computing device 200 as the master slave node of pod 202A. API server 320 can annotate pod 202A with a list of multiple virtual networks and the pod's UUID (pod_uuid). Other forms of pod identifiers can be used. Annotations can be labels indicating the pod configuration of virtual networks, such as "Virtual Network A" and "Virtual Network B".

[0170] Network controller manager 325 listens for new objects from API server 320 and determines that cabin 202A will be instantiated on computing device 200, and determines, based on annotations, that cabin 202A needs to interface with multiple virtual networks indicated in the annotations. For example, the listener may respond to subscribing to API server 320 notifications on a RESTful interface.

[0171] Network controller manager 325 instructs network controller 324 to create a virtual network and to create a virtual network interface (404) for compartment 202A of the virtual network. Network controller manager 325 can annotate the compartment with the corresponding UUID of one or more virtual network interfaces (vni_uuids) to be created by network controller 324, and the corresponding unique private virtual network address (and in some cases, a MAC address). Other forms of identifiers for the virtual network interfaces can be used.

[0172] Network controller 324 can create a list of virtual network interfaces for the virtual network and associate vni_uuids with pod_uuids in interface configuration data 25 of compartment 202A. Network controller 324 can send interface configuration data 25 to virtual router agent 216 of virtual router 220 of computing device 200 and configure the corresponding virtual network interfaces 212A, 212B (406) in computing device 200. The virtual network interface can then represent a veth pair connected to a bridge different from virtual router 220. Virtual router agent 216 can store the association between each vni_uuid and the corresponding configured virtual network interface.

[0173] To configure cabin 202A, orchestration agent 209 obtains the container specification data for cabin 202A and ensures that the container is executed by computing device 200 (408). The container specification data may include the pod_uuid of cabin 202A. Orchestration agent 209 invokes a single network module 206A to configure the virtual network interface of cabin 202A (410). Network module 206A requests (412) and obtains interface configuration data 25 from virtual router agent 216 (414). Network module 206A can obtain interface configuration data 25 from virtual router agent 216 by requesting the interface configuration data of the cabin corresponding to the pod_uuid included in the container specification data of cabin 202A.

[0174] To create each of the virtual network interfaces 212A, 212B indicated in Interface Configuration Data 25 (416), Network Module 206A can insert the virtual network interface into the Network Namespace of Cabin 202A (e.g., as one end of a veth pair of the virtual network interface) (418), and can make any necessary changes on Computing Device 200 (e.g., connecting the other end of the veth pair to the virtual router 220—which may have previously been connected to another bridge). Network Module 206A can configure the virtual network address within the Network Namespace of Cabin 202A and can set routes consistent with the IP Address Management section by invoking the appropriate IPAM plugin.

[0175] Network module 206A notifies virtual router agent 216 of the currently operating virtual network interfaces 212A and 212B (via insert pod 202A) (420). Network module 206A can also obtain virtual network addresses from virtual router agent 216 (422) and update the orchestration control plane by notifying orchestration agent 209 (424).

[0176] These techniques describe how a user can provide a list of networks as annotations in the YAML of container 202A. The network controller manager 325 can parse this network list and create corresponding ports in the virtual router 220. When container 202A is scheduled on computing device 200, network module 206A can query the virtual router agent 216 for ports. The virtual router agent 216 will respond with a list of ports, and for each member of that list, network module 206A will create a tap interface and connect it to container 202A. Because all virtual network interfaces 212A, 212B are created in a single call flow, this provides better performance when creating virtual network interfaces and connecting them to containers. Container 229A can communicate via either virtual network interface 212A, 212B to exchange packets with other containers in the cluster on the corresponding network, or use, for example, a gateway to exchange packets outside the virtual network and the cluster's containers.

[0177] In some cases, these technologies can be used with an orchestration platform to run different container networking functions within a compute infrastructure, such as containerized security appliances or containerized routers. Container networking functions may typically require network interfaces for multiple networks, such as routers, NAT devices, packet analyzers, and firewalls or other security devices. Therefore, these technologies can enable service links with container orchestration platforms for container-centric compute infrastructures.

[0178] For example, any one or more containers 229A can represent a containerized network function or a containerized network function virtualization (NFV) instance. Container 229A of cabin 202A has a virtual network interface 212A with a first virtual network corresponding to VRF 222A and a virtual network interface 212B with a second virtual network corresponding to VRF 222B. Other containerized applications executing in cabins on other slave nodes of computing device 200 or cluster and having virtual network interfaces in the first or second virtual network can exchange packets with containers 229A and containerized network functions. Service chains of one or more containerized network functions can apply service sequences to network packets traversing the service chains. VRF 222 of the virtual router 220 for slave nodes can be configured to forward traffic along service sequences, for example, by configuring the service VRF for the containerized network function as the remaining VRF used for the service.

[0179] In an example use case, for a two-tier application with frontend and database support, the frontend bay can be configured as a virtual network endpoint of virtual network A (corresponding to VRF 222A), and the database bay can be configured as a virtual network endpoint of virtual network B (corresponding to VRF 222B). The containerized firewall can be deployed as a container 229A instance within bay 202A. Virtual network interface 212A is the interface for virtual network A, and virtual network interface 212B is the interface for virtual network B. Packets received from the frontend bay at virtual network interface 212A can be processed by the containerized firewall and output to the backend database bay via virtual network B through virtual network interface 212B.

[0180] Figure 5 This is a flowchart illustrating the creation of multiple network virtual interfaces for virtual execution elements based on namespaces according to the techniques described in this disclosure. (Reference) Figures 2 to 3The operation is described using components of computing devices 200 and 300. API server 320 receives requests to create one or more namespaces and modifies configuration store 328 (502) with configurations for creating the one or more namespaces. In some examples, the request to create one or more namespaces includes namespace specification data 27. For example, namespace specification data 27 may be encoded using a human-readable data serialization language (e.g., YAML, JSON, etc.). Namespace specification data 27 specifies at least one namespace. Additionally, each namespace of the at least one namespace specifies multiple virtual networks. Network controller manager 325 generates namespace configuration data based on namespace specification data 27 and instructs network controller 324 to create virtual networks based on the namespace specification data (504). Network controller manager 325 may instruct network controller 324 in this way, for example, by issuing requests to network controller 324 via an API, or by storing one or more network configuration objects in the configuration store of the virtualized computing infrastructure. In some examples, namespace specification data 27 includes a network topology framework for multiple virtual networks specified by each namespace. In this way, network controller 324 can parse namespace specification data to obtain a network topology framework for creating multiple virtual networks. Steps 502 and / or 504 can be performed as part of the instantiation process.

[0181] API server 320 receives a request to instantiate pod 202A and modifies the configuration store 328 (506) with the configuration used to create pod 202A. Scheduler 322 can select computing device 200 as the master slave node of pod 202A. API server 320 can annotate pod 202A with a list of multiple virtual networks and the pod's UUID (pod_uuid). Other forms of identifiers for pods can be used. Annotations can be labels indicating the pod configuration of virtual networks, such as "Virtual Network A" and "Virtual Network B".

[0182] Network controller manager 325 listens for new objects from API server 320 and determines that cabin 202A will be instantiated on computing device 200, and determines, based on annotations, that cabin 202A needs to interface with virtual networks of multiple virtual networks indicated in the annotations. For example, the listener may respond to subscribing to API server 320 notifications on a RESTful interface.

[0183] Network controller manager 325 instructs network controller 324 to create virtual network interfaces (508) for the virtual network compartment 202A. Network controller manager 325 can annotate the compartment with the corresponding UUID of one or more virtual network interfaces (vni_uuids) to be created by network controller 324, and the corresponding unique private virtual network address (and in some cases, the MAC address). Other forms of identifiers for the virtual network interfaces can be used.

[0184] Network controller 324 can create a list of virtual network interfaces for the virtual network and associate vni_uuids with pod_uuids in interface configuration data 25 of compartment 202A. Network controller 324 can send interface configuration data 25 to virtual router agent 216 of virtual router 220 of computing device 200 and configure the corresponding virtual network interfaces 212A, 212B (510) in computing device 200. The virtual network interface can then represent a veth pair connected to a bridge different from virtual router 220. Virtual router agent 216 can store the association between each vni_uuid and the corresponding configured virtual network interface.

[0185] To configure cabin 202A, orchestration agent 209 obtains the container specification data for cabin 202A and ensures that the container is executed by computing device 200 (512). The container specification data may include the pod_uuid of cabin 202A. Orchestration agent 209 invokes a single network module 206A to configure the virtual network interface of cabin 202A (514). Network module 206A requests (516) and obtains interface configuration data 25 from virtual router agent 216 (518). Network module 206A can obtain interface configuration data 25 from virtual router agent 216 by requesting the interface configuration data of the cabin corresponding to the pod_uuid included in the container specification data of cabin 202A.

[0186] To create each of the virtual network interfaces 212A, 212B indicated in Interface Configuration Data 25 (520), Network Module 206A can insert the virtual network interface into the Network Namespace of Cabin 202A (e.g., as one end of a veth pair of the virtual network interface) (522), and can make any necessary changes on Computing Device 200 (e.g., connecting the other end of the veth pair to the virtual router 220—which may have previously been connected to another bridge). Network Module 206A can configure the virtual network address within the Network Namespace of Cabin 202A and can set routes consistent with the IP address management section by invoking the appropriate IPAM plugin.

[0187] Network module 206A notifies virtual router agent 216 of the currently operating virtual network interfaces 212A and 212B (via insert pod 202A) (524). Network module 206A can also obtain virtual network addresses from virtual router agent 216 (526) and update the orchestration control plane by notifying orchestration agent 209 (528).

[0188] The techniques described herein can be implemented using hardware, software, firmware, or any combination thereof. Various features described as modules, units, or components can be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0189] If implemented in hardware, this disclosure may relate to devices such as processors or integrated circuit devices, for example, integrated circuit chips or chipsets. Alternatively or additionally, if implemented in software or firmware, these techniques may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store instructions for execution by a processor.

[0190] Computer-readable media can form part of a computer program product, which may include packaging material. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0191] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can mean that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., in RAM or cache).

[0192] The code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure can be provided within software or hardware modules.

Claims

1. A controller comprising an orchestrator for a virtualized computing infrastructure, wherein, The arranger is configured to: Receive namespace specification data, the namespace specification data specifying a namespace, a first virtual network for the namespace, and a second virtual network for the namespace, wherein the first virtual network and the second virtual network are members of a primary group of one or more virtual networks of virtual execution elements deployed to the namespace; Receive virtual execution element specification data for a specified virtual execution element, wherein the virtual execution element specification data includes a namespace object specifying the namespace of the virtual execution element, and wherein the virtual execution element specification data specifies a minor group of one or more virtual networks of the virtual execution element; and Based on the namespace specification data and the virtual execution element specification data, one or more requests are sent to the network controller of the virtualized computing infrastructure to create corresponding virtual network interfaces for the first and second virtual networks of the primary group of one or more virtual networks for the virtual execution element, and to create corresponding virtual network interfaces for the secondary group of one or more virtual networks for the virtual execution element.

2. The controller according to claim 1, wherein, The arranger is further configured to: The namespace specification data is used to guide the network controller in creating the first virtual network; and The network controller is guided to create the second virtual network based on the namespace specification data.

3. The controller according to claim 2, further comprising a network controller.

4. The controller according to claim 3, wherein, The network controller is configured to: In response to instructions for creating the first virtual network, the first virtual network in the virtualized computing infrastructure is configured; and In response to the guidance to create the second virtual network, the second virtual network in the virtualized computing infrastructure is configured.

5. The controller according to claim 3, in, The virtual execution element is configured to be deployed to a namespace and instantiated in the computing device of the virtualized computing infrastructure, and The network controller is configured as follows: Interface configuration data is sent to the computing device to configure a first virtual network interface for a first virtual network and a second virtual network interface for a second virtual network, wherein a virtual router is configured within the computing device, and a first virtual route and forwarding instance (VRF) corresponding to the first virtual network and a second VRF corresponding to the second virtual network are configured within the virtual router. Specifically, sending the interface configuration data causes the computing device to attach the first virtual network interface to the virtual execution element and the first VRF, enabling the virtual execution element to perform packet communication on the first virtual network. Specifically, sending the interface configuration data causes the computing device to attach the second virtual network interface to the virtual execution element and the second VRF, so that the virtual execution element can perform packet communication on the second virtual network.

6. The controller according to claim 5, wherein, The interface configuration data includes the identifier of the first virtual network interface of the first virtual network and the identifier of the second virtual network interface of the second virtual network.

7. The controller according to claim 6, wherein, The network controller is configured to send the identifier of the first virtual network interface of the first virtual network and the identifier of the second virtual network interface of the second virtual network to the virtual router proxy of the virtual router of the computing device.

8. The controller according to claim 6, wherein, The network controller is configured to: Send the identifier of the first virtual network interface of the first virtual network and the identifier of the second virtual network of the second virtual network to the virtual router proxy of the virtual router of the computing device; The association between the identifier of the virtual execution element, the identifier of the first virtual network interface of the first virtual network, and the identifier of the second virtual network interface of the second virtual network is sent to the virtual router agent.

9. The controller according to claim 1, wherein, The orchestrator includes a network controller manager for the network controller, wherein the network controller manager is configured to: Assign a first virtual network address to the first virtual network interface; Assign a virtual second network address to the second virtual network interface; The network controller is instructed to configure the first virtual network interface using the first virtual network address of the first virtual network interface; and The network controller is instructed to configure the second virtual network interface using the second virtual network address of the second virtual network interface.

10. The controller according to claim 1, wherein, The secondary group of virtual networks includes a third virtual network that is neither the first virtual network nor the second virtual network.

11. The controller according to claim 1, wherein, The secondary group of virtual networks includes a third virtual network that is the first virtual network.

12. The controller of claim 1, wherein the namespace specification data uses data conforming to a pattern for specifying virtual networks for the network controller to specify the first virtual network and the second virtual network.

13. The controller according to any one of claims 1 to 12, wherein, The virtual execution element includes one or more containers.

14. The controller according to any one of claims 1 to 12, wherein, The virtual execution element includes a Kubernetes module.

15. A network configuration method, comprising: The orchestrator of the virtualized computing infrastructure receives namespace specification data, which specifies a namespace, a first virtual network for the namespace, and a second virtual network for the namespace, wherein the first virtual network and the second virtual network are members of a primary group of one or more virtual networks deployed to the namespace. The orchestrator receives virtual execution element specification data for a specified virtual execution element, wherein the virtual execution element specification data includes a namespace object specifying the namespace of the virtual execution element, and wherein the virtual execution element specification data specifies a minor group of one or more virtual networks of the virtual execution element; and The orchestrator sends one or more requests to the network controller of the virtualized computing infrastructure based on the namespace specification data and the virtual execution element specification data, in order to create corresponding virtual network interfaces for the first and second virtual networks of the primary group of one or more virtual networks for the virtual execution element, and to create corresponding virtual network interfaces for the secondary group of one or more virtual networks for the virtual execution element.

16. The method of claim 15, further comprising: The orchestrator guides the network controller to create the first virtual network based on the namespace specification data; as well as The orchestrator guides the network controller to create the second virtual network based on the namespace specification data.

17. The method of claim 16, further comprising: In response to guidance on creating the first virtual network, the network controller configures the first virtual network in the virtualized computing infrastructure; as well as In response to instructions for creating the second virtual network, the network controller configures the second virtual network within the virtualized computing infrastructure.

18. The method according to any one of claims 15 to 17, further comprising: The network controller manager of the orchestrator assigns a first virtual network address to the first virtual network interface; The network controller manager assigns a virtual second network address to the second virtual network interface; The network controller manager instructs the network controller to configure the first virtual network interface using the first virtual network address of the first virtual network interface; as well as The network controller manager instructs the network controller to configure the second virtual network interface using the second virtual network address of the second virtual network interface.

19. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform a method executed by a controller according to any one of claims 1 to 14.

Images