Containerized routing protocol process for VPNs

Abstract

A containerized routing protocol process for a virtual private network. Generally, this disclosure describes techniques for implementing a virtual private network using a routing protocol using a containerized routing protocol process. In an example, a system includes a container orchestration system for a computing device cluster, the computing device cluster including computing devices, wherein the container orchestration system is configured to: deploy a containerized application to a computing node; and in response to deploying the containerized application to the computing node, configure a virtual routing and forwarding (VRF) instance in the computing node to implement a virtual private network (VPN) for the containerized application.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. patent application No. 17 / 807,700, filed on June 17, 2022, which claims the benefit of U.S. Provisional Patent Application No. 63 / 242,434, filed on September 9, 2021, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to virtualized computing infrastructure, and more particularly to virtual private networks using virtualized computing infrastructure. Background Art

[0004] In a typical cloud data center environment, there are a large number of interconnected servers that provide computing and / or storage capabilities to run various applications. For example, a data center may include a facility that hosts applications and services for subscribers (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, storage systems and application server clusters are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers utilize subscriber support equipment located in various physical hosting facilities to provide a globally distributed infrastructure.

[0005] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. In particular, modern data centers have extensively utilized virtualization environments in which virtual hosts (also referred to herein as virtual execution elements, such as virtual machines or containers) are deployed and executed on the underlying computing platform of physical computing devices.

[0006] Virtualization within a data center can provide several advantages. One advantage is that virtualization can significantly improve efficiency. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become increasingly powerful, making virtualization easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become fungible, such as in cloud-based computing environments, the provisioning and management of computing infrastructure becomes easier. Therefore, in addition to the efficiency and increased return on investment (ROI) provided by virtualization, enterprise IT personnel generally prefer virtualized computing clusters in data centers due to their management advantages.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable execution elements suitable for isolating applications from each other and the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to a container image and executed as a single, lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of making software work across different computing environments. Containers offer the promise of consistent operation from one computing environment to another, virtual or physical.

[0008] Due to the inherent lightweight nature of containers, a single host can typically support many more container instances than regular virtual machines (VMs). Generally speaking, containers have a short lifecycle, can be created and moved more efficiently than VMs, and can also be managed as logically related groups of elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the requirements for container networking solutions: the network should be agile and scalable. VMs, containers, and bare metal servers may need to coexist in the same computing environment, and communication between different application deployments should be enabled. Container networking should also be agnostic to working with the multiple types of orchestration platforms used to deploy containerized applications.

[0009] Managing the deployment and computing infrastructure of application execution infrastructure can involve two main roles: (1) Orchestration - used to automatically deploy, scale, and operate applications across a cluster of hosts and provide computing infrastructure, which can include container-centric computing infrastructure; (2) Network management - used to create virtual networks in the network infrastructure to enable packet communication between applications running on virtual execution environments (such as containers or VMs) and between applications running on traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0010] In general, this disclosure describes techniques for implementing a virtual private network using a routing protocol using a containerized routing protocol process. A container network interface plugin (CNI) is a networking solution for application containers and is a runtime executable that helps configure the interface between the container and other components of the computing device ("host") that hosts the container. The computing device may alternatively be referred to as a "compute node" or "server". The CNI typically creates a network interface, assigns a network address (e.g., an IP address) to the network interface, and may also add routes associated with the network interface, such as routes to a default gateway and one or more name servers.

[0011] In an example of the described technology, a containerized routing protocol daemon (cRPD) is a containerized routing protocol process that is encapsulated as a container to run in, for example, a Linux-based environment. cRPD can execute as a containerized process in user space on a host. Thus, cRPD provides a rich routing software pedigree of a physical router on Linux-based compute nodes. cRPD provides control plane functionality. Existing implementations of cRPD (running on the host) use forwarding provided by the Linux kernel. Therefore, this control plane is containerized. A virtual router is a software entity that provides data plane functionality on a compute node. Compute nodes can host centrally orchestrated and configured VMs or containers. The virtual router can work with an SDN controller to create an overlay network by exchanging routing, configuration, and other data. The virtual router can run as a Linux kernel module, as a DPDK-based process, or on a SmartNIC. DPDK allows the virtual router to process more packets per second than when running as a kernel module. This virtual router can also be containerized. Regardless of how the virtual router is implemented, cRPD and the virtual router together can implement a virtualized provider edge (PE) router that can implement various types of virtual private networks (VPNs) by receiving and advertising routes across the core Layer 3 (L3) network. Example types of VPNs include Layer 3 VPNs (L3VPNs), Layer 2 VPNs (L2VPNs), and Ethernet VPNs (EVPNs), although other types of VPNs can be implemented using the techniques described herein.

[0012] These technologies can provide one or more technical advantages that enable at least one practical application. For example, although the containerized routing protocol process and virtual router components of the virtualized PE router described herein are executed by a server or other computing node (rather than a physical device deployed at the logical edge of a Layer 3 network, such as a PE router), these technologies can allow operators to use a service provider VPN model in a virtualized computing infrastructure. In this way, these technologies can help bridge the gap between information technology (IT) operators and network administrators, allowing IT operators to leverage the underlying Layer 3 network infrastructure to provide virtual private networks to host workloads without having to deploy and configure physical PE routers. In addition, tenants that deploy containerized workloads to operate partially as customer edge (CE) devices can rely on virtualized PE routers to operate similarly to physical PE routers to provide VPNs to interconnect containerized workloads.

[0013] As another example of a technical advantage, while there are many different CNI solutions that can provide networking for containers deployed by container orchestration systems (e.g., Kubernetes), these CNI solutions involve creating a virtual bridge device on the host computing device to interconnect pods executing on the host computing device, and then using tunneling mechanisms (such as VxLAN or Generic Routing Encapsulation (GRE)) to interconnect pods executing on different host computing devices. These CNIs are generally incompatible with other types of CNIs to enable networking for pods deployed inside or outside a cluster of host computing devices (e.g., a Kubernetes cluster). That is, to connect workloads on different computing devices, each computing device must implement the same type of tunnel in the same manner as specified by the CNI. This can prevent connectivity between a first workload in a cluster and a second workload in another cluster, and can also prevent connectivity between a first workload in a cluster and a device attached to a physical router as a CE device, as well as devices reachable via the CE device. By utilizing standard VPN technologies, a virtual PE device can terminate tunnels with other virtualized PE routers and even physical PE devices that also implement these standard VPN technologies, thereby facilitating connectivity between various devices or workloads reachable via any of these virtualized PE routers or physical PE devices.

[0014] In an example, a computing device includes a processing circuit system and a storage device, wherein the processing circuit system is capable of accessing the storage device; a virtual router executed by the processing circuit system and configured to implement a data plane of a virtualized provider edge (PE) router; a containerized application; a virtual network interface that enables communication between the virtual router and the containerized application; a containerized routing protocol process executed by the processing circuit system and configured to implement a control plane for the virtualized PE router, the containerized routing protocol process being configured to: obtain a media access control (MAC) address or an internet protocol (IP) address sent by the containerized application via the virtual network interface; and execute a first routing protocol to generate and output a route including at least one of the MAC address or the IP address to implement a virtual private network (VPN) for the containerized application.

[0015] In an example, a container orchestration system for a computing device cluster including computing devices includes a processing circuit system and a storage device, wherein the processing circuit system is configured to cause the container orchestration system to: deploy a containerized application to a computing node; and in response to deploying the containerized application to the computing node, configure a virtual routing and forwarding (VRF) instance in the computing node to implement a virtual private network (VPN) for the containerized application.

[0016] In an example, a non-transitory computer-readable medium includes instructions that, when executed by processing circuitry of a computing device, cause the computing device to: execute a virtual router to implement a data plane for a virtualized provider edge (PE) router; configure a virtual network interface to enable communication between the virtual router and a containerized application on the computing device; and execute a containerized routing protocol process to implement a control plane for the virtualized PE router to: obtain a media access control (MAC) address or an internet protocol (IP) address sent by the containerized application via the virtual network interface, and execute a first routing protocol to generate and output a route including at least one of the MAC address or the IP address to implement a virtual private network (VPN) for the containerized application.

[0017] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram illustrating an example system in which examples of the techniques described herein may be implemented;

[0019] Figure 2 is a block diagram of an example system for implementing a VPN using a virtualized router with a containerized routing protocol process in accordance with the techniques of this disclosure;

[0020] Figure 3 is a block diagram of an example computing device according to the techniques described in this disclosure;

[0021] Figure 4 is a block diagram of an example computing device operating as an instance of an orchestrator ("orchestrator") master node for a cluster of virtualized computing infrastructure in accordance with the techniques described in this disclosure;

[0022] Figure 5 is a block diagram illustrating an example implementation of a cRPD that an orchestrator can deploy using a pod according to techniques described in this disclosure;

[0023] Figure 6 is a block diagram illustrating an example vRouter agent according to techniques of this disclosure;

[0024] Figure 7 is a conceptual diagram illustrating example operations for programming virtual router forwarding information according to the techniques of this disclosure;

[0025] Figure 8 is a conceptual diagram illustrating example operations for configuring and advertising a virtual network interface in a server having a virtualized router, in accordance with the techniques of this disclosure;

[0026] Figure 9 is a flowchart illustrating an example mode of operation of components of a computing device according to the techniques described herein; and

[0027] Figure 10 is a flow chart illustrating an example mode of operation according to the techniques described herein.

[0028] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0029] Figure 1 is a block diagram illustrating an example system in which the techniques described herein may be implemented. The system includes a virtualized computing infrastructure 8. Generally, a data center 10 provides an operating environment for applications and services at a customer site 11 (shown as "customer 11"), which has one or more customer networks coupled to the data center via a service provider network 7. Each of data centers 10A-10B (collectively, "data centers 10") may host infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls, for example. These techniques will be further described primarily with respect to data center 10A, which is illustrated in greater detail.

[0030] The service provider network 7 is coupled to a public network 15, which may represent one or more networks managed by other providers and, therefore, may form part of a large-scale public network infrastructure (e.g., the Internet). The public network 15 may represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), a corporate LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider operating the service provider network 7, a corporate IP network, or some combination thereof.

[0031] Although customer site 11 and public network 15 are primarily shown and described as edge networks of service provider network 7, in some examples, one or more of customer site 11 and public network 15 may be tenant networks within data center 10 or another data center. For example, data center 10A may host multiple tenants (customers), each of which is associated with one or more virtual private networks (VPNs), each of which may implement one of customer sites 11.

[0032] The service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement Multi-Protocol Label Switching (MPLS) forwarding and, in such a case, may be referred to as an MPLS network or MPLS backbone. In some cases, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers. The service provider network 7 may be a layer 3 network and may represent or be part of a core network.

[0033] In some examples, data center 10A may represent one of a plurality of geographically distributed network data centers. Figure 1 As shown in the example of , data center 10A can be a facility that provides network services to customers. The customers of the service provider can be collective entities such as businesses and governments, or individuals. For example, a network data center can host web services for multiple businesses and end users. Other exemplary services may include data storage, virtual private networks, business engineering, file services, data mining, scientific computing, or supercomputing. Although shown as a separate edge network of the service provider network 7, elements of the data center 10A (such as, one or more physical network functions (PNFs) or virtualized network functions (VNFs)) can be included in the core of the service provider network 7.

[0034] In this example, data center 10A includes storage and / or compute servers (or "nodes") that are interconnected with servers 12A through 12X (collectively, "servers 12"), depicted as coupled to top-of-rack switches 16A through 16N, via a switching fabric 14 provided by one or more layers of physical network switches and routers. Servers 12 are computing devices and may also be referred to herein as "hosts," "host devices," "host computing devices," "compute nodes," or other similar terms. Although Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10A may include multiple additional servers coupled to other TOR switches 16 in data center 10A, such servers having hardware and software components similar to those shown with respect to server 12A.

[0035] The switch fabric 14 in the illustrated example includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A through 16N (collectively, "TOR switches 16") that are coupled to a distribution layer of chassis (or "spine" or "core") switches 18A through 18M (collectively, "chassis switches 18"). Although not shown, the data center 10A may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10A may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0036] In this example, the TOR switches 16 and the chassis switches 18 provide redundant (multi-homed) connectivity for the servers 12 to the IP fabric 20 and the service provider network 7. The chassis switches 18 aggregate traffic flows and provide connectivity between the TOR switches 16. The TOR switches 16 can be network devices that provide Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functionality. The TOR switches 16 and the chassis switches 18 can each include one or more processors and memories and can execute one or more software processes. The chassis switches 18 are coupled to the IP fabric 20, which can perform Layer 3 routing to route network traffic between the data center 10A and the customer sites 11 through the service provider network 7. The switching architecture of the data center 10A is merely an example. For example, other switching architectures can have more or fewer switching layers. The IP fabric 20 can be or include one or more gateway routers.

[0037] The term "packet flow," "traffic flow," or simply "flow" refers to a set of packets originating from a specific source device or endpoint and being sent to a specific destination device or endpoint. For example, a single packet flow can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. This 5-tuple generally identifies the packet flow to which the received packets correspond. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can be a combination of <source network address, destination network address> or <source network address, source port> that refers to the packet.

[0038] Each of servers 12 may represent a computing server. For example, each of servers 12 may represent a computing device, such as an x86 processor-based server, configured to operate according to the techniques described herein. Servers 12 may provide a network function virtualization infrastructure (NFVI) for an NFV architecture, which is an example of a virtualized computing infrastructure.

[0039] Any of the servers 12 can be configured with virtual execution elements by virtualizing the server's resources to provide isolation between one or more processes (applications) executing on the server. "Hypervisor-based" or "hardware-level" or "platform" virtualization refers to the creation of virtual machines, each of which includes a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualization / guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are virtualized from the host server's physical hardware, the executed applications are isolated from both the host's hardware and the other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces (VNIs) for communicating on a corresponding virtual network.

[0040] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., data center 10A). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly allowed by security policy.

[0041] Virtual networks can use the Data Center 10 Gateway Router ( Figure 1 The virtual network can also be used to implement network function virtualization (NFV) and service chaining.

[0042] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. A virtual network can also be implemented using two networks—a physical underlay network consisting of the IP fabric 20 and the switch fabric 14, and a virtual overlay network. The role of the physical underlay network is to provide the "IP fabric," which provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network can provide uniform, low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

[0043] As further described below with respect to virtual router 21A, the virtual routers running in servers 12 can create a virtual overlay network on top of the physical underlying network using a dynamic mesh of "tunnels" between them. For example, these overlay tunnels can be MPLS over GRE / UDP tunnels, VXLAN tunnels, or NVGRE tunnels. The underlying physical routers and switches may not store any per-tenant state for virtual machines or other virtual execution elements, such as any media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may contain only the IP prefix or MAC address of the physical server 12. (A gateway router or switch connecting the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0044] Virtual routers 21A through 21X (collectively, "virtual routers 21") of server 12 typically contain per-tenant state. For example, they may contain a separate forwarding table (routing instance) per virtual network. This forwarding table contains the IP prefixes (in the case of a Layer 3 overlay) or MAC addresses (in the case of a Layer 2 overlay) of virtual machines or other virtual execution elements (e.g., container pods). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses for all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that are locally present on server 12 (i.e., they have at least one virtual execution element present on server 12 and require that routing instance).

[0045] The control plane protocol between the control plane nodes of the network controller 24 or the physical gateway routers (or switches) can be BGP (and can be NETCONF for management). In other words, the same control plane protocol can also be used for MPLS L3VPN and MPLS EVPN. For example, the protocol between the network controller 24 and the virtual router 21 can be based on XMPP.

[0046] “Container-based” or “operating system” virtualization refers to the virtualization of an operating system for running multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by the open source DOCKER container application or by CoreOS Rkt (“Rocket”). Like a virtual machine, each container is virtualized and can be isolated from the host machine and other containers. However, unlike a virtual machine, each container can omit an individual operating system and instead provide a suite of applications and application-specific libraries. Typically, a container is executed by a host machine as an isolated userspace instance and can share an operating system and common libraries with other containers executing on the host machine. Therefore, containers may require less processing power, storage, and network resources than virtual machines. One or more groups of containers can be configured to share one or more virtual network interfaces for communicating on the corresponding virtual networks.

[0047] In some examples, containers are managed by their host kernel to allow for resource (CPU, memory, block I / O, network, etc.) restriction and prioritization without launching any virtual machines, in some cases using a namespace isolation feature that allows for complete isolation of an application's (e.g., a given container's) view of the operating environment, including process trees, networks, user identifiers, and mounted file systems. In some examples, containers can be deployed in accordance with Linux Containers (LXC), an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

[0048] A Kubernetes Pod is a group of one or more logically related containers that have a shared namespace and a shared file system volume. Each pod is assigned a unique IP address. The containers of a pod share a network namespace, which includes IP addresses and network ports. The containers of a pod can communicate with each other using localhost. However, when containers in a pod communicate with entities outside the pod, the containers can share IP addresses and port space. Containers in a pod can also communicate with each other using standard inter-process communication. Containers in different pods have different IP addresses. Containers that want to interact with containers running in different pods can use IP networking to communicate, and this is typically established using the Container Network Interface (CNI).

[0049] Server 12 hosts virtual network endpoints for one or more virtual networks, which operate over a physical network represented here by IP fabric 20 and switch fabric 14. Although primarily described with respect to a data center-based switch network, other physical networks, such as service provider network 7, may be the basis for one or more virtual networks.

[0050] Each of the servers 12 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint for one or more virtual networks configured in the physical network. A virtual network endpoint for a virtual network can represent one or more virtual execution elements for a virtual network interface of a shared virtual network. For example, a virtual network endpoint can be a virtual machine, one or more sets of containers (e.g., pods), or another(s) other virtual execution element(s), such as a layer 3 endpoint for a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide an at least partially independent execution environment for an application. The term "virtual execution element" can also include a pod of one or more containers. As Figure 1 As shown, server 12A hosts multiple virtual network endpoints in the form of pods 22A to 22C, each pod having one or more containers. However, given the hardware resource constraints of server 12, server 12 can execute as many virtual execution elements as possible. Each of the virtual network endpoints can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0051] Server 12 includes respective network interface cards (NICs) 13A through 13X (collectively, "NICs 13"), each of which includes at least one interface to exchange packets with TOR switch 16 over a communications link. For example, server 12A includes NIC 13A, which is shown as having two links to TOR switch 16A. Any NIC 13 can provide one or more virtual hardware components for virtualized input / output (I / O). The virtual hardware components for I / O can be virtualizations of physical NICs ("physical functions"). For example, in single root I / O virtualization (SR-IOV) described in the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to present one or more virtual network interfaces as "virtual functions" for use by respective endpoints executing on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are examples of virtual hardware components. As another example, one or more servers 12 may implement Virtio, which is a paravirtualization framework available, for example, for use with a Linux operating system, Virtio providing simulated NIC functionality as a type of virtual hardware component to provide a virtual network interface to a virtual network endpoint. As another example, one or more servers 12 may implement Open vSwitch to perform distributed virtual multilayer switching for hosted virtual machines between one or more virtual NICs (vNICs), where such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some cases, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, the virtual hardware component is an SR-IOV virtual function. In some examples, any of the servers 12 may implement a Linux bridge that simulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For a Docker implementation of containers hosted by a server, a Linux bridge or other operating system bridge executed on the server that switches packets between containers may be referred to as a "Docker bridge." As used herein, the term "virtual router" may encompass a Contrail or Tungsten Fabric virtual router, Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other device and / or software that resides on a host device and performs switching, bridging, or routing of packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more of servers 12. Virtual router 21A is an example of such a virtual router.

[0052] One or more of the servers 12 may each include a corresponding virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10A to provide a virtual network interface and route packets between virtual network endpoints. Each of the routing instances may be associated with a network forwarding table. Each of the routing instances may include a virtual routing and forwarding instance (VRF) for an Internet Protocol virtual private network (IP-VPN). Packets received by the virtual router 21A (shown as "vROUTER 21A") of the server 12A, for example, from the underlying physical network fabric (i.e., IP fabric 20 and switch fabric 14) of the data center 10A, may include an outer header that allows the physical network fabric to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the server's network interface card 13A, but also a virtual network identifier, such as a VxLAN label or a Multi-Protocol Label Switching (MPLS) label that identifies one of the virtual networks, and a corresponding routing instance executed by virtual router 21 A. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0053] The virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network for the received packet based on the tunnel encapsulation header for the packet, and forwards the packet to the appropriate destination virtual network endpoint for the packet. For server 12A, for example, for each packet outbound from a virtual network endpoint hosted by server 12A (e.g., any pod 22A to 22C), the virtual router 21A attaches a tunnel encapsulation header indicating the virtual network for generating the encapsulated or "tunnel" packet, and the virtual router 21A outputs the encapsulated packet to a physical destination computing device, such as another server in server 12, via the overlay tunnel for the virtual network. As used herein, the virtual router 21 can perform the operations of a tunnel endpoint to encapsulate an internal packet emitted by a virtual network endpoint to generate a tunnel packet and decapsulate the tunnel packet to obtain the internal packet for routing to other virtual network endpoints.

[0054] In various examples, each of the virtual routers 21 can represent a SmartNIC-based virtual router, a kernel-based virtual router (i.e., executed as a kernel module), or a Data Plane Development Kit (DPDK)-enabled virtual router. The DPDK-enabled virtual router 21A can use DPDK as the data plane. In this mode, the virtual router 21A runs as a user space application linked to the DPDK library (not shown). This is a performance version of the virtual router and is typically used by telecommunications companies, where VNFs are typically DPDK-based applications. The performance of the virtual router 21A as a DPDK virtual router can achieve ten times higher throughput than a virtual router operating as a kernel-based virtual router. The physical interface is used by the poll mode driver (PMD) of the DPDK instead of the interrupt-based driver of the Linux kernel.

[0055] The User I / O (UIO) kernel module (such as vfio or uio_pci_generic) can be used to expose the registers of the physical network interface into user space so that the DPDK PMD can access them. When the NIC 13A is bound to the UIO driver, it is moved from the Linux kernel space to the user space and is therefore no longer managed or visible by the Linux OS. Therefore, it is the DPDK application (i.e., the virtual router 21A in this example) that fully manages the NIC 13. This includes packet polling, packet processing, and packet forwarding. The user packet processing steps can be performed by the virtual router 21A DPDK data plane, while the kernel ( Figure 1 This "polling mode" nature allows virtual router 21A DPDK data plane packet processing / forwarding to be more efficient than interrupt mode, especially at high packet rates. There is limited or no interruption and context switching during packet I / O.

[0056] Additional details of an example of a DPDK vRouter can be found in “DAY ONE: CONTRAIL DPDK vROUTER” (2021, Kiran KN et al., Juniper Networks, Inc.), which is incorporated herein by reference in its entirety.

[0057] Server 12 includes and executes containerized routing protocol daemons 25A through 25X (collectively, "cRPD 25"). A containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged as a container and can run in a Linux-based environment. The cRPD can execute in user space on a host as a containerized process. Thus, in some cases, the cRPD makes available a rich spectrum of routing software for physical routers on Linux-based compute nodes (e.g., server 12). The cRPD provides control plane functionality. Thus, this control plane is containerized. For example, cRPD 25A implements a control plane for virtualized router 32A executed by server 12A.

[0058] Meanwhile, the virtual router 21 is a software entity that provides data plane functionality on the server 12. The CRPD 25A can use the forwarding plane provided by the Linux kernel of the server 12A for the kernel-based virtual router 21A. The CRPD 25A can alternatively use a DPDK-enabled instance of the virtual router 21 or an instance executed by a SmartNIC. The virtual router 21A can work with an SDN controller (e.g., a network controller 24) to create an overlay network by exchanging routing, configuration, and other data. The virtual router 21A can be containerized. In some examples, in combination, the containerized cRPD and the containerized virtual router can therefore be a fully functional containerized virtualized router 32A.

[0059] The virtualized computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration platform that provides a container-centric infrastructure for automating the deployment, scaling, and operation of containers to provide a container-centric infrastructure. In the context of a virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements to host servers available to an orchestration platform. Specifically, container orchestration allows for container coordination and refers to, for example, the deployment, management, scaling, and configuration of containers to hosted servers via a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0060] Orchestrator 23 represents one or more orchestration components for a container orchestration system. Orchestrator 23 orchestrates at least containerized RPD 25. In some examples, data plane virtual router 21 is also containerized and orchestrated by orchestrator 23. For example, the data plane may be a DPDK-based virtual router.

[0061] Elements of the automation platform for computing infrastructure 8 include at least servers 12, orchestrator 23, and network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework, in which a cluster master node manages the deployment and operation of containers to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" used herein encompass different orchestration platform terminology used to distinguish between the primary management element of a cluster and similar devices that are the cluster's primary container hosting devices. For example, the Kubernetes platform uses the terms "cluster master node" and "slave node," while the Docker Swarm platform refers to both cluster managers and cluster nodes.

[0062] Orchestrator 23 and network controller 24 may execute on separate computing devices or on the same computing device. Each of orchestrator 23 and network controller 24 may be a distributed application executed on one or more computing devices. Orchestrator 23 and network controller 24 may implement respective master nodes for one or more clusters, each cluster having one or more slave nodes implemented by respective servers 12 (also referred to as "compute nodes").

[0063] Generally, network controller 24 controls the network configuration of the data center 10A fabric to, for example, establish one or more virtual networks for communication of packets between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10A. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator. Additional information regarding the network controller 24 operating with other devices or other software-defined networks in the data center 10A can be found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS”; and U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, entitled “Tunneled Packet Aggregation for Virtual Networks,” each of which is incorporated herein by reference as if fully set forth herein.

[0064] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides a computing infrastructure that may include a container-centric computing infrastructure. The orchestrator 23 and, in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. For example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud may provide a virtualized infrastructure for the container management platform. Figure 4 Describes example components of the Kubernetes container orchestration system.

[0065] In one example, pod 22A is a Kubernetes pod and is an example of a virtual network endpoint. A pod is a group of one or more logically related containers ( Figure 1), shared storage for containers, and options about how to run containers. When instantiated for execution, a pod may also be referred to as a "pod replica." Each container of a pod 22A is an example of a virtual execution element. The containers of a pod are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of a pod can be a set of Linux namespaces, cgroups, and other isolation aspects. Within the context of a pod, individual applications may have additional sub-isolations applied. Typically, containers within a pod have a public IP address and port space and are able to detect each other via localhost. Because they have a shared context, containers within a pod also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different pods have different IP addresses and cannot communicate via IPC without configuration to enable this feature. Containers that are members of different pods typically communicate with each other via the pod IP address.

[0066] Server 12A includes a container platform 19A for running containerized applications, such as pod 22A. Container platform 19A receives a request from orchestrator 23 to retrieve and host a container in server 12A. Container platform 19A retrieves and executes the container.

[0067] Container platform 19A includes a container network interface (CNI) 17A that configures virtual network interfaces for virtual network endpoints. Orchestrator 23 and container platform 19A use CNI 17A to manage networking for pods, including pod 22A. For example, CNI 17A creates virtual network interfaces to connect pods to virtual router 21A and enables containers of these pods to communicate with other virtual network endpoints via the virtual network interfaces. For example, CNI 17A may insert a virtual network interface for a virtual network into the network namespace for the container in pod 22A and configure (or request configuration) a virtual network interface for the virtual network in virtual router 21A so that virtual router 21A is configured to send packets received from the virtual network via the virtual network interface to the containers of pod 22A and to send packets received from the containers of pod 22A via the virtual network interface on the virtual network. CNI 17A may allocate network addresses (e.g., virtual IP addresses for the virtual networks) and may establish routing for the virtual network interfaces. In Kubernetes, by default, all pods can communicate with all other pods without the use of network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a pod virtual network shared by all namespaces, from which service and pod network addresses can be allocated, respectively. In some cases, all pods in all namespaces generated in the Kubernetes cluster can be able to communicate with each other, and the network addresses for all pods can be allocated from the pod subnet specified by the orchestrator 23. When a user creates an isolated namespace for a pod, the orchestrator 23 and the network controller 24 can create a new pod virtual network and a new shared service virtual network for the new isolated namespace. The pods in the isolated namespace generated in the Kubernetes cluster extract network addresses from the new pod virtual network, and the corresponding services for such pods extract network addresses from the new service virtual network.

[0068] Kubernetes networking between pods is performed via a plugin called the Container Network Interface (CNI) (also known as the Container Network Interface plugin). However, the networking capabilities of a typical CNI are quite rudimentary and are not suitable for situations where the containerized network functions served by the CNI play a critical role in the network. As described in this article, a virtualized router is more suitable for these situations. A virtualized router is a router with a containerized control plane that allows an x86 or ARM-based host to become a first-level member of the network routing system to participate in protocols such as Intermediate System to Intermediate System (IS-IS) and Border Gateway Protocol (BGP) and provide transport and multi-tenancy based on Multiprotocol Label Switching / Segment Routing (MPLS / SR). In other words, the platform is not an adjunct to the network (like a customer edge (CE) router), but can operate as a provider edge (PE) router.

[0069] CNI 17A may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17A may conform, at least in part, to the Container Networking Interface (CNI) specification or the rkt Networking Proposal. CNI 17A may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNIs. CNI 17A may alternatively be referred to as a network plugin, CNI plugin, or CNI instance. For example, a separate CNI may be called by the Multus CNI to establish a different virtual network interface for pod 202A.

[0070] CNI 17A is called by the orchestrator 23. For the purposes of the CNI specification, a container can be considered a synonym for a Linux network namespace. The unit to which this corresponds depends on the specific container runtime implementation: for example, in implementations of the application container specification such as rkt, each pod runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a group of entities that are uniquely addressable and can communicate with each other. This can be an individual container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can be conceptually added to or removed from one or more networks. The CNI specification specifies a number of considerations for compliant plugins ("CNI plugins").

[0071] Because cRPD 25A is a cloud-native application, it supports installation using Kubernetes manifests or Helm charts. This includes the initial configuration of cRPD 25A as the control plane for virtualized router 32A, including configuration of routing protocols and one or more VPNs. cRPD can be orchestrated and configured in seconds, with all routing protocol adjacencies with the rest of the network up and running. Ongoing configuration changes throughout the lifecycle of cRPD 25A can be made via the CLI, Kubernetes manifests, NetConf, or Terraform.

[0072] By adopting the Kubernetes CNI framework, the virtualized router 32A can alleviate the traditional operational overhead incurred when using containerized devices instead of their physical counterparts. By exposing appropriate device interfaces, the virtualized router 32A can standardize the operating model of virtual devices to physical devices, thereby eliminating barriers to adoption in the operator's network operating environment. The virtualized router 32A can present a familiar routing device look and feel to any trained operations team. The virtualized router 32A has similar features and capabilities and a similar operating model as hardware-based platforms. Similarly, the domain controller can communicate with and control the virtualized router 32A using the same protocols it uses with any other router, such as Netconf / OpenConfig, gRPC, Path Computation Element Protocol (PCEP), or other interfaces.

[0073] According to the techniques of this disclosure, virtualized router 32A can participate in IS-IS, Open Shortest Path First (OSPF), BGP, and / or other internal or external routing protocols and exchange routing protocol messages with other routers, whether physical routers or virtualized routers 32B through 32X (collectively, "virtualized routers 32") residing on other hosts. In addition, MPLS can be used, typically based on segment routing (SR). This is done for two reasons: to allow traffic engineering when needed, and to support multi-tenancy through the use of VPNs (such as Layer 3 VPNs or EVPNs over MPLS).

[0074] A virtual private network (VPN) offered by a service provider consists of two topological areas: the provider's network and the customer's network. The customer's network is typically located at multiple physical sites and is also private (non-internet). The customer's site typically consists of a set of routers or other network equipment located in a single physical location. The provider's network (which operates over the public internet infrastructure) consists of routers that provide VPN services to the customer's network, as well as routers that provide other services. The provider's network connects the various customer sites, appearing to be a private network to both the customer and the provider.

[0075] To ensure that VPNs remain private and isolated from other VPNs and the public internet, the provider's network maintains a policy that keeps routing information from different VPNs separate. A provider can serve multiple VPNs as long as its policy keeps routes from different VPNs separate. Similarly, a customer site can belong to multiple VPNs as long as it keeps routes from different VPNs separate. Throughout this disclosure, references to customers or customer networks may not necessarily refer to independent entities or enterprises, but rather to data center tenants, sets of workloads connected via VPNs across a Layer 3 network, or some other logical grouping.

[0076] Although developed to operate across service provider networks and the public internet, VPN technology can be provided by any Layer 3 network, and similar terminology is used. The provider network is often referred to as a Layer 3 core network or simply a Layer 3 network or core network. Layer 3 VPNs operate at Layer 3 (the network layer) of the OSI model. A Layer 3 VPN consists of a set of customer networks connected through a core network. A peer-to-peer model is used to connect to customer sites, where the provider edge (PE) router learns customer routes when peering with the customer edge (CE) device. Public routing information is shared across the entire core network using Multiprotocol BGP (MP-BGP), and VPN traffic is forwarded between PE routers using MPLS. Layer 3 VPNs can be based on Rosen & Rekhter, "BGP / MPLS IP Virtual Private Networks (VPNs)" (Request for Comment 4364, Internet Engineering Task Force, Network Working Group, February 2006), which is incorporated herein by reference in its entirety.

[0077] Customer Edge (CE) devices connect to the provider network and may (or may not) provide reachability to other networks. PE devices are part of the Layer 3 core network and connect to one or more CE devices to provide VPN services. In a PE router, the IP routing table (also called the global routing table or default routing table) contains routes to the service provider or underlying network that are not included in the virtual routing and forwarding (VRF) table. Provider edge devices require IP routing tables to reach each other, and VRF tables are required to be able to reach all customer devices on a particular VPN. For example, a PE router with interface A to the CE router and interface B facing the core would place the interface A address in the VRF and the interface B address in the global IP routing table for the default VRF.

[0078] Virtual routing and forwarding (VRF) tables distinguish between routes for different VPNs, as well as VPN routes and provider / underlay routes on PE devices. These routes can include overlapping private network address spaces, customer-specific public routes, and provider routes on PE devices that are useful to customers. A VRF instance consists of one or more routing tables, derived forwarding tables, interfaces that use the forwarding tables, and policies and routing protocols that determine the contents of the forwarding tables. Because each instance is configured for a specific VPN, each VPN has separate tables, rules, and policies to control its operation. A separate VRF table is created for each VPN that has a connection to a CE device. The VRF table is populated with routes received from the directly connected CE device associated with the VRF instance, as well as routes received from other PE routers in the same VPN.

[0079] Layer 3 VPNs use a peer-to-peer routing model between directly connected PE routers and CE devices. This means that multiple hops over the Layer 3 core network are not required to connect a PE router and CE device pair. PE routers distribute routing information to all CE devices belonging to the same VPN, both locally and across the entire provider network, based on BGP route distinguishers. Each VPN has its own routing table, which is coordinated with the routing tables of the CEs and PE peers. A PE router can connect to more than one CE device; therefore, it maintains a common IP routing table and VRF table for each CE device attached to the VPN.

[0080] In a Layer 2 VPN, traffic is forwarded to the router in Layer 2 format. It is transported by MPLS over the Layer 3 core network and then converted back to Layer 2 format at the receiving site. You can configure different Layer 2 formats at the sending and receiving sites. Routing on a Layer 2 VPN is performed by the CE device, which must select the appropriate link to send traffic. The PE router receiving the traffic sends it across the Layer 3 core network to the PE router connected to the receiving CE device. The PE router does not need to store or process VPN routes. The PE router only needs to be configured to send the data to the appropriate tunnel. The PE router uses the Layer 2 VPN interface to transport traffic between CE devices. The VPN topology is determined by the policies configured on the PE router.

[0081] Ethernet VPN (EVPN) is a standards-based technology that provides virtual multipoint bridged connectivity between different Layer 2 domains over an IP or IP / MPLS backbone network. Like other VPN technologies such as Layer 3 VPN and Virtual Private LAN Service (VPLS), an EVPN instance is configured on a provider edge (PE) router to maintain logical service separation between customers. The PE router connects to the CE device, which can be a router, switch, or host. The PE routers then use Multiprotocol BGP (MP-BGP) to exchange reachability information, and encapsulated traffic is forwarded between the PE routers. The elements of the EVPN architecture are the same as other VPN technologies, such as Layer 3 VPN, with the EVPN MAC-VRF being a type of VRF used to store MAC addresses on the PE routers of an EVPN instance. An EVPN instance spans the PE devices participating in a particular EVPN and is therefore conceptually similar to a Layer 3 VPN. Additional information about EVPN can be found in Sajassi et al., “BGP MPLS-Based Ethernet VPN” (Request for Comments 7432, Internet Engineering Task Force, February 2015), which is incorporated herein by reference in its entirety.

[0082] The virtualized router 32A can operate as a provider edge (PE) router (i.e., a virtualized PE router). The virtualized router 32A can exchange VPN routes with other PE routers in the network via BGP, regardless of whether those other PEs are physical routers or virtualized routers 32 residing on other hosts. Each tenant can be placed in a separate VRF table on the virtualized router 32A, thereby providing the correct degree of isolation and security between tenants, just like a regular VPN service. This cleverly solves the problem that Kubernetes itself does not provide such isolation. VPN is a time-tested method for achieving multi-tenancy in a network and is trusted by many large companies around the world who purchase this service from their network service providers.

[0083] In this way, the virtualized router 32 can bring the full range of routing capabilities to the computing infrastructure hosting containerized applications. This allows the platform to fully participate in the operator's network routing system and facilitates multi-tenancy. It can provide the same familiar look, feel, and control plane interface as a hardware-based router, thereby providing a virtual private network for containerized applications.

[0084] In some cases, cRPD 25A can interface with two data planes: the kernel network stack and the DPDK-based virtual router for the compute node. CRPD 25A can leverage the kernel's network stack to establish routes specifically for the DPDK fast path. The routing information received by cRPD 25A can include both underlay routing information and overlay routing information. CRPD 25A can run routing protocols on vHost interfaces visible in the kernel, and cRPD 25A can install forwarding information base (FIB) updates corresponding to the routes learned by the Interior Gateway Protocol (IGP) (underlay) in the kernel FIB (e.g., to enable establishment of a multi-hop Interior Border Gateway Protocol (iBGP) session to these destinations). Simultaneously, virtual router 21A can notify cRPD 25a of the application pod interfaces created by CNI 17A for the compute node. CRPD 25A can advertise the reachability of these pod interfaces to the rest of the network as, for example, L3VPN network layer reachability information (NLRI). The corresponding Multi-Protocol Label Switching (MPLS) routes can be programmed on virtual router 21A because the next hop for these labels is a "POP forwarding" operation to the pod interfaces, and these interfaces are only visible in the virtual router. Similarly, the reachability information received via BGP L3VPN can be programmed only to virtual router 21A because the pods may need such reachability information for forwarding.

[0085] cRPD 25A includes a default VRF 28 (shown as "D.VRF 28") and VRFs 29A and 29B (collectively referred to as "VRF 29"). Default VRF 28 stores the global routing table. cRPD 25A programs forwarding information obtained from VRF 29 into virtual router 21A. In this way, virtual router 21A implements a VPN for VRF 29, which is shown as being included in both virtual router 21A and cRPD 25A.

[0086] cRPD 25A is configured to operate in host network mode, also known as local network. Therefore, cRPD 25A uses the network namespace and IP address(es) of its host (i.e., server 12A). cRPD 25A has visibility and access to network interfaces 30A-30B of NIC 13A, which are inserted into default VRF 28 and are viewed by cRPD 25A as core-facing interfaces. Interfaces 30A-30B are connected to switch fabric 14 and may be Ethernet interfaces. Interface 30 is viewed and used by cRPD 25A as a core-facing interface to provide VPNs because interface 30 can be used to transport VPN service traffic over a Layer 3 network comprised of one or more of switch fabric 14, IP fabric 20, service provider network 7, or public network 15.

[0087] According to the techniques of this disclosure, CNI 17A uses virtual network interface configuration data provided by network controller 24 to configure virtual network interface 26 (shown as "VNI 26") between pods 22A-22C and virtualized router 32A to enable communication between network pods 22 and virtual router 21A, thereby allowing the application of VPN service models typically deployed in service provider networks. Pods 22A-22C are effectively modeled as CE routers or host devices, and pods 22A-22C are able to interact with virtualized router 32A, which is modeled as a PE router, via virtual network interface 26. Virtual network interface 26 effectively becomes a link to an attachment circuit (for L3VPN) or an Ethernet segment (for EVPN) to connect pod 22 to VRF 29 of virtualized router 32A, which operates as a PE router.

[0088] Each of the virtual network interfaces 26 is inserted into at least one of the VRFs 29A to 29B of the virtualized router 32A. Figure 1In the example, pod 22A has a virtual network interface 26 with VRF 29A, pod 22B has a virtual network interface 26 with VRFs 29A and 29B, and pod 22C has a virtual network interface 26 with VRF 29B. Virtual network interfaces 26 can represent veth pairs, where each end of the veth pair is a separate device (e.g., a Linux / Unix device), with one end of each veth pair plugged into a VRF and one end plugged into a pod. A veth pair or one end of a veth pair is sometimes referred to as a "port." A virtual network interface can represent a macvlan network with a media access control (MAC) address assigned to pod 22 and virtual router 21A for communication between pod 22's containers and virtual router 21A. In the case of a DPDK-enabled virtual router 21A, virtual network interfaces 26 can each represent a DPDK (e.g., vhost) interface, where one end of the DPDK interface plugs into a VRF and one end plugs into a pod. In some examples, pod 22 can operate as a vhost server and virtual router 21A can operate as a vhost client to establish a DPDK interface. In some examples, virtual router 21A can operate as a vhost server, with pod 22 operating as a vhost client to establish a DPDK interface. For example, a virtual network interface can alternatively be referred to as a virtual machine interface (VMI), a pod interface, a container network interface, a tap interface, a veth interface, or simply a network interface (in a specific context).

[0089] As a result, cRPD 25A and any pod 22 can become routing protocol peers over virtual network interface 26. The IP address assigned to pod 22 is learned via routing protocol messages exchanged between pod 22 and cRPD 25A. cRPD 25A uses routing protocol peering sessions with routing protocol peers in the core Layer 3 network to execute routing protocols on one or more of core-facing interfaces 30 and exchange routing protocol messages. Routing protocol messages output by cRPD 25A may include IP address information for pod 22, or more specifically, the IP address of pod 22's virtual network interface 26 (which is different from the host IP address of server 12A). Thus, cRPD 25A can extend the coverage for VPNs implemented using VRF 29 to pod 22, thereby enabling the pod to operate as a host device in a CE or VPN.

[0090] In some examples, the same service IP address or shared anycast IP address is given to multiple pods for equal-cost multi-path (ECMP) or weighted ECMP. By using BGP to add a path to advertise this shared IP address to the network, the system can use these load balancing techniques at Layer 3. Existing Kubernetes load balancers provide load balancing based on L4 to L7 applications. While typical layer load balancing uses NAT / firewall or dedicated modules within the forwarding plane, these technologies can be used to implement load balancing using the network routing itself.

[0091] In some examples, a pod may have multiple interfaces to different cRPD VRFs 29, e.g., one for management traffic and another for data traffic. Figure 1 For example, pod 22B may use VRF 29B for management traffic and VRF 29A for data traffic.

[0092] Pod 22 (e.g., pod 22A) may be connected to some physical interface where it learns the IP addresses of other devices, such as user devices where the pod is implementing a mobile network gateway, or customer network subnets where the pod is implementing a CE router or gateway. To advertise these IP addresses to the network, pod 22A treats virtual network interface 26 with VRF 29 as an IP link and advertises routes to these IP addresses to cRPD 25A. CRPD 25A can then advertise reachability to these IP addresses through cRPD 25A and pod 22A as the next hop, again conforming to the VPN service model. cRPD 25A programs virtual router 21 with corresponding forwarding information obtained from VRF 29 and default VRF 28, and virtual router 21 forwards VPN traffic according to the VPN service model to implement the VPN.

[0093] For example, CRPD 25A can use multiple underlay tunnel types (including MPLS, SR-MPLS, SRv6, MPLSoUDP, MPLSoGRE, or IP-in-IP) to apply multiple different types of overlay networks / VPNs, including L3VPN or EVPN (Type-2 / Type-5).

[0094] CNI 17A can be combined with pod 22s Figure 126. Any container of a pod 22 may utilize (i.e., share) any virtual network interface of that pod. Orchestrator 23 may store or otherwise manage virtual network interface configuration data for application deployment. For example, orchestrator 23 may receive a specification for a containerized application (a "pod specification" in the context of Kubernetes) and a network attachment definition from a user, operator / administrator, or other machine system, and network controller 24 may derive configuration data from this information to configure VRF 29 and a default VRF 28; as well as configure virtual network interface 26.

[0095] For example, as part of the process of creating pod 22A, orchestrator 23 may request network controller 24 to create a virtual network interface for VRF 29A indicated in the pod specification and the network attachment definition referenced by the pod specification. According to the techniques of the present disclosure, the network attachment definition and pod specification conform to a new model that allows operators to specify a VPN in the network attachment definition and configure the pod as a member of the VPN along with the network interface specification. Pod 22 can have a different virtual network interface for each network it belongs to. Network controller 24 processes the request to generate interface configuration data for virtual network interface 26 of pod 22A. The interface configuration data may include a container or pod unique identifier and a list or other data structure that specifies network configuration data for configuring each of the virtual network interfaces. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0096] Network controller 24 sends interface configuration data to server 12A, and in some cases, more specifically to virtual router 21A. To configure the virtual network interface for pod 22A, orchestrator 23 may invoke CNI 17A. CNI 17A retrieves the interface configuration data from virtual router 21A and processes it. CNI 17A creates each virtual network interface specified in the interface configuration data. For example, CNI 17A may attach one end of a veth pair implementing virtual network interface 26 to virtual router 21A and the other end of the same veth pair to pod 22A. This can be accomplished using virtio-user.

[0097] The following is example interface configuration data for pod 22A for virtual network interface 26 .

[0098]

[0099]

[0100] Figure 2 6 is a block diagram of an example system that implements a VPN using a virtualized router with a containerized routing protocol process, according to the techniques of this disclosure. System 68 includes a core network 86. Core network 86 is a Layer 3 network and may include one or more switches and P (core) routers. Core network 86 may represent any combination of switching fabric 14, IP fabric 20, service provider network 7, public network 15, the Internet, or a separate Layer 3 network. Core network 86 also includes physical PE router 70. CE device 72 provides reachability to a customer network having network subnet P1.

[0101] The configuration of server 12A is Figure 1 22A for simplicity. The virtualized routers 32A, 32Y have a default VRF 28 and its corresponding host network interface.

[0102] PE router 70, virtualized router 32A, and virtualized router 32Y are configured with VRF 29A to implement VPN 80 over core network 86. VPN 80 may represent an L3VPN, EVPN, or another VPN that operates over a Layer 3 network and uses a routing protocol to advertise VPN routes. PE router 70, cRPD 25A, and cRPD 25Y negotiate a tunnel to form an overlay network for VPN 86.

[0103] According to the techniques of this disclosure, virtual network interface 26 has one end terminated by pod 22A and the other end terminated by VRF 29A. In some examples, cRPD 25A and pod 22A may execute one or more routing protocols to exchange "customer routes," i.e., routes to devices, which may be pod 22A or reachable via pod 22A operating as a CE router or gateway. Routing protocol message 73 includes a route, which may specify an IP prefix for a reachable destination and / or an IP address assigned to pod 22A as a next hop or destination. Routing protocol message 73 may identify an interior gateway protocol, such as BGP, Intermediate System to Intermediate System (IS-IS), or Open Shortest Path First (OSPF). cRPD 25A receives routing protocol message 73 and enters the route into VRF 29A. cRPD 25A then advertises the route as a VPN route for pod 22A in routing protocol message 74. Routing protocol message 74 may be an MP-BGP message that includes the VPN route as an NLRI. For example, the IP address for the advertised pod 22A may be a VPN-IPv4 address.

[0104] PE router 70 and cRPD 25Y receive copies of routing protocol message 73 and enter the VPN route into their corresponding VRF 29A. Using the VPN route, both virtualized router 32Y and PE router 70 can send overlay network packets to pod 22A via VPN 80. Virtualized router 32A receives such packets, identifies VRF 29A based on tunnel header information (e.g., MPLS label) in the packets, and forwards the inner packets to virtual network interface 26 having the IP address assigned to pod 22A. Pod 22A receives the inner packets.

[0105] In some examples, virtualized router 32A implements an EVPN instance and performs data plane learning to learn the MAC addresses of devices connected via virtual network interface 26. These can be advertised across core network 86 by cRPD 25A in routing protocol messages 74, which include EVPN routes, such as Type-2 or Type-5. Such MAC addresses can include the MAC address for pod 22A, as well as any other devices reachable via pod 22A, which operates similarly to a CE device. In such an example, virtual network interface 26 is effectively the Ethernet segment of the EVPN instance. Virtualized router 32 can then receive traffic on its core-facing interface destined for any of these MAC addresses and forward such traffic to pod 22 via virtual network interface 26.

[0106] Figure 3 is a block diagram of an example computing device (e.g., a host) according to the techniques described in this disclosure. Computing device 200 may represent a real or virtual server and may represent Figure 1 12. In this example, the computing device 200 includes a bus 242 that couples hardware components of the computing device 200 hardware environment. The bus 242 couples a network interface card (NIC) 230, a disk 246, and one or more microprocessors 210 (hereinafter referred to as "microprocessors 210"). The NIC 230 can support SR-IOV. In some cases, the front-end bus can couple the microprocessor 210 and the memory device 244. In some examples, the bus 242 can couple the memory device 244, the microprocessor 210, and the NIC 230. The bus 242 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to the bus 242. In some examples, components coupled to the bus 242 control DMA transfers between components coupled to the bus 242.

[0107] The microprocessor 210 may include one or more processors, each of which includes an independent execution unit for executing instructions in accordance with an instruction set architecture stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs), or may be combined within one or more multi-core processors (or "multi-core" processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0108] Disk 246 represents a computer-readable storage medium, which includes volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 210.

[0109] The main memory 244 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and can be accessed by the computer. The main memory 244 provides a physical address space consisting of addressable memory locations.

[0110] The network interface card (NIC) 230 includes one or more interfaces 232 configured to exchange packets using links of the underlying physical network. The interface 232 may include a port interface card having one or more network ports. The NIC 230 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 230 and other devices coupled to the bus 242 may read from / write to the NIC memory.

[0111] Memory 244, NIC 230, disk 246 and microprocessor 210 can provide an operating environment for a software stack including an operating system kernel 380 executed in kernel space. Kernel 380 can represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel or a Windows server operating system kernel, which is available from Microsoft. In some cases, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can include a virtual machine manager (VMM). The operating system including kernel 380 provides an execution environment for one or more processes in user space 245.

[0112] The kernel 380 includes a physical driver 225 for using the network interface card 230. The network interface card 230 may also implement SR-IOV to enable sharing of physical network functions (I / O) between one or more virtual execution elements, such as a container 229A or one or more virtual machines ( Figure 3 (not shown). Shared virtual devices such as virtual functions can provide dedicated resources, allowing each of the virtual execution elements to access the dedicated resources of NIC 230, so that NIC 230 appears to be a dedicated NIC to each of the virtual execution elements. A virtual function can represent a lightweight PCIe function that shares physical resources with the physical function used by the physical driver 225 and with other virtual functions. For a NIC 230 that supports SR-IOV, according to the SR-IOV standard, the NIC 230 can have thousands of available virtual functions, but for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0113] Computing device 200 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from physical switches to software or "virtual" routers, including virtual router 206A, that are physical servers coupled to the switching fabric. A virtual router may be a physical server (e.g., Figure 1A process or thread, or component thereof, executed by a server 12 of the server 12 that dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to decouple the virtual address of the endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executing. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transport packets within and across virtual networks over the physical network. The term "virtual router" as used herein can cover Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other device and / or software that is located on a host device and performs switching, bridging, or routing of packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more of the servers 12. In Figure 2 In the example computing device 200 of , the virtual router 206A executes within the kernel 380, but in some cases, the virtual router 206A can execute in user space as a DPDK-based virtual router within a hypervisor, a host operating system, a host application, or a virtual machine.

[0114] Virtual router 206A can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used for Kubernetes deployments of pods 202A to 202B (collectively, "pod 202"). Virtual router 206A can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 206A can perform network services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0115] Virtual router 206A can be executed as a kernel module or as a user space DPDK process (virtual router 206A is shown here as being in kernel 380). Virtual router agent 314 can also be executed in user space. Virtual router agent 314 has a connection to network controller 24 using a channel, which is used to download configuration and forwarding information. Virtual router agent 314 programs this forwarding state into the virtual router data (or "forwarding") plane represented by virtual router 206A. Virtual router 206A and virtual router agent 314 can be processes.

[0116] Virtual router 206A can be multi-threaded and executed on one or more processor cores. Virtual router 206A can include multiple queues. Virtual router 206A can implement a packet processing pipeline. The pipeline can be assembled by virtual router agent 314 based on the operations to be applied to the packet, from simplest to most complex. Virtual router 206A can maintain multiple instances of the forwarding library. Virtual router 206A can use RCU (read-copy-update) locks to access and update tables.

[0117] To send packets to other compute nodes or switches, the virtual router 206A uses one or more physical interfaces 232. Typically, the virtual router 206A exchanges overlay packets with a workload, such as a VM or pod 202 (in Figure 3 ). Virtual router 206A may have multiple virtual network interfaces (e.g., vifs). These interfaces may include kernel interface vhost0, which is used to exchange packets with the host operating system; and interface pkt0 with virtual router agent 314, which is used to obtain forwarding state from the network controller and send exception packets upstream. One or more virtual network interfaces may correspond to one or more physical network interfaces 232.

[0118] The other virtual network interfaces of the virtual router 206A are used to exchange packets with the workload. Figure 3 . The virtual network interfaces 212 and 213 may be any of the aforementioned types of virtual interfaces. In some cases, the virtual network interfaces 212 and 213 are tap interfaces.

[0119] cRPD 324 can indicate Figure 1 pod 202B is provided with the host IP address of computing device 200 on the underlying network.

[0120] Pod 202B may be assigned its own virtual Layer 3 (L3) IP address for sending and receiving communications, but may not be aware of the IP address of the computing device 200 on which pod 202B executes. Thus, the virtual L3 (network) address may be different from the logical address of the underlying physical computer system (e.g., computing device 200). The virtual network address may be specified in the pod specification or selected by network controller 24 from a pool of addresses in the VPN.

[0121] The computing device 200 includes a virtual router agent 314 that controls the overlay of the virtual network of the computing device 200 and coordinates the routing of data packets within the computing device 200. Generally, the virtual router agent 314 communicates with the network controller 24 for the virtualization infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 200, and more specifically the virtual router 206A, and the virtual network interfaces 212. By configuring the virtual router 206A based on information received from the network controller 24, the virtual router agent 314 can support the configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0122] In one example, a network packet (e.g., a Layer 3 (L3) IP packet or a Layer 2 (L2) Ethernet packet generated or consumed by a container 229A within a virtual network domain) can be encapsulated in another packet (e.g., another IP or Ethernet packet) that is transmitted by a physical network. The packets transmitted in the virtual network may be referred to herein as "inner packets," while the physical network packets may be referred to herein as "outer packets" or "tunnel packets." The encapsulation and / or decapsulation of the virtual network packets within the physical network packets may be performed by the virtual router 206A. This function is referred to herein as tunneling and can be used to create one or more overlay networks. In addition to IPinIP, other example tunneling protocols that can be used include IP over Generic Routing Encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), and the like. The virtual router 206A performs tunnel encapsulation / decapsulation on packets originating from / destined for any container of the pod 202, and the virtual router 206A exchanges packets with the pod 202 via the bridge of the bus 242 and / or NIC 230.

[0123] As described above, the network controller 24 can provide a logically centralized controller to facilitate the operation of one or more virtual networks. For example, the network controller 24 can maintain a routing information base, such as one or more routing tables that store routing information for a physical network and one or more overlay networks. The virtual router 206A implements a virtual routing and forwarding instance (VRF) 222A for a VPN in which the virtual router 206A operates as a tunnel endpoint. Typically, the VRF stores forwarding information for the corresponding virtual network and identifies where data packets are to be forwarded and whether the packets are to be encapsulated in a tunnel protocol, such as using a tunnel header, which can include one or more headers for different layers of the virtual network protocol stack. The VRF can include a network forwarding table that stores routing and forwarding information for the virtual network.

[0124] NIC 230 may receive a tunnel packet. Virtual router 206A processes the tunnel packet to determine the virtual networks of the source and destination endpoints of the inner packet based on the tunnel encapsulation header. Virtual router 206A may strip the Layer 2 header and the tunnel encapsulation header to forward only the inner packet internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN label or an MPLS label, that indicates a virtual network, such as the virtual network corresponding to VRF 222A. VRF 222A may include forwarding information for the inner packet. For example, VRF 222A may map the destination Layer 3 address for the inner packet to virtual network interface 212. In response, VRF 222A forwards the inner packet to pod 202A via virtual network interface 212.

[0125] Container 229A may also provide an internal packet as a source virtual network endpoint. For example, container 229A may generate another Layer 3 internal packet destined for a destination virtual network endpoint executed by another computing device (i.e., not computing device 200) or destined for a container. Container 229A may send the Layer 3 internal packet to virtual router 206A via virtual network interface 212 attached to VRF 222A.

[0126] Virtual router 206A receives the inner packet and the Layer 2 header and determines the virtual network for the inner packet. Virtual router 206A may use any of the aforementioned virtual network interface implementation technologies (e.g., macvlan, veth, etc.) to determine the virtual network. Virtual router 206A uses VRF 222A corresponding to the virtual network of the inner packet to generate an outer header for the inner packet. The outer header includes an outer IP header for the overlay tunnel and a tunnel encapsulation header identifying the virtual network. Virtual router 206A encapsulates the inner packet with the outer header. Virtual router 206A may encapsulate the tunnel packet with a new Layer 2 header that has a destination Layer 2 address associated with a device external to computing device 200, such as TOR switch 16 or one of servers 12. If external to computing device 200, virtual router 206A outputs the tunnel packet with the new Layer 2 header to NIC 230 using physical function 221. NIC 230 outputs the packet on the outbound interface. If the destination is another virtual network endpoint executing on computing device 200 , virtual router 206A routes the packet to the appropriate one of virtual network interfaces 212 , 213 .

[0127] In some examples, a controller of computing device 200 (e.g., Figure 1The network controller 24 of pod 202 configures a default route in each pod in pod 202 so that virtual machine 224 uses virtual router 206A as the initial next hop for outbound packets. In some examples, NIC 230 is configured with one or more forwarding rules so that all packets received from virtual machine 224 are switched to virtual router 206A.

[0128] Pod 202A includes one or more application containers 229A. Pod 202B includes an instance of cRPD 324. Container platform 204 includes container runtime 208, orchestration agent 310, service agent 211, and CNI 312.

[0129] The container engine 208 includes code that is executable by the microprocessor 210. The container runtime 208 can be one or more computer processes. The container engine 208 runs containerized applications in the form of containers 229A. The container engine 208 can represent Dockert, rkt, or other container engines for managing containers. Typically, the container engine 208 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 310, the container engine 208 can obtain images and instantiate them into executable containers in pods 202A to 202B.

[0130] The service agent 211 includes code executable by the microprocessor 210. The service agent 211 can be one or more computer processes. The service agent 211 monitors the addition and removal of services and endpoint objects, and maintains the network configuration of the computing device 200 to ensure communication between pods and containers, for example, using services. The service agent 211 can also manage ip address tables (iptables) to capture traffic to the virtual IP address and port of the service, and redirect the traffic to the proxy port of the pod supported by the proxy. The service agent 211 can represent the kube-proxy of the slave node of the Kubernetes cluster. In some examples, the container platform 204 does not include the service agent 211 or the service agent 211 is disabled to support the configuration of the virtual router 206A and the pod 202 by the CNI 312.

[0131] The orchestration agent 310 includes code that is executable by the microprocessor 210. The orchestration agent 310 may be one or more computer processes. The orchestration agent 310 may represent a kubelet of a slave node of a Kubernetes cluster. The orchestration agent 310 is an agent of the orchestrator, for example, Figure 1Orchestrator 23 receives container specification data for the container and ensures that the container is executed by computing device 200. The container specification data can be in the form of a manifest file sent from orchestrator 23 to orchestration agent 310, or received indirectly via a command line interface, an HTTP endpoint, or an HTTP server. The container specification data can be a pod specification (e.g., a PodSpec—a YAML (yet another markup language) or JSON object describing a pod) for one of pods 202 of container 229. Based on the container specification data, orchestration agent 310 instructs container engine 208 to obtain and instantiate a container image for container 229 for computing device 200 to execute container 229.

[0132] Orchestration agent 310 instantiates or otherwise calls CNI 312 to configure one or more virtual network interfaces for each pod in pods 202. For example, orchestration agent 310 receives container specification data for pod 202A and instructs container engine 208 to create pod 202A using container 229A based on the container specification data for pod 202A. Orchestration agent 310 also calls CNI 312 to configure virtual network interface 212 of a virtual network corresponding to VRF 222A for pod 202A. In this example, pod 202A is a virtual network endpoint of the virtual network corresponding to VRF 222A.

[0133] CNI 312 can retrieve interface configuration data for configuring the virtual network interfaces of pod 202. Virtual router agent 314 operates as a virtual network control plane module that enables network controller 24 to configure virtual router 206A. Unlike the orchestration control plane (including container platform 204 for slave nodes and master node(s), such as orchestrator 23), which manages the provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 for slave nodes and virtual router agent 314) manages the configuration of virtual networks implemented in part in the data plane by virtual router 206A of the slave nodes. Virtual router agent 314 transmits the interface configuration data of the virtual network interfaces to CNI 312, enabling the orchestration control plane element (i.e., CNI 312) to configure the virtual network interfaces based on the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this may enable CNI 312 to obtain interface configuration data for multiple virtual network interfaces of a pod and configure the multiple virtual network interfaces, which may reduce the communication and resource overhead inherent in invoking a separate CNI 312 to configure each virtual network interface.

[0134] According to the techniques of this disclosure, network controller 24 configures VRF 222A for the VPN via virtual agent 314. Container 229 and cRPD 324 execute one or more routing protocols 280. Container 229 and cRPD 324 can be routing protocol peers for routing protocol 280. Routing protocol 280 can include interior gateway protocols such as OSPF, IS-IS, Routing Information Protocol (RIP), internal BGP (IBGP), or other protocols. Container 229A can use routing protocol messages from one of routing protocols 280 to advertise a route to the IP address assigned to container 229A (here, sharing the IP address assigned to pod 202B). For example, such a message can be an OSPF link state advertisement, a RIP response message, a BGP update message, or other routing protocol message advertising a route to the IP address assigned to container 229A. Virtual router 206A forwards the routing protocol message received at VRF 222A to cRPD 324.

[0135] CRPD 324 processes the routing protocol message advertising the route for the IP address assigned to container 229A and imports the IP address into VRF 222A as a VPN-IPv4 address. CRPD 324 then advertises itself as the next hop for the VPN-IPv4 address in routing protocol messages sent to other PE routers. For example, cRPD 324 may output an MP-BGP update message specifying its IP address (also the host IP address) as the next hop for the VPN-IPv4 address. The other PE routers receive the routing protocol message and update their corresponding VRFs for the VPN to forward VPN traffic destined for container 229A to computing device 200 (and cRPD 324). Because cRPD 324 operates in host network mode, specific TCP / UDP ports may be assigned to cRPD 324 for its communications.

[0136] In this way, container 229A and cRPD 324 model the familiar PE-CE VPN service model for the service provider network and can extend the overlay network to container 229A. Because cRPD 324 uses standard routing protocols and VPNs, this model is compatible with physical PE routers that also use these standards, which provides technical advantages over existing CNIs for container-configured networking in virtual computing infrastructures.

[0137] CNI 312 can program cRPD 324 via the management interface of cRPD 324. In some examples, the orchestrator (via orchestration agent 310) pushes the initial configuration template as a ConfigMap to CNI 312. The ConfigMap can be a Kubernetes ConfigMap.

[0138] When pod 202B, which includes cRPD 324, starts, CNI 312 (which deploys pod 202B) also operates as a controller to process the initial configuration template and generate configuration data for cRPD 324. The configuration data may conform to a management interface format, such as Netconf, CLI, or proprietary.

[0139] The purpose of configuration templates is to allow users / operators an easy way to customize the configuration to suit their network needs (e.g., specific IPG to use, BGP families that should be enabled, policy configuration, etc.). CNI 312 replaces variables in the initial configuration template with values ​​specific to computing device 200 (e.g., loopback address, IS-IS address, SR node index, etc.), allowing users / operators to define a common configuration template that can be applied to instances of cRPD 324 running on multiple nodes in a cluster.

[0140] Furthermore, the controller can listen to events to detect updates to the ConfigMap and update the base configuration at runtime without having to restart / redeploy cRPD 324 and virtual router 206A, which may cause service interruption.

[0141] The advantage of using templates over Custom Resource Definitions (CRDs) is that, given the large number of configuration parameters supported by cRPD 324, CRD object definitions can become unwieldy and difficult to develop, and building the corresponding software logic to handle these objects is time consuming. This can limit the set of models that use CRDs and require software changes when new features / network models are added. (For example, the addition of segment routing that is not defined by an existing CRD.) Some configurable features (such as routing policies) will take a lot of effort to express as CRDs and software. A CRD-based approach will simply copy the same CLI knobs that already exist in cRPD 324, albeit in a different format (e.g., json, yaml, Netconf, openconfig, yang) and without much benefit. A template-based approach provides access to all CLI knobs without the need for software updates and / or development work. An example initial configuration template with variables and some logic is as follows:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154] Figure 4 is a block diagram of an example computing device operating as an instance of an orchestrator master node for a cluster of virtualized computing infrastructure in accordance with the techniques of this disclosure. Figure 4 The computing device 1300 may represent one or more real or virtual servers. Thus, the computing device 1300 may, in some cases, implement one or more master nodes for a corresponding cluster.

[0155] Although shown and described as being executed by a single computing device 1300, the scheduler 1322, API server 1320, network controller manager 1326, SDN controller 1324, SDN controller manager 1325, and configuration repository 1328 may be distributed among multiple computing devices 300 that make up a computing system or hardware / server cluster. In other words, each computing device in the multiple computing devices 1300 may provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, API server 1320, network controller manager 1326, SDN controller 1324, SDN controller manager 1325, or configuration repository 1328. The SDN controller 1324 may represent Figure 1 Scheduler 1322, API server 1320, controller manager 1326, and SDN controller manager 1325 may implement an example instance of orchestrator 23. SDN controller manager 1325 may represent an example implementation of a Kubernetes cloud controller manager or Kube-Manager.

[0156] In this example, computing device 1300 includes a bus 1342 that couples hardware components of the computing device 1300 hardware environment. Bus 1342 couples a network interface card (NIC) 1330, a disk 1346, and one or more microprocessors 1310 (hereinafter referred to as "microprocessors 1310"). In some cases, a front-side bus can couple microprocessors 1310 and memory devices 1344. In some examples, bus 1342 can couple memory devices 1344, microprocessors 1310, and NIC 1330. Bus 1342 can represent a peripheral component interface (PCI) Express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

[0157] The microprocessor 1310 may include one or more processors, each of which includes an independent execution unit for executing instructions in accordance with an instruction set architecture stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs), or may be combined within one or more multi-core processors (or "multi-core" processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0158] Disk 1346 represents a computer-readable storage medium, which includes volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 1310.

[0159] The main memory 1344 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and can be accessed by the computer. The main memory 1344 provides a physical address space consisting of addressable memory locations.

[0160] The network interface card (NIC) 1330 includes one or more interfaces 3132 configured to exchange packets using links of the underlying physical network. The interfaces 3132 may include a port interface card having one or more network ports. The NIC 1330 may also include, for example, on-card memory for storing packet data. Direct memory access transfers between the NIC 1330 and other devices coupled to the bus 1342 may read from / write to the NIC memory.

[0161] Memory 1344, NIC 1330, disk 1346 and microprocessor 1310 can provide an operating environment for a software stack including an operating system kernel 1314 executed in kernel space. Kernel 1314 can represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel or Windows server operating system kernel, which is available from Microsoft. In some cases, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from a virtual machine, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can include a virtual machine manager (VMM). The operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. Kernel 1314 includes a physical driver 1325 for using the network interface card 230.

[0162] The computing device 1300 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from physical switches to software or "virtual" routers of physical servers coupled to the switching fabric, e.g., Figure 2 The computing device 1300 may use one or more dedicated virtual networks to configure the slave nodes of the cluster.

[0163] API server 1320, scheduler 1322, controller manager 1326, and configuration store may implement a master node for a cluster and may alternatively be referred to as a “master component.” The cluster may be a Kubernetes cluster, and the master node is a Kubernetes master node, in which case the master component is a Kubernetes master component.

[0164] The API server 1320 includes code executable by the microprocessor 1310. The API server 1320 can be one or more computer processes. The API server 1320 validates and configures data for objects such as virtual execution elements (e.g., pods of containers), services, and replication controllers. A service can be an abstraction that defines a logical set of pods and policies for accessing the pods. The set of pods that implement the service is selected based on the service definition. A service can be partially implemented as a load balancer, or otherwise include a load balancer. The API server 1320 can implement a representational state transfer (REST) ​​interface to process REST operations and provide a front end to the shared state of the corresponding cluster stored in the configuration repository 1328. The API server 1320 can authenticate and authorize requests. The API server 1320 communicates with other components to instantiate virtual execution elements in the computing infrastructure 8. The API server 1320 can represent a Kubernetes API server.

[0165] Configuration repository 1328 is the backing store for all cluster data. Cluster data can include cluster state and configuration data. Configuration data can also provide a backend for service discovery and / or provide locking services. Configuration repository 1328 can be implemented as a key-value store. Configuration repository 1328 can be a central database or a distributed database. Configuration repository 1328 can represent an etcd repository. Configuration repository 1328 can also represent a Kubernetes configuration repository.

[0166] Scheduler 1322 includes code executable by microprocessor 1310. Scheduler 1322 can be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., container pods) and selects slave nodes on which the virtual execution elements will run. Scheduler 1322 can select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, location, etc. Scheduler 1322 can represent a Kubernetes scheduler.

[0167] Typically, API server 1320 can call scheduler 1322 to schedule a virtual execution element, which can select a slave node and return an identifier of the selected slave node to API server 1320. API server 1320 can write the identifier in association with the virtual execution element to configuration repository 1328. API server 1320 can call orchestration agent 310 for the selected slave node, which can cause the container engine 208 of the selected slave node to obtain the virtual execution element from the storage server and create the virtual execution element on the slave node. The orchestration agent 310 of the selected slave node can update the state of the virtual execution element to API server 1320, which saves this new state to configuration repository 1328. In this way, computing device 1300 instantiates a new virtual execution element in computing infrastructure 8.

[0168] Controller manager 1326 includes code executable by microprocessor 1310. Controller manager 1326 can be one or more computer processes. Controller manager 1326 can be embedded in the core control loop to monitor the shared state of the cluster by getting notifications from API server 1320. Controller manager 1326 can attempt to move the state of the cluster towards a desired state. Example controllers (not shown) managed by controller manager 1326 can include replication controllers, endpoint controllers, namespace controllers, and service account controllers. Controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminated pod garbage collection, cascade deletion garbage collection, node garbage collection, etc. Controller manager 1326 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0169] The SDN controller 1324 includes code that is executable by the microprocessor 1310. The SDN controller 1324 may include one or more computer processes. The SDN controller 1324 may represent Figure 1 1324. The SDN controller 1324 may be a logically centralized but physically distributed software-defined network (SDN) controller responsible for providing management, control, and analytics capabilities for the virtualized network. In particular, the SDN controller 1324 may be a logically centralized control plane and management plane for the computing infrastructure 8 and orchestrate vRouter for one or more slave nodes.

[0170] The SDN controller 1324 can provide cloud networking for computing architectures operating on top of the network infrastructure. Cloud networking can include private clouds for enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) for cloud service providers (CSPs). Private clouds, VPCs, and IaaS use cases can involve multi-tenant virtualized data centers, such as Figure 1 In this scenario, multiple tenants in a data center share the same physical resources (physical servers, physical storage, and physical networks). Each tenant is assigned its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; and virtual networks). These logical resources are isolated from each other unless specifically permitted by security policy. The data center's virtual networks can also interconnect with physical IP VPNs or L2 VPNs.

[0171] The SDN controller 1324 can provide network function virtualization (NFV) to networks such as business edge networks, broadband subscriber management edge networks, and mobile edge networks. NFV involves the orchestration and management of network functions such as firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, and wide area network (WAN) optimization in virtual machines, containers, or other virtual execution elements rather than physical hardware devices. The main drivers of network service virtualization in this market are time to market and cost optimization.

[0172] The SDN controller 1324 programs the network infrastructure elements to create a virtual network and may create interface configurations for virtual network interfaces of the virtual network.

[0173] Additional information regarding example network controllers can be found in International Application No. PCT / US2013 / 044378 and U.S. Patent Application No. 14 / 226,509, which are incorporated herein by reference.

[0174] The SDN controller manager 1325 includes code executable by the microprocessor 1310. The SDN controller manager 1325 can be one or more computer processes. The SDN controller manager 1325 operates as an interface between orchestration-oriented elements (e.g., the scheduler 1322, the API server 1320, the controller manager 1326, and the configuration repository 1328) and the SDN controller 1324. Generally speaking, the SDN controller manager 1325 monitors new objects (e.g., pods and services) in the cluster. The SDN controller manager 1325 can isolate pods in a virtual network and connect pods to services.

[0175] The SDN controller manager 1325 can be executed as a container of the master node of the cluster. In some cases, using the SDN controller manager 1325 enables disabling the service proxy (e.g., Kubernetes kube-proxy) of the slave node, so that all pod connections can be implemented using a virtual router, as described herein.

[0176] The SDN controller manager 1325 can use the controller framework for the orchestration platform to listen to (or otherwise monitor) changes to objects defined in the API and add annotations to some of these objects. An annotation can be a label or other identifier that specifies an object attribute (e.g., "Virtual Network Green"). The SDN controller manager 1325 can use an interface to the SDN controller 1324 to define network objects such as virtual networks, virtual network interfaces, and access control policies to create a network solution for the application. The SDN controller 1324 can implement the network solution in the computing infrastructure by, for example, configuring one or more virtual networks and virtual network interfaces in a virtual router.

[0177] According to the technology of this disclosure, the following example deployment configuration of the application includes a pod and the network information of the pod, which references a NetworkAttachmentDefinition for a VPN. A NetworkAttachmentDefinition is a custom resource for managing the network attachment definition of a pod and is defined in a network attachment definition specification.

[0178]

[0179]

[0180] This metadata information is replicated to each pod replica created by controller manager 1326. When SDN controller manager 1325 is notified of these pods, it can program the VRF (the "blue-net" NetworkAttachmentDefinition referenced in the example above) for the VPN listed in the annotation in cRPD 324 (executed by any node where a pod replica will be deployed). The host's CNI 312 will create a virtual network interface for each pod replica (e.g., pod 202A) with a unique private virtual network address (here, taken from the IPv4 1.1.1.2 / 30 subnet) for the VRF. Within the pod, the interface will be named "net1," and CNI 312 will configure the other end of the interface in cRPD 324. The interface will be configured with ipconfig information and any routes to be installed within the pod. In this way, these techniques allow operators to use NetworkAttachmentDefinition to represent routing instances and bind pods to NetworkAttachmentDefinitions to orchestrate VPNs according to the PE-CE VPN service model. For example, the new "cni-args" object enables CNI 312 to configure a pod similar to configuring a CE router or host, including "routes" for subnets reachable via the pod.

[0181] Based on these specifications of the pod and NetworkAttachmentDefinition, the SDN controller manager 1325 converts the specifications into configuration data that can be consumed by the cRPD 324. This configuration data can be used to configure routing instances and cause the cRPD 324 to advertise routing prefixes, establish protocol sessions, and other aspects of control plane signaling, all of which can be triggered by the SDN controller manager 1325 configuring the cRPD 324 via its interface. In some examples, the SDN controller 1324 can configure the cRPD 324 via its interface.

[0182] The following is an example of a NetworkAttachmentDefinition. A NetworkAttachmentDefinition is a custom resource object that describes how to attach a pod to an object's logical or physical network reference. The following example conforms to the pattern of a NetworkAttachmentDefinition that has been extended to orchestrate a VPN according to the techniques of this disclosure. The orchestrator 23 can receive the NetworkAttachmentDefinition and create a corresponding object instance. The SDN controller manager 1325 can monitor or listen for events regarding the NetworkAttachmentDefinition object instance and configure the virtual computing infrastructure to implement the intent in the NetworkAttachmentDefinition, including configuring the cRPD 324 and the virtualized PE router 32.

[0183] The following blue-l3vpn.yaml defines an L3VPN named "blue-net" that can be referenced by the pod specification above. This L3VPN specifies the VRF name and VRF route targets, which the SDN controller manager 1325 can use to create import policies for advertised routing information and export policies for advertised routing information, including routes for any pods attached to that VRF:

[0184]

[0185]

[0186] The following blue-evpn.yaml defines an EVPN named "blue-net" that can be referenced by the pod specification above. This EVPN specifies the VRF name and VRF route target, which the SDN controller manager 1325 can use to create import policies for advertised routing information and export policies for advertised MAC addresses (here, EVPN type 5 messages), including for any pods attached to this VRF. blue-evpn.yaml also specifies the VxLAN virtual network identifier for EVPN-VXLAN:

[0187]

[0188]

[0189] The following vswitch.yaml includes multiple NetworkAttachmentDefinitions for various bridge domains for L2VPN, thereby enabling the virtualized router 35 to operate as a virtual switch:

[0190]

[0191]

[0192]

[0193]

[0194]

[0195] The following pod specification references the NetworkAttachmentDefinitions above. The SDN controller manager 1325 creates corresponding interfaces for any pod replicas to use the bridge domain defined in the NetworkAttachmentDefinitions in the vswitch.yaml above. In this example, the IP addresses for the pod replicas are allocated from the IPAM in the NetworkAttachmentDefinitions.

[0196]

[0197]

[0198]

[0199] Additional techniques according to the present disclosure are described below. Contrail is an example network controller architecture. Contrail CNI may be a CNI developed for Contrail. A Contrail controller may be an example of an SDN controller described in this disclosure, such as network controller 24.

[0200] Figure 5 1440 can be deployed as a microservice in Docker, coreOS (rkt), or other container platforms.

[0201] cRPD 1440 includes a management interface 1400, which can represent one or more of a command line interface (CLI), Netconf, Secure Shell (SSH), PCEP, a Simple Network Management Protocol (SNMP) interface, or other management interfaces. Management interface 1400 can support YANG, OpenConfig, or other configuration data formats. Management interface 1400 can receive configuration data from automation system 1420 and output telemetry data to telemetry system 1422. The CNI can program cRPD 1440 via management interface 1400, as described in further detail above.

[0202] cRPD 1440 implements routing protocols 1402, which may include BGP, OSPF, IS-IS, LDP, segment routing, and may receive static routes for programming from a controller or automation system (represented by programmability 1424). cRPD 1440 includes routing infrastructure 1404 to support routing protocols 1402. Routing infrastructure 1404 may include a routing information base (RIB), a RIB manager, a label information base (LIB), and a LIB manager. Routing infrastructure 1404 may implement bidirectional forwarding detection (BFD). cRPD 1440 includes a forwarding information base (FIB) adaptation layer 1406 to integrate cRPD 1440 into the data plane by enabling configuration of forwarding information in the data plane. For example, FIB adaptation layer 1406 may implement gRPC, Netlink, or rtsock interfaces to program a vRouter (e.g., a DPDK-based vRouter). The FIB adaptation layer 1406 may implement another type of interface to program a vRouter, kernel-based vSwitch, SmartNIC, network processor, ASIC-based forwarding chip, or other data plane.

[0203] Figure 6is a block diagram illustrating an example vRouter agent according to techniques of the present disclosure. The vRouter agent 314 includes a gRPC server 520 for exchanging data with a cRPD 324 (gRPC client) via a generic interface 340. The APIs of the gRPC server 520 include a virtual machine interface (VMI) API 530 for exchanging virtual network interface data and requests, a configuration API 532 for exchanging configuration data and requests, and a routing API 534 for exchanging routes and requests—including for enabling the cRPD 324 to program routes to the vRouter 206A via the vRouter agent 314. A synchronization module 544 programs the vRouter 206A with virtual network interfaces (e.g., portions of a veth pair or virtio-vhost interface between a DPDK pod and the DPDK-based vRouter 206A) and programs the vRouter 206A with routing information.

[0204] Interface 540 may represent a data structure storing data describing a virtual network interface for an application pod executing on a server executing vRouter agent 314. Port service 542 listens for requests from CNI 312, such as a request to add a new "port" for an application pod. Port service 542 may convert the request into a subscription request to cRPD 324 via interface 340, which retrieves virtual network interface configuration information from cRPD 324. Port service 542 may be implemented using a REST server.

[0205] In this manner, the vRouter agent 314 provides a common interface 340 (e.g., Figure 3 ), for overlay traffic originating from or destined for application pods on servers. This generic interface 340 can be implemented by any controller, routing protocol process, or other agent because it relies on gRPC rather than a proprietary interface.

[0206] Figure 7 is a conceptual diagram illustrating example operations 1480 for programming vRouter forwarding information in accordance with techniques of this disclosure. cRPD 324 determines that the forwarding information should be updated and performs the update by adding, changing, or deleting routes and adjusting next hops in the forwarding plane. The translator component of cRPD 324 can translate instructions for programming the hardware forwarding plane into equivalent instructions for virtual router agent 314 and invoke the northbound interface of virtual router agent 314 (provided by gRPC server 520 in some examples), which programs the forwarding information base of virtual router 206A to perform packet forwarding accordingly.

[0207] Figure 8 is a conceptual diagram illustrating example operations 1500 for configuring and advertising a virtual network interface in a server with a virtualized router, in accordance with techniques of this disclosure.

[0208] To provide reachability through the tunnel, cRPD 324 can be configured with the necessary protocols (IGP, BGP, etc.) vRouter 206A will manage the network interface facing the physical core - routing traffic will be sent and received through this interface, but cRPD 324 is started in host network mode and has access to the physical interface.

[0209] cRPD 324 can be configured with the necessary protocols via the network controller using NETCONF and other configuration protocols. cRPD 324 will establish adjacencies for various protocols; using its routing protocols to learn and advertise routing information (including reachability to application containers). cRPD 324 needs to program this learned routing information into vRouter agent 314. vRouter 206A will provide a bidirectional gRPC channel 340 for communication back and forth with cRPD 324. Data objects (routes, VRFs, interfaces, etc.) can be modeled in protocol buffers.

[0210] like Figure 8 As shown, the cRPD-based CNI 312 can create a veth equivalent pair for each application container in the application container when notified by the Kubernetes / orchestration agent 310. CNI 312 is responsible for assigning IP addresses to these interfaces. One end of the veth equivalent pair will terminate in the interface of the application container. As for the other end, CNI 312 will request vRouter 206A to begin monitoring this end of the veth equivalent interface. This facilitates all tunneled traffic flowing from the physical port to the application container. CNI 312 can rely on an IP address manager (IPAM) to obtain IP address information. CNI 312 can therefore obtain IP address information from the virtual router agent 314 received from the SDN controller manager 1325, from a DHCP server, static allocation, or from a custom allocator. In some examples, the virtual network interfaces of at least some application containers are DPDK-enabled. Therefore, a virtio-net interface can be attached to the application container (or pod), and the vhost-net end attached to the DPDK-enabled vRouter 206A.

[0211] In some examples, since vRouter 206A now manages one side of these veth interfaces, these interfaces are not visible to kernel 380. Consequently, these interfaces are not visible to cRPD 324, and therefore cRPD 324 cannot announce reachability information to the outside world. To address this issue, a veth-equivalent interface can be made visible to cRPD 324. This is not an interface that cRPD 324 can run routing protocols on (since this requires the use of kernel facilities such as sockets, the TCP / IP stack, etc.). This interface is used to inform cRPD 324 of the reachability it needs to advertise.

[0212] In some cases, vRouter 206A can notify cRPD 324 of this interface directly. This may be preferable because it is similar in some ways to how current VRFs are handled in cRPD 324. Additionally, if the interface is down, vRouter 206A can notify cRPD 324. If cRPD is up, vRouter 206A can let cRPD know about all interfaces it is monitoring again.

[0213] Using these interfaces, cRPD 324 can advertise MPLS reachability to the application container. cRPD 324 can advertise vrf-table-labels or per-nexthop labels (where next-hop represents a veth equivalent) or per-prefix labels. When this MPLS route can be installed on vRouter 206A, vRouter agent 314 will have the ability to convert the veth equivalent to the actual veth interface.

[0214] The following is another example sequence of operations between various components according to one or more aspects of the present disclosure:

[0215] I. Interactions between various components to create initial connections

[0216] 1. cRPD 324 324 learns about the kernel in the form of vhost0 through netlink.

[0217] 2. The domain / network controller configures the (IGP and BGP) protocol configurations via Netconf or other management protocols on the cRPD 324. Alternatively, an operator can perform this operation manually using the CLI on the cRPD 324.

[0218] 3. cRPD 324 establishes an IGP adjacency and learns network reachability and segment routing information.

[0219] 4. cRPD 324 programs this reachability information into the host kernel via the existing netlink channel.

[0220] 5. cRPD 324 establishes a BGP session using the connectivity learned via IGP.

[0221] 6. BGP learns L3VPN routes through this BGP session.

[0222] 7. cRPD 324 learns the workload interface from virtual router 206 A. cRPD 324 creates a subnet (eg, / 30) and interface route ( / 32) corresponding to the interface.

[0223] 8. CNI 312 configures the workload interface under a specific VRF on cRPD 324. For DPDK interfaces, CNI 312 also interacts with virtual router 206A to start the virtio-net / vhost-net interface between the pod and virtual router 206A. Virtual router 206A then notifies cRPD 324 (via a gRPC-based FIB channel) that the interface has become available.

[0224] 9. cRPD 324 sends the vrf interface mapping to virtual router 206A.

[0225] 10. cRPD 324 imports the l3vpn routes received in step 6 into the appropriate vrfs and resolves them through the SR tunnel in step 3.

[0226] 11. cRPD 324 installs these tunnel routes (in the vrf.inet(6).0 table) into virtual router 206A. (Virtual router 206A needs to do the translation from vhost0 to physical port 0).

[0227] 12. Additionally, cRPD 324 advertises the l3vpn route for the vrf route from step 7.

[0228] 13. cRPD 324 installs the mpls.0 route with pop label and forward to workload semantics.

[0229] II. Interactions between various components when the workload interface is closed.

[0230] 1. When CNI 312 triggers a request to remove an interface, virtual router 206A sends a request to cRPD

[0231] 324 Notification interface deleted.

[0232] 2. cRPD 324 deletes subnet and interface routes.

[0233] 3. cRPD 324 sends a reachability recall to the workload.

[0234] 4. cRPD 324 deletes the mpls.0 tunnel route with pop and forwarding to workload semantics from virtual router 206A.

[0235] III. Interactions between various components removed by VRF.

[0236] 1. cRPD 324 internally deletes the l3vpn route received in step 6 (interaction I.) from the appropriate vrf.

[0237] 2. cRPD 324 sends deletes for these routes (in the vrf.inet(6).0 table) with the tunnel next hop to the vRouter.

[0238] 3. In addition, cRPD 324 withdraws the l3vpn route from step 8 for the vrf route (Interaction I.)

[0239] 4. cRPD 324 sends a delete to virtual router 206A to obtain the mpls.0 route with pop label and forward to workload semantics.

[0240] In one aspect of the present disclosure, a set of software components provides CNI functionality that addresses the networking requirements unique to cloud-native 5G network environments. The software components include a containerized routing protocol daemon (cRPD) to support a network service mesh (NSM) architecture. The set of software components supports the NSM architecture and can provide additional capabilities such as hybrid networking (between physical and virtual infrastructure), direct reachability to pods from outside the compute node cluster, for example, via advertising via protocols such as BGP, and dynamic tunneling using various technologies such as MPLS, SRv6, IP-IP / VxLAN / GRE, and IPsec.

[0241] In this use case, the 5G O-RAN network can be deployed using cloud-native technologies and follows the 5G split, where the DU (distributed unit) and CSR (cell site router) are virtualized and the compute nodes in the pod run as instances of the virtualized router. This set of software components can operate as a cell site router to provide L3 reachability for the mid-range of the 5G network.

[0242] The software component uses cRPD 324 to distribute Layer 3 (L3) network reachability information for pods not only within the cluster but also outside the cluster. cRPD also programs the data plane on each compute node. To achieve better network packet I / O performance, DU applications can run in application pods to bypass the kernel network stack and abstractions and thereby send / receive packets directly from the physical NIC using, for example, a zero-copy mechanism. The Data Plane Development Kit (DPDK) is one such framework, and a DPDK-based virtual router can be used as a user-space data plane that leverages DPDK for high forwarding performance for this purpose.

[0243] Software components can include a DPDK-based virtual router to support DPDK applications. The CNI plugin manages the DPDK configuration for the application and programs the virtual router. This can include establishing a vhost control channel and allocating IP (e.g., both IPv4 and IPv6) and MAC addresses, advertising pod IP addresses, and detecting and reversing routing when a pod is considered down or removed.

[0244] Most existing use cases for cRPD either provide control-plane-only routing functionality, such as BGP route reflectors, or drive the forwarding plane: a kernel-based or ASIC-based forwarding plane on white-box platforms. The rise of containers and cloud computing has led to the need for container orchestration platforms to manage the lifecycle of containers. Kubernetes (K8s) is an orchestration platform for running containerized applications in a cluster computing environment. It provides automatic deployment, scaling, networking, and management of containerized applications. A K8s pod consists of one or more containers representing an application instance and is the smallest unit that K8s can handle. All containers in a pod share the same network namespace. The Container Network Interface (CNI) provides networking for application pods in Kubernetes. It is responsible for establishing pod interfaces, address allocation and networking between pods in a k8s cluster, and network isolation between different workloads.

[0245] CNI 312 may be CNI functionality as well as capabilities useful for supporting a Network Service Mesh (NSM) architecture.

[0246] While many CNI solutions exist, catering primarily to data center use cases, the technology described in this article can address the unique networking requirements of cloud-native 5G environments by interacting with cRPD 324 to provide NSM capabilities. CNI supporting NSM architecture provides additional capabilities such as hybrid networking (between physical and virtual infrastructure), direct reachability to pods from outside the cluster, for example, through advertising via protocols such as BGP, and dynamic tunneling using various technologies such as MPLS, SRv6, IP-IP / VxLAN / GRE, IPsec, etc.

[0247] 5G O-RAN networks can be deployed using cloud-native technologies and follow the 5G 7.2 split, where the DU (distributed unit) and CSR (cell site router) are virtualized and run on servers. The CNI 312 acts as a cell site router to provide L3 reachability for the mid-range.

[0248] cRPD 324 can distribute the layer 3 network reachability information of pods not only within the Kubernetes cluster (in a Kubernetes deployment) but also outside the cluster.cRPD 324 is also responsible for programming the corresponding forwarding plane virtual router 206A on each compute node / server.

[0249] vRouter 206A supports DPDK applications. CNI 312 is responsible for establishing the DPDK configuration for the application and programming vrouter 206A. This includes establishing the vhost control channel and allocating IP (both IPv4 and IPv6) and MAC addresses, advertising pod IP addresses, and detecting and reversing routing when a pod is considered down or deleted.

[0250] Other features provided by aspects described in this disclosure include:

[0251] Use L3 routing protocols such as BGP and IS-IS to advertise the network reachability of pods

[0252] Use BGP to advertise reachability within and outside the cluster

[0253] Network namespaces (supported with L3VPN and EVPN Type-5)

[0254] ACLs and network policies for security and QoS

[0255] Supported tunnels: MPLS, SR-MPLS, SRv6, SR-MPLSoIPv6, VxLAN, IPIP, GRE

[0256] Dynamic tunnels using BGP signaling

[0257] IPsec tunnel for business encryption

[0258] · Network policies for providing security and isolation

[0259] Integrates with the DPDK version of the vRouter 206A for improved forwarding performance, encapsulation, packet filtering, and QoS

[0260] As a delivery of a set of containers that can be deployed in K8s using a YAML specification file.

[0261] The set of components that make up CNI 312, virtualized routers, SDN controller 1324, and SDN controller manager 1325 can be considered Kubernetes CNI, referred to herein as Platter CNI - this is different from the CNI 312 plug-in, but rather includes the network solution of the container orchestration system and can implement the container network interface.

[0262] The CNI 312 and virtualized router may provide one or more of the following features:

[0263] Network namespace: Application pods should be reachable via non-default network namespaces or routing instances implemented using L3VPN.

[0264] IPv6 underlay: Support IPv6 underlay as required by the use case. IGP protocols should be able to exchange IPv6 routes. BGP protocol sessions should be established using IPv6 addresses.

[0265] IPv6 overlay: IPv6 overlay is supported by assigning IPv6 addresses to pods and advertising them via BGP.

[0266] BGP: The disk runs on each node in the k8s cluster and uses BGP to advertise pod reachability to the network. Routes advertised via BGP can carry SRv6 label stacks or other tunnel encapsulation attributes.

[0267] IGP: Each node will participate in the IGP layer to learn about the reachability of other BGP peers and route reflectors. IS-IS can be used to advertise host / node addresses to the network.

[0268] SRv6: Pod services can be transmitted through SRv6 tunnels. IS-IS is used to obtain segment routing SID information.

[0269] vrouter-dpdk: To achieve better packet I / O performance, vrouter-dpdk is supported as the data plane in some cases. This includes allocating IP and MAC addresses, generating the appropriate DPDK configuration for the application, vrouter programming, and route advertisement.

[0270] Deployment in a K8s cluster is performed using a YAML file that contains various details about all containers that are part of the CNI: the repository where the image is hosted, the initialization order, environment variables, configuration, and license key information. Typically, the YAML file will be customized to suit Kubernetes deployments. An example YAML configuration for the platter CNI (platter.yml) is provided below:

[0271]

[0272]

[0273]

[0274]

[0275]

[0276]

[0277]

[0278]

[0279]

[0280] pod YAML configuration

[0281] The pod YAML should export the pod UID as KUBERNETES_POD_UID which may be required by DPDK applications.

[0282] Notes

[0283] Comments can be used to set the following optional configuration details required by the disk:

[0284] VRF name: used to add the interface to the routing instance

[0285] VRF target: used to advertise instance routes via BGP

[0286] Example Configuration

[0287] An example application YAML configuration with environment variables and annotations is shown below. A different schema for the pod YAML named "du" was provided earlier in this disclosure.

[0288]

[0289]

[0290] Node configuration

[0291] The initial version of the platter will use a statically defined pod network configuration that is loaded using a config map file. This config map is read during the platter CNI installation and stored as a file on each node. The configuration file contains detailed information about each application on a per-interface basis, including IP addresses and routing instance details. Each time the platter CNI is called to establish a pod interface, the pod name and interface name are used as keys to find the interface configuration details needed to bring up the interface.

[0292] Example node configuration diagram

[0293] A sample application YAML configuration (platter-node-config.yaml) with environment variables and comments looks like this:

[0294]

[0295]

[0296]

[0297]

[0298] Figure 9 is a flow chart illustrating an example mode of operation of a computing device. Operation 900 is about Figure 2 200, but may be performed by other computing devices of the present disclosure. Virtual router 206A implements a data plane for the virtualized PE router (902). CNI 312 configures a virtual network interface (e.g., a DPDK-enabled channel) to enable communication between virtual router 206A and the containerized application represented by container 229A (904). Containerized RPD 324 implements a control plane for the virtualized PE router (906). Containerized RPD 324 obtains a MAC address or IP address sent by the containerized application (908). Containerized RPD 324 executes a routing protocol to generate and output a route that includes at least one of the MAC address or IP address to implement a VPN for the containerized application (910).

[0299] Figure 10 1 is a flow chart illustrating an example mode of operation of a container orchestration platform, such as orchestrator 23. Orchestrator 23 deploys a containerized application (e.g., server 12A) to a compute node (702). In response to deploying the containerized application to the compute node, orchestrator 23 configures a virtual routing and forwarding (VRF) instance in the compute node to implement a virtual private network (VPN) for the containerized application (704).

[0300] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units, or components can be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit system can be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0301] If implemented in hardware, the present disclosure may relate to an apparatus, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0302] Computer-readable media may form part of a computer program product, which may include packaging materials. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0303] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0304] The code or instructions may be software and / or firmware executed by a processing circuit system including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure may be provided within a software module or a hardware module.

Claims

1. A computing device for a virtual private network, comprising: processing circuitry and a storage medium, wherein the processing circuitry has access to the storage medium; a containerized application stored in the storage medium and executed by the processing circuitry; a virtual router stored in the storage medium and executed by the processing circuitry, the virtual router being configured to implement a data plane for a virtualized provider edge (PE) router configured to provide a virtual private network (VPN) for the containerized application; as well as a containerized routing protocol process, stored in the storage medium, executed by the processing circuitry, and configured to implement a control plane for the virtualized PE router, the containerized routing protocol process being configured to: Obtaining one of a media access control (MAC) address or an internet protocol (IP) address of the containerized application; generating and outputting the following route according to a first routing protocol, wherein the route includes at least one of the MAC address or the IP address, so as to implement the VPN for the containerized application; as well as The virtual router is programmed with forwarding information for the route, the forwarding information causing the virtual router to forward data packets to the containerized application.

2. The computing device of claim 1, wherein the containerized routing protocol process is configured to establish a routing protocol peering session with a physical router external to the computing device according to the first routing protocol.

3. The computing device of claim 1 , wherein the containerized routing protocol process is configured to establish a routing protocol peering session according to the first routing protocol with a different virtualized PE router external to the computing device. 4 . The computing device of claim 1 , wherein the VPN is an Ethernet VPN (EVPN), and the route comprises an EVPN Type-2 route or an EVPN Type-5 route. 5 . The computing device of claim 1 , wherein the VPN is a Layer 3 VPN (L3 VPN), and the route is included in a Multi-Protocol Border Gateway Protocol (MP-BGP) message.

6. The computing device of any one of claims 1 to 5, wherein the containerized routing protocol process is configured to: receive a routing protocol message from the containerized application according to a second routing protocol, the routing protocol message indicating the IP address.

7. The computing device according to any one of claims 1 to 5, wherein the containerized routing protocol process is configured to operate in host network mode to access a host network interface of the computing device, and The containerized routing protocol process outputs the routes via the host network interface.

8. The computing device according to any one of claims 1 to 5, further comprising: A virtual routing and forwarding instance VRF for the VPN, the VRF being configured by the containerized routing protocol process, The VRF is configured with endpoints of a virtual network interface.

9. The computing device according to any one of claims 1 to 5, further comprising: a container network interface (CNI) plugin, executed by the processing circuitry and configured to: Configuring a virtual network address for the containerized application for a virtual network interface in the virtual router; as well as The containerized routing protocol process is provided with an IP address for the virtual network interface.

10. The computing device according to claim 9, The containerized routing protocol process is configured to establish a routing protocol peering session with the containerized application using the IP address for the virtual network interface according to a second routing protocol.

11. The computing device according to any one of claims 1 to 5, The containerized routing protocol process is configured to execute a second routing protocol to establish a routing protocol peer session with the containerized application to receive the IP address sent via the virtual network interface.

12. A container orchestration system for a computing node cluster including computing nodes, the container orchestration system comprising: processing circuitry and storage media, wherein the processing circuitry is configured to cause the container orchestration system to: Deploying containerized applications to the computing nodes; as well as In response to deploying the containerized application to the compute node, configuring the containerized routing protocol process with virtual routing and forwarding (VRFs) to generate routes to implement a virtual private network (VPN) for the containerized application by communicating with or transmitting configuration data to be consumed by the containerized routing protocol process, and instructing the containerized routing protocol process to: program a virtual router with forwarding information for the routes, the virtual router being deployed to the compute node and configured to implement a data plane for a virtualization provider edge (PE) router, the forwarding information causing the virtual router to forward data packets to the containerized application.

13. The container orchestration system according to claim 12, wherein a network attachment definition specification defines the VRF instance, and The container orchestration system is configured to configure the VRF instance in response to determining that a specification for the containerized application references a network attachment definition in the network attachment definition specification.

14. The container orchestration system according to any one of claims 12 to 13, wherein the container orchestration system is configured to: The route with the Internet Protocol (IP) prefix is ​​sent to a container network interface (CNI) executing on the compute node, so that the CNI programs the containerized application with the IP prefix to indicate a network reachable via the containerized application, wherein a specification for the containerized application includes the IP prefix.

15. The container orchestration system according to any one of claims 12 to 13, wherein the container orchestration system is configured to: The route with the media access control (MAC) address is sent to a container network interface (CNI) executing on the compute node so that the CNI programs the containerized application with the MAC address, wherein a specification for the containerized application includes the MAC address.

16. A non-transitory computer-readable medium comprising instructions that, when executed by processing circuitry of a computing device, cause the computing device to: executing a virtual router configured to implement a data plane for a virtualized provider edge (PE) router configured to provide a virtual private network (VPN) for containerized applications; and Executing a containerized routing protocol process, the containerized routing protocol process configured to implement a control plane for the virtualized PE router to: Obtaining one of a Media Access Control (MAC) address or an Internet Protocol (IP) address of the containerized application, and executing a first routing protocol configured to generate and output a route including at least one of the MAC address or the IP address to implement a virtual private network (VPN) for the containerized application, and The virtual router is programmed with forwarding information for the route, the forwarding information causing the virtual router to forward data packets to the containerized application.

17. The non-transitory computer-readable medium of claim 16, wherein the containerized routing protocol process is configured to implement a control plane for the virtualized PE router, the control plane: A routing protocol message is received from the containerized application according to a second routing protocol, the routing protocol message indicating the IP address.

18. The non-transitory computer readable medium of claim 16, wherein the containerized routing protocol process is configured to operate in host network mode to access a host network interface of the computing device, and The containerized routing protocol process is configured to implement the control plane for the virtualized PE router, and the control plane outputs the routes via the host network interface.

19. The non-transitory computer-readable medium of claim 16, wherein the containerized routing protocol process is configured to implement the control plane for the virtualized PE router using a virtual routing and forwarding instance (VRF), wherein the VRF is configured with endpoints of a virtual network interface.

20. The non-transitory computer-readable medium of claim 16, wherein the instructions, when executed by processing circuitry of a computing device, cause the computing device to: Execute the Container Network Interface (CNI) plugin, which is configured to: configuring a virtual network address for the containerized application for a virtual network interface in the virtual router; and The containerized routing protocol process is provided with an IP address for the virtual network interface.

Images