Systems, methods, and storage media for virtualizing computing infrastructure

Abstract

This disclosure relates to systems, methods, and storage media for virtualized computing infrastructure. Generally, this disclosure describes techniques for providing a hybrid data plane, which may include a kernel-based data plane and a Data Plane Development Kit (DPDK)-based data plane. An exemplary system includes a DPDK-based virtual router and a kernel network stack configured to send and receive data packets via a physical network interface, and the kernel network stack configured to perform tunneling processing on data packets destined for containerized applications and received by the DPDK-based virtual router via the physical interface.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 813,027, filed July 15, 2022, and U.S. Provisional Patent Application No. 63 / 366,619, filed June 17, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to virtualized computing infrastructure, and more specifically to a hybrid data plane for containerized platforms. Background Technology

[0004] In a typical cloud data center environment, there exists a large collection of interconnected servers providing computing and / or storage capacity to run a variety of applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., the data center's customers). A data center may, for example, host all infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers utilize subscriber support facilities located in various physical host facilities to provide infrastructure distributed globally.

[0005] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have widely utilized virtualized environments, in which virtual machines, also referred to in this paper as virtual execution elements, such as virtual machines or containers, are deployed on and run on the underlying computing platform of physical computing devices.

[0006] Virtualization within a data center, or any environment comprising one or more servers, offers several advantages. One advantage is the significant efficiency improvements it can provide. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become increasingly powerful, making virtualization easier and more efficient. A second advantage is the significant control virtualization provides over the computing infrastructure. As physical computing resources become alternative resources, such as in cloud-based computing environments, the provisioning and management of computing infrastructure becomes easier. Therefore, enterprise IT personnel often prefer virtualized computing clusters in data centers because of the management advantages they offer, in addition to the efficiency and increased return on investment (ROI) provided by virtualization.

[0007] Containerization is a virtualization scheme based on operating system-level virtualization. A container is a lightweight, portable execution unit for applications that are isolated from each other and from the host. Such isolation systems refer to containers, such as those provided by the open-source DOCKER container application or by CoreOS Rkt (“Rocket”). Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit a separate operating system, instead providing an application suite and application-specific libraries. Typically, containers are executed by the host as isolated user-space instances and can share the operating system and common libraries with other containers running on the host. Therefore, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces to communicate on corresponding virtual networks.

[0008] Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to a container image and executed as a single lightweight package on any host or virtual machine that supports the underlying container architecture. In this way, containers solve the problem of how to make software work across different computing environments. Containers offer the promise of consistent operation from one computing environment to another virtual or physical environment.

[0009] Leveraging the inherently lightweight nature of containers, a single host can typically support more container instances than a traditional virtual machine (VM). Short-lived containers can often be created and moved more efficiently than VMs, and they can also be managed as logically related groups of components (e.g., containerized workloads, sometimes called "pods" on some orchestration platforms like Kubernetes). These container characteristics influence the requirements of container networking solutions: the network must be flexible and scalable. VMs, containers, and bare-metal servers may need to coexist in the same compute environment, enabling communication between different deployments of applications. Container networks should also be able to work with the various types of orchestration platforms used to deploy containerized applications.

[0010] The computing infrastructure that manages the deployment and execution of applications can involve two main roles: (1) orchestration—for automating the deployment, scaling, and operation of applications across host clusters and providing the computing infrastructure, which may include container-centric computing infrastructure; and (2) network management—for creating virtual networks within the network infrastructure to enable packetized communication between applications running in virtual execution environments such as containers or VMs, as well as between applications running in traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0011] Generally, this disclosure describes techniques for providing a hybrid data plane for compute nodes, comprising two distinct data planes, each capable of supporting different communication protocols. Existing systems typically provide a single data plane, or, if multiple, disjoint data planes. A single data plane may not meet the evolving needs of containerized applications on a server. For example, some containerized applications—deployable using pods—may require the high throughput and low latency offered by the Data Plane Development Kit (DPDK) data plane, while other pods may require encapsulation, services, and / or routing not supported by the DPDK data plane. In a disjoint model, one set of architecture (kernel-oriented) interfaces is managed by the kernel data plane, while another set is managed by the DPDK data plane. In this model, traffic from kernel pod interfaces is intended to be forwarded via the kernel kernel interfaces, and traffic from DPDK pods is intended to be forwarded via the DPDK architecture interfaces. However, achieving such separation for the corresponding control plane and routing protocols is, if not impossible, highly complex. Furthermore, there is no guarantee that inbound or outbound packets will use the interface corresponding to the intended data plane. This can lead to inefficient packet processing or even crashes within the data plane. As an example, a compute node implementing a disjoint data plane might have a first port owned by the kernel data plane and a second port owned by the DPDK data plane. Because compute nodes typically cannot inform external nodes which port to use to communicate with pods on the compute node, packets that should be processed by the DPDK data plane might arrive at the interface owned by the kernel (e.g., the first port). When this happens, the kernel must forward the packets to the DPDK data plane, resulting in packet overhead and inefficient processing. Furthermore, the DPDK data plane typically processes network traffic at a much higher rate than the kernel data plane. As a result, the kernel can receive packets intended for the DPDK data plane at a high rate. This can lead to significant CPU utilization or resource memory issues, which can further degrade overall system performance.

[0012] Given the above, a single data plane or disjoint data planes provided by existing systems may not meet the changing needs of different workloads executed on the server. Furthermore, disjoint data planes may lead to inefficient handling of network traffic destined for the DPDK data plane that is reaching the kernel data plane.

[0013] The hybrid data plane described in this paper supports cloud-native routers in a manner that offers technical advantages over existing systems. For example, a technical advantage of the hybrid data plane model described herein is that it avoids failures or inefficiencies by configuring a DPDK data plane in compute nodes to have all the necessary infrastructure interfaces. Pods or other workloads can be configured to utilize one or both of the kernel data plane and the DPDK data plane, which are provided as part of the hybrid data plane. The pod can be configured to use the data plane best suited to the pod's communication needs (i.e., the communication needs of the pod containing the containerized application deployed in that pod). For example, pods requiring high throughput and / or low latency can be configured to utilize the DPDK data plane, while pods requiring more complex encapsulation not supported by the DPDK data plane can utilize the kernel-based data plane. The kernel data plane can perform encapsulation and forward the encapsulated packets to the DPDK data plane for delivery via the interfaces owned by the DPDK data plane. Because the DPDK data plane is allocated all the physical interfaces managed by the cloud-native router, incoming packets are received by the DPDK data plane. The DPDK data plane may include logic for determining whether an incoming data packet should be processed by the DPDK data plane or transmitted to the kernel data plane for decapsulation or other processing not supported by the DPDK data plane.

[0014] The hybrid data plane disclosed herein may include a kernel data plane and a DPDK data plane. Pods or other containerized workloads can be configured to utilize either of the hybrid data planes that best meets their requirements. As an example, a pod requiring high data transfer rates and low latency can be configured to use the DPDK data plane. However, the DPDK data plane typically does not support all the different types of encapsulation and upper layers that the kernel data plane can support, such as IPsec support, SRv6, IP-IP, EVPN-VxLAN, L3VPN over SRv6, etc. Therefore, pods or other container units requiring such capabilities can be configured to use the kernel data plane. However, for the control plane, it may be difficult to effectively control both the kernel data plane and the DPDK data plane on the same compute node.

[0015] In some aspects, all physical interfaces of compute nodes managed by cloud-native routers are assigned to DPDK-enabled virtual routers. Pods configured to use DPDK can directly use the interfaces provided by the DPDK-enabled virtual routers. Pods requiring capabilities not supported by DPDK-enabled virtual routers can be configured to use kernel data plane interfaces. In at least some aspects, this technology can involve assigning all network interfaces of a server to one or more DPDK-based virtual routers on the server. DPDK-enabled virtual routers can provide data communication and routing capabilities for pods configured to use, for example, DPDK, and pods can benefit from this aspect of providing a hybrid data plane that offers high-speed, low-latency communication. This technology can further involve a kernel configured with a network stack to provide capabilities that may be lacking in DPDK-enabled virtual routers. For example, a kernel (or a virtual router configured as a kernel module) can provide a data plane supporting IPsec, SRv6, IP-IP, EVPN-VxLAN, L3VPN over SRv6, etc. Pods requiring this support can be configured with virtual network interfaces with a kernel to send data to the kernel for processing by the kernel network stack. The kernel is configured to route all traffic to DPDK-enabled routers, which act as pathways for traffic received from the kernel. That is, the DPDK-enabled virtual routers perform little or no processing on packets and send them through one of the physical interfaces assigned to the DPDK-based virtual routers. Therefore, the kernel data plane and the DPDK-enabled data plane are combined to form a hybrid data plane.

[0016] As described above, a hybrid data plane can be implemented using a container orchestration platform to support cloud-native routers. A Container Networking Interface (CNI) is a networking solution for application containers and is a runtime executable that assists in configuring the interface between the container and other components of the compute device (“host”) that hosts the container. This compute device can be a member of a pod. The compute device can also be alternatively referred to as a “compute node” or a “server.” CNIs typically assign network addresses (e.g., IP addresses) to network interfaces and can also add interface-related routes, such as routes for the default gateway and one or more name servers.

[0017] A virtual router is a software entity that provides packet routing and forwarding data plane functionality on a compute node. The compute node can be a centrally orchestrated and provisioned VM or container host. Virtual routers can work with SDN controllers to create upper-layer networks by exchanging routing, configuration, and other data. Virtual routers can run as Linux kernel modules or as DPDK-based processes. DPDK allows virtual routers to process more packets per second than is possible when running as a kernel module. The virtual router's data plane can be containerized. Combined, containerized cRPD and containerized DPDK-based virtual routers can thus form a fully functional containerized router.

[0018] Compute nodes can be used to implement portions of (5G) cellular networks using a cloud-native open radio access network (“O-RAN” or “open RAN”) architecture. The cloud can be built using containers and Kubernetes. Cell site router functions can be implemented on compute nodes hosting distributed unit (DU) 5G functions as containerized applications. That is, DU functions can be implemented as Kubernetes pods on these compute nodes. At a very high level, the DU function will consume RAN traffic, process it, and send it to control unit (CU) functions hosted in the data center via tunneling technology.

[0019] To meet the routing and forwarding performance requirements that may be involved in 5G network usage scenarios, compute nodes can be configured to use a cloud-native router with a hybrid data plane. This cloud-native router operates as a control plane, with cRPD running on the compute node, and is configured to include both kernel-based routing and DPDK-based virtual routers. Kernel-based routing can handle routing and encapsulation that DPDK-based virtual routers cannot, while DPDK-based virtual routers provide a fast path data plane for pods that do not require more complex encapsulation.

[0020] In the example, a system is described that includes processing circuitry; a containerized application; a physical network interface; a virtual router based on a Data Plane Development Kit (DPDK) configured to execute on the processing circuitry to send and receive data packets via the physical network interface; and a kernel network stack configured to execute on the processing circuitry to perform tunneling processing on data packets destined for the containerized application and received by the DPDK-based virtual router via the physical interface.

[0021] In another example, a method is described, comprising: a virtual router based on a Data Plane Development Kit (DPDK) executed by a processing circuit receiving a first data packet destined for a containerized application from a physical interface; the DPDK-based virtual router determining whether a kernel network stack executed by the processing circuit should perform tunneling processing on the first data packet; and in response to determining that the kernel network stack should perform tunneling processing on the first data packet, the DPDK-based virtual router forwarding the first data packet to the kernel network stack.

[0022] In another example, a computer-readable storage medium is described that includes computer-executable instructions, which, when executed, cause one or more processors executing a DPDK-based virtual router to perform operations including: receiving a first data packet destined for a containerized application; determining whether a kernel network stack executed by the one or more processors should perform tunneling processing for the first data packet; and forwarding the first data packet to the kernel network stack in response to determining that the kernel network stack should perform tunneling processing for the first data packet.

[0023] Details of one or more examples of the invention are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0024] Figure 1A A block diagram of an example mobile network system is illustrated in accordance with the technology described in this disclosure.

[0025] Figure 1B This is a block diagram illustrating an example computing infrastructure that can implement the techniques described in this article.

[0026] Figure 2 A more detailed illustration of the technology according to this disclosure. Figure 1A and Figure 1B A block diagram of an example implementation of a part of a network system.

[0027] Figure 3 This is a block diagram illustrating an example server according to the technology disclosed herein.

[0028] Figure 4 This is a block diagram illustrating an example server with example control and data service flows within the server according to the technology of this disclosure.

[0029] Figure 5 This is a conceptual diagram depicting a sequence of operations in a virtual router that leads to routing planning, according to an exemplary aspect of this disclosure.

[0030] Figure 6AThis is a block diagram of an example computing device (e.g., a host) based on the techniques described in this disclosure.

[0031] Figure 6B yes Figure 6A The example computing device is shown in the block diagram as an example of an orchestrator master node in a cluster of virtualized computing infrastructure.

[0032] Figure 7 This is a block diagram of an example computing device (e.g., a server) including a hybrid data plane according to the techniques described in this disclosure.

[0033] Figure 8A and Figure 8B This is a sequence diagram illustrating an example of processing outbound and inbound data packets according to the technology of this disclosure.

[0034] Figure 9 This is a sample deployment specification for a pod based on the technology disclosed herein.

[0035] Figure 10 This is a flowchart illustrating the operation of a DPDK-based virtual router.

[0036] Throughout the specification and drawings, the same reference numerals denote the same elements. Detailed Implementation

[0037] "Container-based" or "OS-based" virtualization refers to the virtualization of multiple isolated systems running on a single machine (virtual or physical). These isolated systems are represented by containers, such as those provided by the Kubernetes Container Runtime Interface - Open Container Initiative (CRI-O), containerization, open-source Docker container applications, or those provided by CoreOS Rkt ("Rocket"). Like virtual machines, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit a separate operating system and only provides an application suite and application-specific libraries. Typically, containers are executed as isolated user-space instances by a host (e.g., one of the network devices 107) and can share the operating system and common libraries with other containers running on the host. Therefore, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces to communicate on corresponding virtual networks.

[0038] In some examples, containers are managed by their host kernel to allow for the throttling and prioritization of resources (CPU, memory, block I / O, network, etc.) without requiring any virtual machines to be started. In some examples, containers can be deployed based on Linux Containers (LXC), an operating system-level virtualization approach for running multiple isolated Linux systems (containers) on a controlling host using a single Linux kernel.

[0039] 5G employs a cloud-native approach, where functional blocks are broken down into microservices. These microservices are deployed as containers on an x86 platform orchestrated by Kubernetes (K8s). This includes 5G core control plane functions such as Access and Mobility Management (AMF) and Session Management (SMF), RAN control plane functions such as CU-CP, Service Management and Orchestration (SMO), near real-time and non-real-time Radio Intelligent Controller (RIC), and even some data plane functions such as CU-DP and DU.

[0040] Kubernetes networking between Pods is achieved via a plugin called the Container Networking Interface (CNI). However, the networking capabilities of a typical CNI are rather basic and unsuitable when the containerized networking functions served by the CNI play a critical role in telecommunications networks. As discussed in this article, Cloud Native Routers (CNRs) offer a better fit for these situations. A Cloud Native Router is a containerized router that allows x86- or ARM-based hosts to become first-class members of a network routing system, involving protocols such as Intermediate System to Intermediate System (IS-IS) and Border Gateway Protocol (BGP), and providing transport and multi-tenancy based on Multiprotocol Label Switching / Segmentation Routing (MPLS / SR). In other words, the platform is not an adjunct to the network (such as a Customer Edge (CE) router) but can operate as a Provider Edge (PE) router.

[0041] Cloud-local routers can offer one or more advantages over conventional routers. A router has a control plane and a forwarding plane. The control plane participates in dynamic routing protocols and exchanges routing information with other routers in the network. It downloads the results to the forwarding plane in the form of prefixes, next hops, and associated SR / MPLS labels. The implementation described in this paper is modular in the sense that the control plane is unaware of the exact details of how the forwarding plane is implemented. In hardware routers, the forwarding plane can be based on a custom ASIC. In contrast, cloud-local routers are virtualized routers. However, the routing protocol software is functionally similar in both cases. This means that cloud-local routers benefit from the same highly integrated and robust protocol implementations as hardware-based routers that support some of the world's largest networks.

[0042] Cloud-native routers use a containerized routing protocol daemon (cRPD) control plane and a virtual router (VRB) forwarding plane to deliver high-performance networking over small coverage areas, using packages that function similarly to non-virtual routers, and Physical Network Functions (PNFs). The forwarding plane can be implemented via a choice of DPDK, a Linux kernel, or a smart NIC. Complete integration provides a Kubernetes CNI-compliant package that can be deployed in a Kubernetes environment (e.g., with Multus enabled).

[0043] Servers can be Kubernetes workers / compute nodes (or “minions”). Pods can be vertically connected to cloud-native routers. Pods may require multiple network interfaces, facilitated in some cases by the Multus meta-CNI. Each of these interfaces can be mapped to a different Layer 3 VPN on the cloud-native router to support multiple network slices. The CNI can dynamically add or remove interfaces between the pod and the virtual router when triggered by Kubernetes pod events. The CNI can also dynamically update the cRPD control plane container, leveraging host routes and corresponding Layer 3 VPN mappings for each pod interface, in the form of route delimiters and route targets. Layer 3 VPNs can be implemented using Virtual Routers and Forwarding Instances (VRFs). The cRPD control plane is then programmed to the virtual router forwarding plane via the gRPC interface. In this way, the cloud-native router is introduced into the data path, supporting F1 interfaces to CUs running in edge or regional DC sites. While primarily described in relation to O-RAN applications such as Distributed Units, cloud-native router technology can be applied to configuring host-based virtual routers for other containerized applications.

[0044] Because CNR is a cloud-native application, it supports installation using either a Kubernetes manifest or a Helm diagram. These include initial router configuration, including routing protocols and a Layer 3 VPN for supporting slices. CNR can be orchestrated and configured in seconds, with all routing protocols adjoining and running with the rest of the network. Configuration changes made during CNR's lifetime, such as adding or removing network slices, can be made via CLI, Kubernetes manifest, NetConf, or Terraform.

[0045] By adopting the Kubernetes CNI framework, cloud-native routers can mitigate the traditional operational overhead incurred when using containerized devices instead of their physical counterparts. By exposing appropriate device interfaces, cloud-native routers can standardize the operating model of virtual devices against physical devices, thereby eliminating barriers to adoption within carrier network operating environments. Cloud-native routers can present a familiar router look and feel to any trained operations team. They possess similar features and capabilities as hardware-based platforms, along with a similar operating model. Similarly, domain controllers can use protocols they use with any other Junos router to communicate with and control cloud-native routers, such as Netconf / OpenConfig, gRPC, the Path Computing Unit Protocol (PCEP), and the Programmable Router Daemon (pRPD) application programming interface (API).

[0046] Figure 1A The block diagram of an example mobile network system is illustrated in accordance with the technology described in this disclosure. The mobile network system 100 may be a 5G network implementing 5G standards published by, for example, the 3rd Generation Partnership Project (3GPP), the Open Radio Access Network (“O-RAN” or “ORAN”) Alliance, the European Telecommunications Standards Institute (ETSI), the Internet Engineering Task Force (IETF), and the International Telecommunication Union (ITU).

[0047] 5G networks, built around cloud-native principles, allow for the decomposition of mobile fronthaul and midhaul networks. As a result, service providers can avoid being locked into specific appliance vendors and can combine effective solutions from different vendors at different tiers and locations to build and deliver mobile network systems. This can improve radio access networks (RANs), particularly by making them more open, resilient, and scalable.

[0048] O-RAN-based networks decompose the baseband unit (BBU) found in traditional telecommunications networks into three functional units: radio unit (RU), distributed unit (DU), and centralized unit (CU). The different functions of the RU, DU, and CU can be implemented by software executed by x86-based or ARM-based host servers. The CU can also be further separated into different control plane (CU-CP) and user plane (CU-UP) functions for further control and user plane decoupling (CUPS). This decoupling helps to provide deployment flexibility—different combinations of RU, DU, and CU can be deployed in the same or different locations. For example, in latency-critical situations, the RU, DU, and CU can be placed together at the edge. O-RAN compliant DUs and CUs are typically referred to as O-DU and O-CU, respectively. Additional data plane elements, known as user plane functions (UPFs), operate in the mobile core network 7 to forward traffic between the CU and data network 15. Additional control plane elements also operate in the mobile core network 7. These control plane components include Network Slice Selection Function (NSSF), Policy Control Function (PCF), Authentication Server Function (ASUF), Access and Mobility Management Function (AMF), Network Exposure Function (NEF), Network Function Library Function (NRF), Application Function (AF), Unified Data Management (UDM), and Session Management Function (SMF).

[0049] Mobile network system 100 includes a radio access network 9 and a mobile core network 7. The radio access network 9 includes RUs 14 located at various cellular network sites (“cell sites”). Each RU 14 consists of a LO PHY and an RF transmitter. The LO PHY component can be implemented using dedicated hardware for high-performance packet processing.

[0050] RU 14 is connected to DU 22A-22X (collectively, "DU 22") via a fronthaul network. The fronthaul network connects the LO PHY and HI PHY and is used by RU 14 and DU 22 to implement the 5G F2 interface. DU 22 manages the radio packet transmissions of RU 14. In some cases, such packet transmissions conform to the Common Packet Radio Interface (CPRI) and / or Enhanced CPRI (ePRI) standards, or IEEE 1914.3. DU 22 can implement Radio Link Control (RLC), Media Access Control (MAC), and the HIPHY layer. DU 22 is controlled at least in part by CU 13A-13B (collectively, "CU 13").

[0051] DU 22 connects to CU 13 via a midrange network, which can be used by DU 22 and CU 13 to implement the 5G F1 interface. CU 13 can implement the Radio Resource Control (RRC) and Data Packet Convergence Protocol (PDCP) layers. CU 13 connects to the mobile core network 7 via a backhaul network. Both the midrange network and the backhaul network can be wide area networks (WANs).

[0052] In the radio access network 9 of the mobile network system 100, a gNodeB includes one of CU 13 and one of DU 22. A CU can support multiple DUs to implement multiple gNodeBs. And one or more RUs can be supported by a single DU. Therefore, for example, regarding... Figure 1A One of CU 13A, DU 22A, and RU 14 can form an eNodeB, while the other of CU 13A, DU 22B (of server 12B), and RU 14 can form another eNodeB.

[0053] like Figure 1A As shown, any DU 22 may or may not be located at a cell site including an RU 14 supported by the DU. DU 22X is located at a cell site, while DU 22A-22N are located at a local data center, and together they support multiple RU 14s. The mobile network system 100 may have a radio access network 9 comprising thousands of cell sites, each cell site having one or more RU 14s and optionally one or more DU 22s. Whether located at a cell site or off-site, the DU is typically within 20 km of the supported RUs. CU 13 is in Figure 1A The data center is shown as being located in a regional data center, typically within 40km of the supported DU 22.

[0054] Radio access network 9 connects to mobile core network 7 to exchange data packets with data network 15. Mobile core network 7 may be a 5G core network, and data network (DN) 15 may represent, for example, one or more service provider networks and services, the Internet, third-party services, IP multimedia subsystem, or other networks.

[0055] Mobile network system 100 includes multiple servers 12A-12X to execute DU 22. Each server 12 can be a real or virtual server that hosts / executes software implementing DU 22. Such software may include one or more applications that are deployed, for example, as virtual machines or containers, to server 12. Although in Figure 1A It is not shown in the diagram, but CU 13 can also be executed by the server.

[0056] The combination of DU 22, mid-range network, CU 13 and backhaul network effectively realizes the IP-based transmission network between radio unit 14 and mobile core network 7.

[0057] Cloud-local routers 20A-20X (“CNR 20A-20X” and collectively “CNR 20”) provide Layer 3 routing functionality between DU 22 and CU 13. These CNR 20s can be executed on the same server 12 as one or more DU 22s to provide edge router functionality to such DU 22s. In some examples, any CNR in the CNR 20 can be deployed to a local data center along with one or more DU 22s for which the CNR provides IP services, as shown with respect to CNR 20A-20N, i.e., where the local data center includes a server 12 executing DU 22s for one or more cell sites. In some examples, the CNR can be deployed on, for example, Figure 1A The CNR deployed at the cell site shown is referred to as a "virtualized cell site router".

[0058] Each CNR 20 is implemented using one of the containerized routing protocol daemons 24A-24X (“cRPD 24A-24X” and collectively “cRPD 24”). More specifically, each CNR 20 uses the corresponding cRPD in cRPD 24 as the control plane for implementing Layer 3 routers. cRPD provides control plane routing functionality. For example, cRPD can execute lower-layer IP (IPv4 / IPv6) routing protocols such as Intermediate System-Intermediate System (IS-IS) and Border Gateway Protocol (BGP); advertise the reachability of DU 22 within and outside the cluster to CU 13; implement network namespaces (supported using L3VPN and EVPN Type-5 advertisements); implement access control lists (ACLs) and network policies for security, network isolation, and quality of service (QoS); support tunneling and traversal protocols (e.g., MPLS, SR-MPLS, SRv6, SR-MPLSoIPv6, SR-MPLSoIPv4, VxLAN, IP-in-IP, GRE); support dynamic tunneling techniques using BGP signaling; support encryption of IPSec tunnels; and program the forwarding plane of the server's CNR using learned and / or configured routing information to provide Layer 3 packet forwarding, encapsulation, packet filtering, and / or QoS between one or more of DU 22 and one of CU 13.

[0059] For example, CNR 20A performed by server 12A includes cRPD 24A and the forwarding plane of server 12A (e.g., a SmartNIC, a kernel-based forwarding plane, or a data plane development kit (DPDK)-based forwarding plane). cRPD 24A provides one or more of the aforementioned routing functions to program the forwarding plane of CNR 20A to advertise Layer 3 routes for DU 22A outside the cluster—including traversing the mid-range network to CU 13A—and forward Layer 3 packets between DU 22A and CU 13A in other tasks. In this way, the technology implements a cloud-native, containerized cell site router 20, which runs on the same server 12 as the containerized DU 22, thus significantly reducing mid-range latency between DU 22 and CU 13.

[0060] As a containerized router, the CNR 20 allows x86-based or ARM-based hosts to become first-class members of the network routing system, participating in protocols such as IS-IS and BGP, and providing MPLS / SR-based transport and multi-lease. Therefore, the CNR20 can operate as a provider edge (PE) router for transmitting Layer 3 packets between DU 22, CU 13, and the mobile core network 7, rather than as an auxiliary device to the network (similar to a customer edge (CE) router).

[0061] Furthermore, in some examples, the integration of cRPD 24 and the host-based forwarding plane can also deliver Kubernetes CNI-compliant packages that can be deployed in Kubernetes environments. Execution by a single server consisting of DU 22 and CNR 20 avoids the need for dual-box solutions with separate DUs and routers, potentially reducing cost, power, and space requirements, which is particularly attractive for cell sites. Application workloads can be containerized network functions (CNFs), such as DUs.

[0062] Orchestrator 50 represents a container orchestration platform. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services running on such virtual execution elements by host servers available to the orchestration platform. Container orchestration specifically allows for container orchestration and involves, for example, the deployment, management, scaling, and configuration of containers to host servers by a container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes, Docker Swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS. Orchestrator 50 orchestrates at least containerized RPD 24 of DU 22 and CNR 20. In some examples, the data plane of CNR20 is also containerized and orchestrated by orchestrator 50. The data plane can be a DPDK-based data plane, a kernel-based data plane, or a hybrid data plane, as further described below.

[0063] Containers, including those implementing the containerized routing protocol daemon 24, can be deployed to a virtualization environment using a cluster-based framework, in which the cluster master node manages the deployment and operation of containers to one or more cluster worker nodes. The terms "master node" and "worker node" used herein encompass different orchestration platform terms used for similar devices, distinguishing between the primary management element of a cluster and the primary virtual execution element of the device hosting the cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "worker node," while the Docker Swarm platform refers to both the cluster manager and cluster nodes. A cluster node can be represented by a server 12 or a virtual machine on it.

[0064] The orchestrator 50 and the software-defined networking (SDN) controller 70 can execute on separate computing devices or on the same computing device. Each of the orchestrator 50 and the SDN controller 70 can be a distributed application executing on one or more computing devices. The orchestrator 50 and the SDN controller 70 can implement corresponding master nodes for one or more clusters, each cluster having one or more worker nodes implemented by a corresponding server 12. Typically, the SDN controller 70 controls the network configuration of the radio access network 9 to facilitate packetized communication between DU 22, CU 13, and the mobile core network 7. The SDN controller 70 can distribute routing and configuration information to the control plane elements of the radio access network 9, specifically to cRPD 24. The SDN controller 70 can, for example, program segmented routing headers, configure L3VPN, and configure VRFs in the routers of the radio access network 9 (including cloud-local router 20). The SDN controller 70 can implement one or more southbound protocols for configuring routers, switches, and other network devices in the mid-range and backhaul networks, and for configuring CNR 20. Exemplary southbound protocols may include the Path Computing Element Protocol (PCEP), BGP, Netconf, OpenConfig, and another protocol for configuring cRPD 24, etc. Additional information about L3VPNs can be found in "BGP / MPLS IP Virtual Private Networks (VPNs)," Request for Comments 4364, Network Working Group of Internet Engineering Task Force, February 2006, which is incorporated herein by reference in its entirety.

[0065] SDN controller 70 can provide a logically centralized, and in some cases physically centralized, controller. In some examples, SDN controller 70 can operate in response to configuration input received from orchestrator 50 and / or administrator / operator. SDN controller 70 can be programmed with NFV infrastructure (NFVI) such as server 12, network switches / routers, and / or other network infrastructure. In the case of NFVI programming, SDN controller 70 can configure aspects of the operating system kernel to configure L3 IP routing, Linux bridging, iptables, network namespaces, and / or virtual switches.

[0066] Additional information regarding the example SDN controller 70, virtual router, and virtual router agent can be found in the following: International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORKPACKET FLOWS”; U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled “TUNNELEDPACKET AGGREGATION FOR VIRTUAL NETWORKS”; and NETWORK CONTROLLER HORIZONTAL SCALING FOR NETWORKDEVICE CONFIGURATIONS SESSION MANAGEMENT, filed June 30, 2021; each of which is incorporated herein by reference as if fully set forth herein.

[0067] Typically, orchestrator 50 controls the deployment, scaling, and operation of containers across a cluster of servers 12, as well as the provision of compute infrastructure, which may include container-centric compute infrastructure. Orchestrator 50, and in some cases, network controller 70, may implement corresponding cluster hosts for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which can provide virtualization infrastructure to the container management platform.

[0068] Figure 1B This is a block diagram illustrating an example computing infrastructure 108 that can implement the techniques described herein. Typically, data center 110 provides an operating environment for applications and services at customer site 111 (illustrated as "Customer 111"), which has one or more customer networks coupled to the data center via service provider network 107. Data center 110 may be... Figure 1AThe implementation of a local data center and / or regional data center is shown. Data center 110 may, for example, host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. Service provider network 107 is coupled to public network 115, which may represent one or more networks managed by other providers and thus may form part of a large-scale public network infrastructure, such as the Internet. Public network 115 may represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by the service provider operating service provider network 107, an enterprise IP network, or some combination thereof. Public network 115 may be Figure 1A The data network 15 or the network accessible via the data network 15.

[0069] Although customer site 111 and public network 115 are illustrated and primarily described as the edge network of service provider network 107, in some examples, one or more of customer site 111 and public network 115 may be a tenant network within data center 110 or another data center. For example, data center 110 may host multiple tenants (customers), each associated with one or more Virtual Private Networks (VPNs), each VPN potentially implementing one of customer sites 111.

[0070] Service provider network 107 provides packet-based connectivity to attached customer sites 111, data centers 110, and public networks 115. Service provider network 107 may represent a network owned and operated by a service provider to interconnect multiple networks. Service provider network 107 may implement Multiprotocol Label Switching (MPLS) forwarding and in such instances may be referred to as an MPLS network or MPLS backbone. In some instances, service provider network 107 represents multiple interconnected autonomous systems, such as the Internet, providing services from one or more service providers. In some aspects, service provider network 107 may be... Figure 1A The implementation of the mobile core network 7.

[0071] In some examples, data center 110 can represent one of many geographically distributed network data centers. For example... Figure 1BAs illustrated in the example, data center 110 may be a facility that provides network services to customers. The service provider's customers may be collective entities, such as businesses and governments or individuals. For example, a network data center may host web services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although illustrated as a separate edge network of service provider network 107, elements of data center 110, such as one or more physical network functions (PNFs) or virtualized network functions (VNFs), may be included within the core of service provider network 107.

[0072] In this example, data center 110 includes storage and / or compute servers (or “nodes”) interconnected via a switch fabric 114 provided by one or more layers of physical network switches and routers, wherein server 12 is depicted as coupled to top-of-rack switches 116A-116N. Although in Figure 1B Only server 12A coupled to TOR switch 116A is shown in detail, but other servers 12 may be coupled to other TOR switches 116 in data center 110.

[0073] The switching fabric 114 in the example shown includes interconnected top-of-rack (TOR) (or other "leaf") switches 116A-116N (collectively referred to as "TOR switches 116"), which are coupled to a distribution layer of chassis (or "backbone" or "core") switches 118A-118M (collectively referred to as "chassis switches 118"). Although not shown, data center 110 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridging devices, cable modems, application accelerators, or other network devices. Data center 110 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0074] In this example, TOR switch 116 and chassis switch 118 provide redundant (multihomed) connectivity to server 12 to IP fabric 120 and service provider network 107. Chassis switch 118 aggregates traffic and provides connectivity between TOR switches 116. TOR switch 116 may be a network device providing Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching capabilities. TOR switch 116 and chassis switch 118 may each include one or more processors and memory, and may execute one or more software processes. Chassis switch 118 is coupled to IP fabric 120, which may perform Layer 3 routing to route network traffic between data center 110 and customer site 111 via service provider network 107. The switching architecture of data center 110 is merely an example. For example, other switching architectures may have more or fewer switching layers. IP fabric 120 may include one or more gateway routers.

[0075] The terms "data packet stream," "transaction stream," or simply "stream" refer to a group of data packets originating from a specific source device or endpoint and being sent to a specific destination device or endpoint. A single data packet stream can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. This 5-tuple typically identifies the data packet stream corresponding to a received data packet. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a data packet can refer to a combination of <source network address, destination network address> or <source network address, source port>.

[0076] Server 12 may each represent a compute server or a storage server. For example, each server 12 may represent a computing device configured to operate according to the techniques described herein, such as an x86 processor-based server. Server 12 may provide an NFVI architecture for Network Functions Virtualization Infrastructure (NFVI).

[0077] Server 12 may be configured with kernel 380. Kernel 380 can manage the operations of server 12, including scheduling processes, threads or other executable units, as well as the management devices, file systems and memory of server 12. Kernel 380 may represent a Linux kernel, other Unix variant kernels, or other operating system kernels that include a network stack and are capable of packet forwarding.

[0078] Any server in Server 12 can be configured with virtual execution elements using the resources of the virtualization server to provide isolation between one or more processes (applications) running on the server. "Hypervisor-based," "hardware-level," or "platform" virtualization refers to the creation of virtual machines, each containing a guest operating system for executing one or more processes. Typically, virtual machines provide a virtualization / guest operating system for executing applications in an isolated virtual environment. Because virtual machines are virtualized from the physical hardware of the host server, the executing applications are hardware isolated from the host and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communication on the corresponding virtual network.

[0079] A virtual network is a logical construct implemented on top of a physical network. Virtual networks can be used as an alternative to VLAN-based isolation and to provide multi-tenancy in virtualized data centers, such as Data Center 110. Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly permitted by security policies.

[0080] Virtual networks can use data center 110 gateway routers ( Figure 1B (Not shown) Connects to and extends on top of Physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Networks (L3 VPNs) and Ethernet Virtual Private Networks (EVPNs). Virtual networks can also be used to implement Network Functions Virtualization (NFV) and service links.

[0081] Virtual networks can be implemented using various mechanisms. For example, each virtual network can be implemented as a Virtual Local Area Network (VLAN), a Virtual Private Network (VPN), etc. Virtual networks can also be implemented using two networks: a physical lower-layer network consisting of an IP infrastructure 120 and a switching infrastructure 114, and a virtual upper-layer network. The role of the physical lower-layer network is to provide the "IP infrastructure," which provides unicast IP connections from any physical device (server, storage device, router, or switch) to any other physical device. The lower-layer network can provide unified low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

[0082] As further described below regarding virtual router 121A, the virtual routers running in server 12 create a virtual upper-layer network over the physical lower-layer network using a mesh of dynamic "tunnels" between them. These upper-layer tunnels can be, for example, MPLS via GRE / UDP tunnels, or VXLAN tunnels, or NVGRE tunnels. The lower-layer physical routers and switches may not store any per-tenant state of the virtual machines or other virtual execution elements, such as any media access control (MAC) addresses, IP addresses, or policies. The forwarding tables of the lower-layer physical routers and switches may, for example, contain only the IP prefix or MAC address of physical server 12. (The gateway router or switch connecting the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0083] The virtual routers 121 of server 12 typically contain the state of each tenant. For example, they may contain a separate forwarding table (routing instance) for each virtual network. This forwarding table contains the IP prefixes (in the case of Layer 3 or higher) or MAC addresses (in the case of Layer 2 or higher) of virtual machines or other virtual execution elements (e.g., container pods). No single virtual router 121 needs to contain all the IP prefixes or all the MAC addresses of all virtual machines throughout the entire data center. A given virtual router 121 only needs to contain those routing instances that are locally present on server 12 (i.e., those with at least one virtual execution element present on server 12).

[0084] The control plane protocol between the control plane nodes of the SDN controller 70 or the physical gateway router (or switch) can be BGP (and can also be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3VPN and MPLS EVPN. The protocol between the SDN controller 70 and the virtual router 121 can be based on, for example, XMPP.

[0085] Server 12 is a virtual network endpoint of one or more virtual networks operating on the physical network represented herein by IP structure 120 and switching structure 114. Although the description is primarily directed at data center-based switching networks, other physical networks such as service provider network 107 may host one or more virtual networks.

[0086] Each server 12 may host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint for configuring one or more virtual networks in a physical network. A virtual network endpoint of a virtual network may represent one or more virtual execution elements sharing a virtual network interface of that virtual network. For example, a virtual network endpoint may be a virtual machine, a collection of one or more containers (e.g., pods), or another virtual execution element, such as a layer 3 endpoint of a virtual network. The term "virtual execution element" encompasses virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" may also include one or more container pods. Figure 1B As shown, server 12A hosts virtual network endpoints in the form of pods 122A and 122B, each virtual network endpoint having one or more containers. However, given the hardware resource limitations of server 12, server 12 can execute as many virtual execution elements as is actually required. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 113A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 116A. Other examples of virtual network interfaces are described below.

[0087] Each of the servers 12 includes at least one network interface card (NIC) 113, each including at least one interface for exchanging data packets with the TOR switch 116 via a communication link. For example, server 12A includes NIC 113A. Any of the NICs 113 can provide one or more virtual hardware components 121 for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of the physical NIC (“physical function”). For example, in Single Root I / O Virtualization (SR-IOV), as described in the Peripheral Device Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or “network adapter”) is virtualized to present one or more virtual network interfaces as “virtual functions” for use by corresponding endpoints running on server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are an example of virtual hardware components 121. As another example, one or more servers 12 may implement virtualization, i.e., a hypervirtualization framework available for the Linux operating system, which provides emulated NIC functionality as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement an open vSwitch to perform distributed virtual multi-tier switching between one or more virtual NICs (vNICs) of the hosted virtual machines, where such vNICs may also represent a type of virtual hardware component that provides virtual network interfaces to virtual network endpoints. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function. In some examples, any server in server 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interface. For a Docker implementation of containers hosted by a server, a Linux bridge, or another operating system bridge running on the server, exchanging packets between containers may be referred to as a "Docker bridge." The term "virtual router" as used herein may include a control or tungsten-structured virtual router, an open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software located on a host device that perform the exchange, bridging, or routing of packets between virtual network endpoints in one or more virtual networks, wherein the virtual network endpoints are hosted by one or more servers.

[0088] Any of the NICs in the 113 series may include an internal device switch to exchange data between virtual hardware components associated with the NIC. For example, for an SR-IOV-enabled NIC, the internal device switch may be a Virtual Ethernet Bridge (VEB) to exchange data between SR-IOV virtual functions, and correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. The internal device switch may alternatively be referred to as a NIC switch, or, for an SR-IOV implementation, an SR-IOV NIC switch. The virtual hardware components associated with the NIC 113A may be associated with a Layer 2 destination address, which may be assigned by the NIC 113A or the software process responsible for configuring the NIC 113A. Physical hardware components (or “physical functions” of the SR-IOV implementation) are also associated with Layer 2 destination addresses.

[0089] Each of one or more servers 12 may include a virtual router 121 that performs one or more routing instances for a corresponding virtual network within data center 110 to provide a virtual network interface and route packets between virtual network endpoints. In some aspects, virtual router 121 may be incorporated as part of CNR 20. In some aspects, the virtual router may be a virtual router implemented in the kernel space memory of server 12 and may be referred to as a "kernel-based" virtual router. In some aspects, virtual router 121 may be implemented in the user storage space of server 12 and support a DPDK interface. Such a virtual router may be referred to as a "DPDK" virtual router.

[0090] Each routing instance of virtual router 121 can be associated with a network forwarding table. Each routing instance can represent a Virtual Router and Forwarder Instance (VRF) for Internet Protocol-Virtual Private Network (IP-VPN). Packets received by virtual router 121A of server 12A, for example from the underlying physical network structure of data center 110 (i.e., IP structure 120 and switching structure 114), may include an external header to allow the physical network structure to tunnel the payload or “internal packet” to the physical network address of network interface card 113A of server 12A performing the virtual router function. The external header may include not only the physical network address of server network interface card 113A, but also a virtual network identifier, such as a VxLAN label or a Multiprotocol Label Switching (MPLS) label, which identifies one of the virtual networks and the corresponding routing instance performed by virtual router 121A. Internal packets include an internal header with a destination network address that conforms to the virtual network addressing space identified by the virtual network identifier.

[0091] Virtual router 121 terminates the upper-layer tunnel of the virtual network and determines the virtual network of the received packet based on the tunnel encapsulation header of the packet, and forwards the packet to the appropriate destination virtual network endpoint. For server 12A, for example, for each packet outbound from a virtual network endpoint (e.g., pod 122A) hosted by server 12A and coupled to virtual router 121A, virtual router 121A appends a tunnel encapsulation header indicating the virtual network of the packet to generate an encapsulated or “tunneled” packet, and virtual router 121A outputs the encapsulated packet via the upper-layer tunnel of the virtual network to a physical destination computing device, such as another in server 12. As used herein, virtual router 121 can perform the operations of a tunnel endpoint to encapsulate internal packets initiated by a virtual network endpoint to generate a tunnel packet, and decapsulate the tunnel packet to obtain internal packets for routing to other virtual network endpoints.

[0092] exist Figure 1B In the example, Virtual Router 121A is a virtual router with the Data Plane Development Kit (DPDK) enabled. That is, Virtual Router 121A uses DPDK as its data plane. In this mode, Virtual Router 121A runs as a user-space application linked to a DPDK library (not shown). This is a performance version of the virtual router and is typically used by telecommunications companies where VNFs are often DPDK-based applications. Virtual Router 121A, as a DPDK virtual router, can achieve significantly higher throughput than a virtual router operating as a kernel-based virtual router. The physical interface is used by the DPDK's polling-mode driver (PMD) instead of the Linux kernel's interrupt-based driver, thus eliminating much of the context-switching overhead associated with interrupt-based drivers.

[0093] User I / O (UIO) kernel modules such as vfio or uio_pci_generic can be used to expose the registers of the physical network interface in user space, making them accessible by the DPDK PMD. When NIC 113A is bound to the UIO driver, it moves from the Linux kernel space to user space and is therefore no longer managed or visible by the Linux OS. Thus, it is the DPDK application (i.e., virtual router 121A in this example) that fully manages NIC 113A. This includes packet polling, packet processing, and packet forwarding. User packet processing steps can be performed by the virtual router 121A DPDK data plane, where the kernel (… Figure 1B(The kernel is not shown in the diagram) is limited or the kernel does not participate. The nature of this "polling mode" makes the data plane packet processing / forwarding of the Virtual Router 121A DPDK much more efficient than interrupt mode, especially at high packet rates. Limited interrupts and context switching are present or absent during packet I / O.

[0094] Additional details of the DPDK virtual router example can be found in “DAY ONE: CONTRAIL DPDK VROUTER,” 2021, Kiran KN et al., Juniper Networks, Inc., which is incorporated herein by reference in its entirety.

[0095] The computing infrastructure 108 implements an automated platform for deploying, scaling, and operating virtual execution elements across servers 12 to provide virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration platform that provides container-centric infrastructure for automating the deployment, scaling, and operation of containers to provide container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements by host servers available to the orchestration platform. Container orchestration specifically allows container orchestration and involves, for example, the deployment, management, scaling, and configuration of containers to host servers by a container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes, Docker Swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0096] The components of the automated platform for computing infrastructure 108 include at least server 12, orchestrator 50, and SDN controller 70. Containers can be deployed to a virtualized environment using a cluster-based framework, where the cluster master node manages the deployment of containers to one or more cluster worker nodes and operations on those worker nodes. The terms "master node" and "worker node" used herein encompass different orchestration platform terms used for similar devices, distinguishing between the primary management element of a cluster and the primary container hosting device of a cluster. For example, the Kubernetes platform uses the terms "cluster master" and "worker node," while the Docker cluster platform refers to both the cluster manager and cluster nodes.

[0097] The orchestrator 50 and the SDN controller 70 can execute on separate computing devices or on the same computing device. Each of the orchestrator 50 and the SDN controller 70 can be a distributed application executing on one or more computing devices. The orchestrator 50 and the SDN controller 70 can implement corresponding master nodes for one or more clusters, each cluster having one or more worker nodes (also referred to as "compute nodes") implemented by a corresponding server 12.

[0098] Typically, the SDN controller 70 controls the network configuration of the data center 110 architecture to, for example, establish one or more virtual networks for packetized communication between virtual network endpoints. The SDN controller 70 provides a logically and, in some cases, physically centralized controller to facilitate the operation of one or more virtual networks within the data center 110. In some examples, the SDN controller 70 may operate in response to configuration input received from the orchestrator 50 and / or administrators / operators. Additional information regarding an SDN controller 70 operating in conjunction with other devices or other software-defined networking systems in data center 110 can be found in International Application No. PCT / US2013 / 044378, filed June 5, 2013, entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORKPACKET FLOWS”, and U.S. Patent Application No. 14 / 226,509, filed March 26, 2014, entitled “TUNNELED PACKET AGGREGATION FOR VIRTUAL NETWORKS”, each of which is incorporated herein by reference as if fully set forth herein.

[0099] Typically, orchestrator 50 controls the deployment, scaling, and operation of containers across a cluster of servers 12, as well as the provision of compute infrastructure, which may include container-centric compute infrastructure. Orchestrator 50, and in some cases, SDN controller 70, can implement corresponding cluster hosts for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which can provide virtualization infrastructure to the container management platform. Example components of the Kubernetes orchestration system are referenced below. Figure 6B describe.

[0100] In one example, pods 122A and 122B are Kubernetes pods and examples of virtual network endpoints. A pod is one or more logically related containers. Figure 1B(Not shown in the diagram) A pod is a group of containers that share storage and options for how to run them. When instantiated for execution, a pod can alternatively be called a "pod replica". Each container in Pods 122A and 122B is an example of a virtual execution element. Containers within a pod always reside together on a single server, are jointly scheduled, and run in a shared context. A pod's shared context can be a set of Linux namespaces, cgroups, and other aspects of isolation. Within a pod's context, individual applications can have further sub-isolation. Typically, containers within a pod have a common IP address and port space and can detect each other via the localhost. Because they have a shared context, containers within a pod also communicate with each other using inter-process communication (IPC). Examples of IPC include System V semaphores or POSIX shared memory. Typically, containers that are members of different pods have different IP addresses and cannot communicate via IPC without the configuration required to implement this feature. Containers that are members of different pods typically communicate with each other via the pod's IP address instead.

[0101] Server 12A includes container platform 119A for running containerized applications, such as Pods 122A and 122B. Container platform 119A receives requests from orchestrator 50 to acquire and host containers on server 12A. Container platform 119A acquires and executes the containers.

[0102] Container platform 119A includes a container network interface (CNI) 117A, which is configured as a virtual network interface for virtual network endpoints. Orchestrator 50 and container platform 119A use CNI 117A to manage networking of pods including pods 122A and 122B. For example, CNI 117A creates virtual network interfaces to connect pod 122A to virtual router 121A and creates virtual network interfaces to connect pod 122B to kernel 380. CNI 117A thus enables containers of such pods to communicate with other virtual network endpoints via their respective virtual network interfaces through virtual networks. The CNI 117A can, for example, insert a virtual network interface for a virtual network into the network namespace of a container in Pod 122A, and configure (or request configuration) the virtual network interface for the virtual network in virtual router 121A, such that virtual router 121A is configured to send packets received from the virtual network via the virtual network interface to the container in Pod 122A, and to send packets received from the container in Pod 122A via the virtual network interface on the virtual network. The CNI 117A can assign network addresses (e.g., virtual IP addresses for the virtual network) and can establish routes for the virtual network interface. In Kubernetes, by default, all pods can communicate with all other pods without using Network Address Translation (NAT). In some cases, orchestrator 50 and SDN controller 70 create service virtual networks and pod virtual networks shared by all namespaces, from which service and pod network addresses are assigned respectively. In some cases, all pods in all namespaces generated in a Kubernetes cluster may be able to communicate with each other, and network addresses for all pods may be assigned from the pod subnets specified by orchestrator 50. When a user creates an isolated namespace for a pod, the orchestrator 50 and the SDN controller 70 can create new pod virtual networks and new shared service virtual networks for the new isolated namespace. Pods in the isolated namespace created in the Kubernetes cluster extract network addresses from the new pod virtual network, and the corresponding services for such pods extract network addresses from the new service virtual network.

[0103] CNI 117A can represent a library, plugin, module, runtime, or other executable code for Server 12A. CNI 117A may at least partially conform to the Container Network Interface (CNI) specification or the rkt networking proposal. CNI 117A may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNIs. CNI 117A may alternatively be referred to as a network plugin, CNI plugin, or CNI instance. For example, a separate CNI can be called by a Multus CNI to establish different virtual network interfaces for pod 122A.

[0104] CNI 117A is invoked by orchestrator 50. For the purposes of the CNI specification, a container can be considered synonymous with a Linux network namespace. What unit this corresponds to depends on the specific container runtime implementation: for example, in implementations of application container specifications such as rkt, each pod runs in a unique network namespace. However, in Docker, a network namespace typically exists for each individual Docker container. For the purposes of the CNI specification, a network refers to a uniquely addressable set of entities that can communicate with each other. This can be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can conceptually be added to or removed from one or more networks. The CNI specification specifies several considerations for conforming plugins (“CNI plugins”).

[0105] exist Figure 1B In the example, Pod 122A includes a containerized DPDK workload designed to use DPDK to accelerate packet processing, such as exchanging data with other components using the DPDK library. Pod 122B is configured to utilize the network services and stack provided by kernel 380.

[0106] CNI 117A can be combined Figure 1B One or more other components shown configure data interfaces 126 and 127 for pods 122A and 122B. These interfaces can be of different types. Data interface 126 and data interface 127 may be referred to herein as virtual network interfaces. Other example types of virtual network interfaces are described below. Any containers of pods 122A and 122B can utilize, i.e., share any virtual network interface of their respective pods.

[0107] Pod 122A is configured with a data interface 127, which is used for high-throughput packet processing, more specifically, for sending and receiving packets using a virtual router 121A for high-throughput applications. Pod 122A and virtual router 121A exchange packets using data interface 127. Data interface 127 can be a DPDK interface. Pod 122A and virtual router 121A can use a vhost to establish a data interface 127. Pod 122A can operate according to an aggregation model. Pod 122A can use virtual devices, such as Virtio devices with vhost-user adapters, for user-space container inter-process communication using data interface 127. In this way, pod 122A's workload is built on DPDK and uses DPDK to operate data interface 127. Pod 122A and virtual router 121A can use a vhost to establish a DPDK interface. In some examples, pod 122A can operate as a vhost server, with virtual router 121A acting as a vhost client to establish the DPDK interface.

[0108] Pod 122B is configured to have a data interface 126 for sending and receiving packets with kernel 380. The workload of pod 122B may not be configured for high-throughput data communication and may require encapsulation or other network services that vrouter 121A cannot provide.

[0109] Network packets sent and received by pods 122A and 122B can take different paths to and from NIC 113A. For example, pod 122A exchanges network packets with virtual router 121A via data interface 127. Virtual router 121A sends or receives network packets from NIC 113A. Virtual router 121A and NIC 113A form the DPDK data plane for pod 122A.

[0110] Pod 122B exchanges network packets with kernel 380. Kernel 380 processes network packets sent and received by pod 122B, including encapsulating or decapsulating packets as necessary. In some respects, kernel 380 uses data interface 125 to send and receive packets via virtual router 121A. In this case, virtual router 121A acts as a "passthrough" relative to kernel 380 and does not perform any network processing on packets received from kernel 380 for transmission via NIC 113 or packets received from NIC 113A destined for pod 122B. Kernel 380, virtual router 121A, and NIC 113A form the kernel data plane.

[0111] Virtual router 121A and kernel 380 are configured to interoperate as further described below to provide hybrid data plane 150. Hybrid data plane 150 allows pods such as pods 122A and 122B to be configured to use either the DPDK data plane or the kernel data plane of hybrid data plane 150.

[0112] Regarding pod 122B, the virtual network interface can represent a virtual Ethernet (“veth”) pair, where each end of the pair is a separate device (e.g., a Linux / Unix device), with one end assigned to pod 122B and the other end assigned to kernel 380. The veth pair, or the ends of the veth pair, are sometimes referred to as “ports.” The virtual network interface can also represent a macvlan network with Media Access Control (MAC) addresses assigned to pod 122B and kernel 380, used for communication between the container of pod 122B and the network stack of kernel 380. The virtual network interface can alternatively be referred to as, for example, a Virtual Machine Interface (VMI), pod interface, container network interface, tap interface, veth interface, or simply a network interface (in a specific context).

[0113] exist Figure 1B In example server 12A, pods 122A and 122B are virtual network endpoints in one or more virtual networks. Orchestrator 50 may store or otherwise manage configuration data for application deployment that specifies virtual networks and designates pods (or one or more containers within them) as virtual network endpoints of those virtual networks. Orchestrator 50 may receive configuration data from, for example, users, operators / administrators, or other machine systems.

[0114] As part of the process of creating pod 122A or 122B, orchestrator 50 requests SDN controller 70 to create a corresponding virtual network interface (indicated in the configuration data) for the virtual network. A pod can have different virtual network interfaces for each virtual network it belongs to. For example, each of data interface 126 and data interface 127 can be a virtual network interface for a specific virtual network. Additional data interfaces can be configured for other virtual networks. SDN controller 70 processes this request to generate interface configuration data for the virtual network interfaces of pods 122A and 122B. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include the network name, assigned virtual network address, MAC address, and / or domain name server value. Below is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0115] SDN controller 70 sends interface configuration data to server 12A, and more specifically, to virtual router 121A in some cases. To configure virtual network interfaces for pods (e.g., pod 122A or pod 122B), orchestrator 50 can invoke CNI 117A. CNI 117A obtains and processes the interface configuration data from virtual router 121A. CNI 117A creates each virtual network interface specified in the interface configuration data.

[0116] Regular CNI plugins are invoked by the container platform / runtime to receive add commands from the container platform to add containers to a single virtual network. Such plugins can then be invoked to receive Del (ete) commands from the container / runtime and remove containers from the virtual network. The term "invocation" can refer to the instantiation of a software component or module in memory into executable code for processing circuitry to execute.

[0117] Figure 2 A more detailed illustration of the technology according to this disclosure. Figure 1A and Figure 1B A block diagram of an example implementation of a network system. System 200 includes CUs 213A-213K, each of which can represent Figure 1A Any of the CU 13. In this example, L3VPN and tunnels 231A-231K are used to implement multiple network slices (e.g., 5G network slices) to connect the DU22A to different CU 213A-213Ks for the respective network slices.

[0118] Network slicing provides a way to segment a mobile network to support specific types of services or businesses, or even host service providers that don't own the physical network (multi-lease). Furthermore, each slice can be optimized based on capacity, coverage, connectivity, security, and performance characteristics. Because slices can be isolated from each other, as if they were physically separate in the control and user planes, the user experience across network slices will be identical, just as if they were separate networks. Network slicing can span all domains of a network, including software applications running on network nodes (both memory and processing), the kernel transport network's specific configuration, access network configuration, and terminal devices. Network slicing enables multiple operators to securely share a mobile network, but by separating their own users from other users, and allowing different applications for different users to utilize different network slices offering widely different performance characteristics.

[0119] The cloud-local router 20A includes a virtual router forwarding plane (virtual router) 121A, which is configured with VRF 212A-212K (collectively referred to as "VRF 212") for corresponding network slices implemented in a corresponding L3VPN. CNR 20A and routers 204A-204B use tunnels 231A-231K that connect VRF 212 to VRF 210A-210K on routers 204A-204B to implement this virtual router forwarding plane. Each tunnel 231A-231K may represent SR-MPLSoIPv6 or other tunnel types mentioned above. Each of routers 204A-204K can be used in a data center with one or more servers to implement any one or more of CU 213A-213K (e.g., Figure 1A The gateway router of the data center (100). The data center may include a data center architecture to exchange mobile data traffic between the router and the CU. In some cases, one or more servers in the data center may also perform a UPF for mobile networks, in which case the data center architecture may also exchange mobile data traffic between the CU and the UPF.

[0120] Each of the VRF 212A-212K has a corresponding virtual network interface to the DU 22A. Each of the virtual network interfaces in the DU 22A can therefore be mapped to a different L3VPN in the CNR 20A to, for example, support one of multiple network slices. As described further below, when triggered by a pod event from the orchestrator 50, the CNI117A of server 12A ( Figure 1BThe virtual network interface between pods (DU 22A deployed in this paper) and virtual router 121A can be dynamically added or removed; in some examples, it can also be deployed as a container. CNI 117A also dynamically updates the cRPD 24A (control plane of CNR 20A) using host routes and corresponding Layer 3 VPN mappings for each DU 22A / pod virtual network interface, in the form of route delimiters and route targets. Correspondingly, cRPD 24A can optionally be programmed to the virtual router 121A (data plane of CNR 20A) using the gRPC interface. In this way, CNR 20A is introduced into the data path as a cloud-native router to support, for example, the F1 interface of a CU 213A-213K that can be implemented in edge or regional data center sites. In various examples, virtual router 121A can represent a SmartNIC-based virtual router, a kernel-based virtual router, or a DPDK-based virtual router.

[0121] Figure 3 This is a block diagram illustrating an example server according to the technology of this disclosure. Server 300 can represent... Figure 1A and Figure 1B Any one of the servers 12. In some cases, server 300 is configured to implement both a cloud-native router and a distributed unit for use. Figure 1A DU22A and Figure 1A The cloud-local router 20A forwards mobile data traffic between data planes within the same box. Server 300 can be a bare metal server or a virtual machine. Figure 6A and Figure 6B The example hardware architecture of server 300 is described in the document.

[0122] Kernel 380 manages the operations of server 300, including scheduling processes, threads, or other executable units, and managing server 300's devices, file systems, and memory. Kernel 380 can represent the Linux kernel, other Unix variant kernels, or other operating system kernels that include a network stack and are capable of packet forwarding.

[0123] Server 300 includes one or more network interface cards (NICs) 321A-321B (collectively referred to as "NIC 321"), each having one or more hardware interfaces 320 and 322. In a 5G radio access network deployment, interface 320 of NIC 321A can be connected to the RU via a physical cable. Interface 320 implements an F2 interface. Interface 322 of NIC 321B can be coupled to a mid-range network via a physical cable for sending / receiving mobile data traffic to / from the CU. Interface 322 implements an F1 interface. In some examples, server 300 may have a single NIC with one or more interfaces 322.

[0124] Server 300 can host pods 328A-328L (collectively referred to as "pod 328"). In some cases, pod 328 can be a DU. pod 328 can be... Figure 1B The implementations of pods 122A and 122B. pods 328A-328L are endpoints from the perspective of virtual router 206A, and in particular can represent upper-layer endpoints that have been programmed into one or more virtual networks in virtual router 206A.

[0125] At higher levels, the DPDK-based virtual router data or forwarding plane (“virtual router”) 206A is programmed by the virtual router agent 314 using forwarding information to achieve fast packet paths. The virtual router agent 314 can be a user-space process. The virtual router agent 314 can have a northbound interface 340 for receiving configuration and routing information from a control plane process such as cRPD 324. cRPD 324 can be... Figure 1A and Figure 1B An example of cRPD 24A. Virtual router agent 314 has a southbound interface 341 for programming virtual router 206A. The term "virtual router" as used herein may specifically refer to the virtual router forwarding plane, or to a combination of the virtual router forwarding plane (e.g., virtual router 206A) and the corresponding virtual router agent (e.g., virtual router agent 314).

[0126] cRPD 324 may have a northbound interface for exchanging configuration and routing information with SDN controller 70. Containerized networking interface 312 may be a CNI plugin that configures the interface of container workloads (pods 328A to 328L in this example) via DPDK-based virtual router 206A. Orchestrator 50 may orchestrate the workloads of DPDK-based virtual router 206A, cRPD 324, and / or pod 328 on server 300 via requests sent to orchestration agent 310 on server 300. In some cases, workloads may have multiple interfaces and multiple types of interfaces (e.g., some with virtual router 206A, some with NIC 321A). Therefore, CNI 312 may represent a combination of CNIs or unified CNIs capable of configuring workloads with multiple types of interfaces. Multiple CNIs may be controlled by a master CNI such as Multus. When orchestrator 50 is a Kubernetes host, custom resource limits (CRDs) can be implemented for orchestrator 50 to support multi-tenancy and network isolation.

[0127] Orchestrator 50 orchestrates pods (e.g., pod328A-328L) including container workloads via orchestration agent 310. CNI 312 configures a virtual interface between the pod and the data plane, which can be a DPDK-based virtual router, a kernel-based virtual router, or a SmartNIC-based virtual router. Figure 3 In the example shown, virtual router 206A is a DPDK-based virtual router. In situations such as... Figure 3 In some of the examples shown, CNI 312 configures the virtual interface of each pod as the vhost user interface of a DPDK-based virtual router 206A. In some examples, CNI 312 configures the veth pair of each pod to the virtual router 206A. In some examples, the virtual router 206A has a bonded interface to NIC 321B, which may be a DPDK-enabled Intel-based NIC. The bonded interface facilitates packet load balancing between the structural interfaces. Additional description of configuring the virtual interface can be found in U.S. Patent 10,728,145, published July 28, 2020, which is incorporated herein by reference in its entirety.

[0128] In Kubernetes deployments, CNI 312 provides networking for application workloads. This includes, for example, configuring interfaces, managing IP addresses, and access control lists; announcements include... Figure 1A and Figure 1B The reachability of workloads within the Kubernetes cluster (e.g., worker nodes) of either server 300 or server 12; establishing network namespaces.

[0129] cRPD 324 can include elements of Network Service Mesh Architecture (NSM), service discovery, external endpoints, and tunneling. cRPD 324 can use external routing protocols such as Border Gateway Protocol (BGP) to advertise pod reachability to both inside and outside the Kubernetes cluster. cRPD 324 can use interior gateways and other routing protocols such as IS-IS, OSPF, and Label Distribution Protocol (LDP) to participate in lower-layer networking. cRPD 324 can also use protocols / technologies such as MPLS, MPLSoUDP or MPLSoGRE, VxLAN; SR-MPLS, SRv6, SRv4, and / or IPSec tunneling technologies to provide support for higher-level L3VPN layers.

[0130] Virtual router 206A exposes the corresponding interface 382A of physical interface 322 to kernel 380. That is, for each physical interface, virtual router 206A exposes the interface to kernel 380. Each interface 382A can be a vhost interface and can terminate at the default VRF 381A. Kernel 380 can therefore send and receive network packets with virtual router 206A via interface 382.

[0131] In some examples, cRPD 324 runs routing protocols and can exchange routing protocol messages with routers outside of server 300. Furthermore, cRPD 324 can utilize the kernel 380 network stack to obtain network topology information of the underlying network. cRPD 324 can use this topology information to establish routing protocol adjacencies with external routers. Interface 382A provides cRPD 324 with access to physical interface 322 via kernel 380 and virtual router 206A, and thus access to the underlying network accessible via physical interface 322. As an example, such an underlying network may include a mid-range network, a switching infrastructure for the local data center where server 300 resides, etc. Virtual router 206A can be configured with a route that causes virtual router 206A to forward network packets received at one of physical interfaces 322 and destined for the corresponding IP address of the corresponding interface in interface 382 to kernel 380 via that corresponding interface in interface 382A.

[0132] Kernel 380 outputs network packets to cRPD 324 via interface 384. Interface 384 can represent a system call interface / API exposed by kernel 380, file system, threads, sockets, or other mechanisms such as cRPD 324 that can be used to receive packets from and inject packets into kernel 380. cRPD 324 operates as the control plane of the router implemented by server 300, while the DPDK-based virtual router 206A operates as the router's fast path forwarding plane. Figure 1A In the 5G environment shown, cRPD 324 operates as the control plane for CNR 20A, while virtual router 206A and kernel 380 operate as pod 328 and / or CNR 20A. Figure 1B It provides a data or forwarding plane. Utilizing the CNI 312 of cRPD324, L3VPN can be used to facilitate multi-tenancy, for example, to implement network slicing for different tenants; ACLs and network policies for applications; and IPSec for high security. The DPDK-based virtual router 206A acts as a data or forwarding plane, forwarding data traffic between pod328 and physical interface 322 in a manner that excludes kernel 380.

[0133] Server 300 has two data planes for packet forwarding: a first data plane 394 implemented by kernel 380 and a second data plane 392 implemented by virtual router 206A. The second data plane 392 can be... Figure 1B The implementation of the hybrid data plane 150. The DPDK-based virtual router 206A is configured with "ownership" of physical interface 322. In some respects, the DPDK-based virtual router on server 300, such as virtual router 206A, is configured with ownership of all physical interfaces of NIC 321 on server 300. Physical interface 322 can be a VPN attachment circuit for VRF 12. Physical interface 322 can be associated with a corresponding interface of virtual router 206A, through which virtual router 206A sends and receives traffic via physical interface 322.

[0134] The first data plane 394 and the second data plane 392 can store different routes for the lower-layer network and the upper-layer network, respectively. The first data plane 394 and the second data plane 392 can independently perform forwarding lookups and forward traffic using their respective stored routes. cRPD 324 is a routing protocol process for handling lower-layer and upper-layer routes. After learning routes through routing protocols or from the SDN controller 70, cRPD 324 can selectively program lower-layer routes into the kernel 380 and upper-layer routes into the virtual router 206A (via virtual router agent 314). In some respects, the server 300 can be configured and operated as described below, such that the different data planes 394 and 392 are combined into a hybrid data plane.

[0135] Figure 4 This is a block diagram illustrating an example server with example control and data service flows within the server according to the technology of this disclosure. Server 400 may be similar to the server described herein, such as... Figure 1A , Figure 1B and Figure 2 Server 12, and Figure 3 Server 300. pod328A-328L can be DU in some cases. The vhost interface and vhost0 interface 382A are exposed to kernel 380 by virtual router 206A, and in some cases, are exposed to virtual router 206A by kernel 380. The vhost interface 382A has an associated lower-level host IP address for receiving traffic "at the host". Therefore, kernel 380 can be a network endpoint of the lower-level network including server 400 as a network device, which has the IP address of vhost interface 382A. The application layer endpoint can be cRPD 324 or other processes managed by kernel 380.

[0136] Lower-layer networking refers to the physical infrastructure that provides connectivity between nodes (typically servers) in a network. The lower-layer network is responsible for delivering data packets over this infrastructure. Lower-layer network devices use routing protocols to determine IP connections. Typical routing protocols used for routing purposes on lower-layer network devices are OSPF, IS-IS, and BGP. Upper-layer networking refers to the virtual infrastructure that provides connectivity between virtual workloads (typically VMs / pods). This connection is built on top of the lower-layer network and allows for the construction of virtual networks. Upper-layer services (i.e., virtual networking) are typically encapsulated in IP / MPLS tunnels or other tunnels routed by the lower-layer network. Upper-layer networks can operate across all or a subset of lower-layer network devices and are multi-tenant via virtualization.

[0137] Control traffic 402 can represent routing protocol traffic executed by cRPD 324 using one or more routing protocols. In server 400, control traffic 402 can be received via physical interface 322 owned by virtual router 206A. Virtual router 206A is programmed with a route to the host IP address of vhost0 interface 382A and a receive next hop, causing virtual router 206A to send traffic received at physical interface 322 and destined for the host IP address of vhost0 interface 382A to kernel 380 via vhost0 interface 382A. From the perspective of cRPD 324 and kernel 380, all such control traffic 402 will appear to originate from vhost0 interface 382A. Therefore, cRPD 324 routes vhost0 interface 382A as the forwarding next hop for that route. cRPD 324 selectively installs some routes to virtual router agent 314 and the same (or others) routes to kernel 380, as described further below. Virtual router agent 314 will receive Forwarding Information Base (FIB) updates corresponding to some routes received by cRPD 324. These routes will be directed to vHost0 interface 382A, and virtual router 206A can automatically translate or map vHost0 interface 382A to physical interface 322.

[0138] Routing information programmed by cRPD 324 can be categorized into lower and upper layers. cRPD 324 will install lower-layer routes to the kernel 380 because it may require reachability to establish additional protocol adjacencies / sessions with external routers, such as BGP multi-hop sessions over reachability provided by IGP. cRPD 324 supports selective filtering of FIB updates to specific data planes (e.g., kernel 380 or virtual router 206A) using routing policy construction that allows matching with RIBs, route instances, prefixes, or other attributes.

[0139] The control traffic 402 sent by cRPD 324 to the virtual router 206A via the vhost0 interface 382A can be sent out by the virtual router 206A from the corresponding physical interface 322 for the vhost0 interface 382A.

[0140] like Figure 4As shown, CNI 312 can create a virtual network (referred to herein as a "pod") interface for each of the applications pods 328A-328L when notified by orchestrator 50 via orchestration agent 310. One end of the pod interface terminates at the container included in the pod. CNI 312 can request virtual router 206A to begin monitoring the other end of the pod interface, and cRPD 324 facilitates the exclusive forwarding of traffic from physical interface 322 destined for application containers in DPDK-based pods 328A-322L using DPDK, without involving kernel 380. The reverse process applies to traffic initiated by pods 328A-328L.

[0141] However, because the DPDK-based virtual router 206A manages these virtual network interfaces of pods 328A-328L, the virtual network interfaces are unknown to the kernel 380. Server 400 can use a tunnel dedicated to the DPDK forwarding path to internally send and receive upper-layer data traffic 404 between the DPDK-based pods 328A-328L, the virtual router 206A, and the NIC 321B.

[0142] Thus, in server 400, cRPD 324 is connected to two data planes: kernel 380 and the DPDK-based virtual router 206A. cRPD 324 utilizes the kernel 380 network stack to establish routes dedicated to DPDK fast paths. cRPD 324 receives routing information including both lower-layer and upper-layer routing information. cRPD 324 runs routing protocols on the vHost interface 382A, which is visible in kernel 380, and cRPD 324 can install FIB updates in kernel 380 corresponding to the routes learned from IGP (lower-layer routing information). This enables the establishment of multi-hop iBGP sessions to those destinations indicated in such IGP-learned routes. Again, cRPD 324 routing protocol adjacencies involve kernel 380 (and vHost interface 382A) because kernel 380 executes the network stack.

[0143] Virtual Router Agent 314 of Virtual Router 206A notifies cRPD 324A of the application pod interfaces of pods 328A-328L. These pod interfaces are created by CNI 312 and exclusively managed by Virtual Router Agent 314 (i.e., not involving kernel 380). These pod interfaces are unknown to kernel 380. cRPD 324 can advertise the reachability to these pod interfaces to the rest of the network as L3VPN routes including Network Layer Reachability Information (NLRI). In the context of a 5G mobile network, such L3VPN routes can be stored in the VRF of Virtual Router 206A for different network slices. The corresponding MPLS routes can be programmed by cRPD 324 to Virtual Router 206A only, without being programmed to kernel 380, via interface 340 with Virtual Router Agent 314. This is because the next hop for these MPLS labels is a pop-up and forwarding to the pod interface used by one of pods 328A-328L; these interfaces are only visible in virtual router 206A and not in kernel 380. Similarly, reachability information received on BGP L3VPN can be selectively programmed to virtual router 206A by cRPD 324, as such routes are only needed for forwarding traffic generated by pods 328A-328L. The routes programmed to virtual router 206A constitute upper-layer routes for the upper network.

[0144] Figure 5 This is a conceptual diagram depicting a sequence of operations in a virtual router that leads to routing planning, according to exemplary aspects of this disclosure. The sequence of operations is described relative to components of server 400, but can be performed by components of any server described in this disclosure, such as... Figure 1A , Figure 1B and Figure 2 Server 12 or Figure 4Server 400. CNI 312 has a block of IP addresses reserved for pods. Virtual Router Agent 314 listens for port add and port delete messages, such as those concerning the saving service, where "port" corresponds to a virtual network interface. CNI 312 sends a port add message to Virtual Router Agent 314 (502). The port add message includes the identifier of the virtual network for the port and the IP address assigned by CNI 312 to the pod. (CNI 312 can configure the pod at the other end of the virtual network interface separately.) Virtual Router Agent 314 creates a virtual network interface (here referred to as a Virtual Machine Interface or VMI, which is an example of a virtual network interface) in interface 540 (504). Virtual Router Agent 314 uses the VMI add message and the default VRF identifier to configure the virtual network interface in Virtual Router 206A (506). Virtual Router Agent 314 subscribes to cRPD 324 instead of the SDN controller using a VMI subscription message, which includes the virtual network name and IP address received in the port add message (508). cRPD 324 sends a VMI Config message to Virtual Router Agent 314, which contains the correct VRF identifier for the virtual network interface (512). Optionally, a VRF add message is used to add the VRF to Virtual Router Agent 314 if needed (510). Virtual Router Agent 314 sends a VMI Update message with the correct VRF identifier to Virtual Router 206A, causing Virtual Router 206A to attach the virtual network interface to the correct VRF (514). cRPD 324 assigns a service label and uses a route add message to Virtual Router Agent 314 to add routes and next hops (e.g., MPLS routes for BGP IP-VPN) (516). cRPD 324 also advertises routes to the pod to its peer routers (518), which may include other cRPDs, routers in the underlying network, or other routers. Virtual Router Agent 314 uses the route forwarding information received in the route addition message from cRPD 324 to configure Virtual Router 206A (520).

[0145] Figure 6A This is a block diagram of an example computing device (e.g., a host) based on the techniques described in this disclosure. Figure 6A The computing device 600 can represent a real or virtual server, and can represent Figure 1A and Figure 1B Server 12 Figure 3 Server 300 or Figure 4This is an exemplary instance of any of the servers 400. In this example, computing device 600 includes a bus 642 that couples hardware components of the computing device 600 hardware environment. Bus 642 couples a network interface card (NIC) 630, a storage disk 646, and one or more microprocessors 610 (hereinafter, "microprocessor 610"). NIC 630 may have SR-IOV capability. In some cases, a front-side bus may couple microprocessor 610 and memory device 644. In some examples, bus 642 may couple memory device 644, microprocessor 610, and NIC 630. Bus 642 may represent a Peripheral Component Interface (PCI) Faster (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 642. In some examples, components coupled to bus 642 control DMA transfers between components coupled to bus 642.

[0146] The microprocessor 610 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, the instructions being stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC), or may be combined within one or more multi-core processors (or “many-core” processors), wherein each multi-core processor is implemented using a single IC (i.e., a chip multiprocessor).

[0147] Disk 646 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technique for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape pod, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by the microprocessor 610.

[0148] Memory 644 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM, or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and can be accessed by a computer. Main memory 644 provides a physical address space consisting of addressable memory locations.

[0149] Network interface card (NIC) 630 includes one or more interfaces 632 configured to exchange data packets using links of the underlying physical network. Interface 632 may include a port interface card with one or more network ports. NIC 630 may also include on-card memory for, for example, storing data packets. Direct memory access transfers between NIC 630 and other devices coupled to bus 642 can read from / write to the NIC memory.

[0150] Memory device 644, NIC 630, storage disk 646, and microprocessor 610 provide an operating environment for a software stack, which includes an operating system kernel 380 executing in kernel space. Kernel 380 may represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel, available from Microsoft. In some instances, the operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include Kernel-Based Virtual Machine (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open-source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). The operating system including kernel 380 provides an execution environment for one or more processes in user space 645.

[0151] The kernel 380 includes a physical driver 625 for using a network interface card 630. The network interface card 630 can also implement SR-IOV to allow for the use of containers such as 629A-629B or one or more virtual machines (…). Figure 6A Physical network functions (I / O) are shared between one or more virtual execution elements (not shown in the diagram). Shared virtual devices, such as virtual functions, provide dedicated resources so that each virtual execution element can access the dedicated resources of the NIC 630, thus appearing as a dedicated NIC for each virtual execution element. Virtual functions can represent physical functions used with the physical driver 625 as well as lightweight PCIe functions that share physical resources with other virtual functions. For an SR-IOV-enabled NIC 630, the NIC 630 can have thousands of available virtual functions according to the SR-IOV standard, but for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0152] Computing device 600 can be coupled to a physical network switching infrastructure, which includes an upper-layer network of software or "virtual" routers that extend the switching infrastructure from physical switches to physical servers coupled to the switching infrastructure. This upper-layer network includes virtual routers 206A and 206B (collectively referred to as "virtual router 206"). The virtual router can be a process or thread or a component thereof executed by a physical server, such as... Figure 1A and Figure 1B Server 12 Figure 3 Server 300 and / or Figure 4 Server 400 dynamically creates and manages one or more virtual networks that can be used to communicate between virtual network endpoints. In one example, the virtual router uses an upper-layer network to implement each virtual network, which provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which that endpoint is running. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various technologies can be used to transmit data packets within and across virtual networks over the physical network. The term "virtual router" as used herein can include Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other devices and / or software residing on a host device and performing the exchange, bridging, or routing of data packets between virtual network endpoints in one or more virtual networks, wherein the virtual network endpoints are hosted by one or more servers 12. Figure 6A In the example computing device 600, virtual router 206A executes as a DPDK-based virtual router in user space, and virtual router 206B executes in kernel space 380. In some respects, in various implementations, the virtual router can execute within a hypervisor, host operating system, host application, or virtual machine.

[0153] The Virtual Router 206 can replace and include the virtual routing / bridging functionality of a Linux Bridge / OVS module, which is typically used in Kubernetes deployments of pods including pods 602A-602N (collectively referred to as "pod 602"). Figure 6A In the example shown, pod 602A includes one or more application containers 629A, and pod 602B includes one or more application containers 629B. pod 602N includes an instance of cRPD 324.

[0154] Virtual router 206 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 206 can perform networking services such as application security policies, NAT, multicast, mirroring, and load balancing.

[0155] exist Figure 6A In the example shown, virtual router 206A is executed as a DPDK procedure in user space 656. Virtual router agent 314 can also be executed in user space. Figure 2 In the example computing device 600, virtual router 206A executes as a DPDK-based virtual router in user space; however, in various implementations, virtual router 206A may execute within a hypervisor, host operating system, host application, or virtual machine. Virtual router agent 314 has a connection to SDN controller 70 using a channel, which is used to download configuration and forwarding information. Virtual router agent 314 programs this forwarding state to the virtual router data (or "forwarding") plane represented by virtual routers 206A and 206B. Virtual router 206A and virtual router agent 314 may be processes.

[0156] Virtual router 206 can be multi-threaded and execute on one or more processor cores. Virtual router 206 can include multiple queues. Virtual router 206 can implement a packet processing pipeline. The corresponding virtual router agent 314 can stitch the pipeline together in a manner ranging from the simplest to the most complex, depending on the operations to be applied to the packets. Virtual router 206 can maintain multiple forwarding base instances. Virtual router 206 can use RCU (Read Copy Update) locks to access and update tables (e.g., table 627).

[0157] To send data packets to other compute nodes or switches, virtual router 206A uses one or more physical interfaces 632. Typically, virtual router 206 exchanges upper-layer data packets with workloads such as VMs or pods 602. Virtual router 206 may have multiple virtual network interfaces (e.g., vifs). These interfaces may include a kernel interface vhost0 for exchanging data packets with the host operating system; an interface pkt0 for the corresponding virtual router agent 314 to obtain forwarding status from the network controller and send exception packets. One or more virtual network interfaces may exist corresponding to one or more physical network interfaces 632.

[0158] Other virtual network interfaces of virtual router 206 can be used to exchange packets with workloads. These virtual network interfaces can be any of the types of virtual interfaces described above. In some cases, virtual network interface 612 can be a tap interface.

[0159] The Virtual Router 206B is a kernel-based virtual router. In this case, the Virtual Router 206B is installed as a kernel module within the operating system. The Virtual Router 206B registers itself with the TCP / IP stack to receive packets from any desired operating system interface. This interface can be a bond, physical interface, tap (for VMs), veth (for containers), etc. Figure 6A In the example shown, virtual router 206B has a veth interface with the vRF 622B of virtual router 206B. In this mode, virtual router 206B relies on the operating system to send and receive packets from different interfaces. For example, the operating system can expose a tap interface supported by the vhost-net driver to communicate with the VM. Once virtual router 206B registers packets from this tap interface, the TCP / IP stack sends all packets to it. Virtual router 206B sends packets via the operating system interface. Furthermore, NIC queues (physical or virtual) are handled by the operating system. Packet processing can operate in interrupt mode, which generates interrupts and may result in frequent context switching. When high packet rates are present, the overhead associated with frequent interrupts and context switching can overwhelm the operating system and lead to poor performance.

[0160] In such Figure 6A In the DPDK-based deployment of the illustrated virtual router 206A, the virtual router 206A is installed as a user-space application 645 linked to the DPDK library. This can result in faster performance than a kernel-based deployment, especially in the presence of high packet rates. The physical interface 632 is used by the DPDK's polling-mode driver (PMD) instead of the kernel's interrupt-based driver. The registers of the physical interface 632 can be exposed to user space 645 for PMD access; the physical interface 632 bound in this way is no longer managed by or visible to the host operating system, and the DPDK-based virtual router 206A manages the physical interface 632. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the virtual router 206A's DPDK data plane. The nature of this "polling-mode" makes the packet processing / forwarding of the virtual router 206A's DPDK data plane much more efficient than the interrupt-mode approach at high packet rates. Compared to the kernel-mode virtual router 206A, there are relatively fewer interrupts and context switches during packet I / O, and in some cases, interrupts and context switches during packet I / O can be completely avoided.

[0161] Typically, each pod 602 can be assigned one or more virtual network addresses for use within a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by virtual router 206. Pod 602B can be assigned its own virtual Layer 3 (L3) IP address, for example, for sending and receiving communications, but may not know the IP address of the computing device 600 on which pod 602B executes. Therefore, the virtual network address can be different from the logical address of the underlying physical computer system (e.g., computing device 600).

[0162] Virtual router agent 314 controls the upper layer of the virtual network of computing device 600 and orchestrates the routing of data packets within computing device 600. Typically, virtual router agents 314A and 314B work in conjunction with the SDN controller 70 of the virtualization infrastructure. Figure 1A , Figure 1B and Figure 2 The SDN controller generates commands to create virtual networks and configure network virtualization endpoints, such as computing device 600, and more specifically, corresponding virtual routers 206A and 206B, and virtual network interfaces 612A and 612B. By configuring virtual router 206A based on information received from the SDN controller 70, virtual router agent 314 can support configuring network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0163] In one example, network packets generated or consumed by container 629A within a virtual network domain, such as Layer 3 (L3) IP packets or Layer 2 (L2) Ethernet packets, can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted over the physical network. Packets transmitted within the virtual network may be referred to herein as “internal packets,” while physical network packets may be referred to herein as “outbound packets” or “tunnel packets.” The encapsulation and / or decapsulation of virtual network packets within physical network packets can be performed by virtual router 206A. This function is referred to herein as tunneling and can be used to create one or more upper-layer networks. Besides IPinIP, other example tunneling protocols that can be used include IP over Generic Routing Encapsulation (GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), etc. Virtual router 206A performs tunneling encapsulation / decapsulation on packets originating from / to any container of pod 602, and virtual router 206A exchanges packets with pod 602 via a bridge of bus 642 and / or NIC 630.

[0164] As described above, the SDN controller 70 can provide a logically centralized controller to facilitate the operation of one or more virtual networks. The SDN controller 70 can, for example, maintain a routing information base, such as one or more routing tables storing routing information for the physical network and one or more upper-layer networks. Virtual routers 206 implement one or more Virtual Router and Forwarding Instances (VRFs) 622A-622B for the corresponding virtual network, wherein the corresponding virtual router 206 operates as a corresponding tunneling technology endpoint. Typically, each VRF 622 stores forwarding information for the corresponding virtual network and identifies where packets will be forwarded and whether packets will be encapsulated in a tunneling technology protocol, such as using a tunneling technology header that may include one or more headers for different layers of the virtual network protocol stack. Each of the VRFs 622 may include a network forwarding table storing routing and forwarding information for the virtual network.

[0165] NIC 630 can receive tunneled packets and forward them to the appropriate virtual router. As an example, virtual router 206A processes tunneled packets to determine the virtual networks of the source and destination endpoints of the internal packets based on the tunnel encapsulation header. Virtual router 206A can strip the Layer 2 header and tunnel encapsulation header to forward only internal packets internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN label or MPLS label, indicating the virtual network, such as the virtual network corresponding to VRF 622A. VRF 622A may include forwarding information for internal packets. For example, VRF 622A may map the destination Layer 3 address of the internal packets to virtual network interface 212. VRF 622A forwards the internal packets to pod 602A in response via virtual network interface 212.

[0166] Pod 602A includes one or more application containers 629A, while pod 602B includes one or more application containers 629B. Containers such as 629A or 629B can also act as source internal packets for source virtual network endpoints. Container 629A, for example, can generate Layer 3 internal packets destined for a destination virtual network endpoint executed by another computing device (i.e., non-computing device 600) or another container. Container 629A can send Layer 3 internal packets to virtual router 206A via virtual network interface 212 attached to VRF 622A.

[0167] Virtual router 206A receives internal data packets and Layer 2 headers, and determines the virtual network used for the internal data packets. Virtual router 206A can use any of the aforementioned virtual network interface implementation technologies (e.g., macvlan, veth, etc.) to determine the virtual network. Virtual router 206A uses a VRF622A corresponding to the virtual network used for the internal data packets to generate an external header for the internal data packets. This external header includes an external IP header for upper-layer tunneling and a tunnel encapsulation header identifying the virtual network. Virtual router 206A encapsulates the internal data packets together with the external header. Virtual router 206A can utilize a new Layer 2 header to encapsulate tunnel packets, which has a header that is compatible with devices external to computing device 600 (e.g., TOR switch 116). Figure 1B The destination layer 2 address is associated with either the computing device 600 or server 12. If the destination is outside the computing device 600, the virtual router 206A outputs a tunneled packet with a new layer 2 header to the NIC 630. The NIC 630 outputs the packet on its outbound interface. If the destination is another virtual network endpoint executed on the computing device 600, the virtual router 206A routes the packet to virtual network interface 212. Figure 2 The appropriate one of them.

[0168] In some examples, Figure 1A , Figure 1B and Figure 2 The controller of the computing device 600 (e.g., SDN controller 70) configures a default route in each pod 602 such that the pod uses virtual routers (e.g., virtual routers 206A and 206B) as the initial next hop for outbound packets.

[0169] Container platform 604 includes container engine 608, orchestration agent 310, service agent 611, and CNI 312. Container engine 608 includes code executable by microprocessor 610. Container engine 608 can be one or more computer processes. Container engine 608 runs containerized applications in the form of containers 629A-629B. Container engine 608 can represent Dockert, rkt, or other container engines used to manage containers. Typically, container engine 608 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating containers. A container is an executable instance of an image. Based on instructions from orchestration agent 310, container engine 608 can obtain images and instantiate them as executable containers in pods 602A-602B.

[0170] Service agent 611 includes code executable by microprocessor 610. Service agent 611 can be one or more computer processes. Service agent 611 monitors the addition and removal of services and endpoint objects, and it maintains the network configuration of computing device 600 to ensure communication between pods and containers, such as the use of services. Service agent 611 can also manage iptables to capture traffic to the virtual IP address and port of services and redirect traffic to the proxy port of the proxy backup pod. Service agent 611 can represent the Kube agent of the worker nodes of the Kubernetes cluster. In some examples, container platform 604 does not include service agent 611, or service agent 611 is disabled to support CNI 312 configuration of virtual router 206A and pod 602.

[0171] The orchestration agent 310 includes code executable by the microprocessor 610. The orchestration agent 310 can be one or more computer processes. The orchestration agent 310 can represent a Kubelet of a worker node in a Kubernetes cluster. The orchestration agent 310 is an agent for the orchestrator, for example... Figure 1A , Figure 1B and Figure 2 The orchestrator 50 receives container specification data for the container and ensures that the container is executed by the compute device 600. The container specification data may be a manifest file sent from the orchestrator 50 to the orchestration agent 310, or a manifest file received indirectly via a command-line interface, HTTP endpoint, or HTTP server. The container specification data may be the pod specification (e.g., a podSpec – a YAML (Yellow Markup Language) or JSON object describing the pod) of one of the pods 602 of container 629. Based on the container specification data, the orchestration agent 310 instructs the container engine 608 to obtain and instantiate the container image of container 629 so that container 629 can be executed by the compute device 600.

[0172] Orchestration agent 310 instantiates or otherwise invokes CNI 312 to configure one or more virtual network interfaces for each pod 602. For example, orchestration agent 310 receives container specification data for pod 602A and instructs container engine 608 to create pod 602A with container 629A based on the container specification data for pod 602A. Orchestration agent 310 can also receive container specification data for pod 602B and instruct container engine 608 to create pod 602B with container 629B based on the container specification data for pod 602B. Orchestration agent 310 also invokes CNI 312 to configure virtual network interface 612A corresponding to VRF 622A for pod 602A and virtual network interface 612B corresponding to VRF 622B for pod 602B. In this example, pod 602A is the virtual network endpoint corresponding to VRF 622A, and pod 602B is the virtual network endpoint corresponding to VRF 622B.

[0173] CNI 312 obtains interface configuration data for configuring the virtual network interface of pod 602. Virtual router agents 314A and 314B operate as virtual network control plane modules to enable SDN controller 70 to configure virtual routers 206A and 206B, respectively. Unlike the orchestration control plane (including container platform 604 for worker nodes and master nodes, such as orchestrator 50) which manages the provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including SDN controller 70 for worker nodes and virtual router agent 314) manages the configuration of the virtual network, partially implemented in the data plane by the virtual routers of the worker nodes. Virtual router agent 314 transmits interface configuration data for the virtual network interface to CNI 312 so that the orchestration control plane element (i.e., CNI 312) can configure the virtual network interface according to the configuration state determined by SDN controller 70, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Additionally, this allows CNI 312 to obtain interface configuration data for multiple virtual network interfaces of a pod and configure multiple virtual network interfaces, which reduces the communication and resource overhead inherent in calling CNI 312 individually to configure each virtual network interface.

[0174] Figure 6B yes Figure 6A The following is a block diagram illustrating an example computing device operating as an orchestrator master node in a cluster of virtualized computing infrastructure. In this example, computing device 600 can represent one or more real or virtual servers. Therefore, in some instances, computing device 600 can implement one or more master nodes for a given cluster. Figure 6B In the example shown, the computing device includes a scheduler 652, an API server 650, a network controller manager 656, a network controller 654, a network controller manager 655, and a configuration storage 658.

[0175] Although the scheduler 652, API server 650, network controller manager 656, network controller 654, network controller manager 655, and configuration storage 658 are illustrated and described as being executed by a single computing device 600, they can be distributed among multiple computing devices 600 constituting a computing system or hardware / server cluster. In other words, each of the multiple computing devices 600 can provide a hardware operating environment for one or more instances of any one or more of the scheduler 652, API server 650, network controller manager 656, network controller 654, network controller manager 655, or configuration storage 658. Network controller 654 can represent... Figure 1A , Figure 1B and Figure 2 An exemplary instance of SDN controller 70. Scheduler 652, API server 650, controller manager 656, and network controller manager 655 may implement exemplary instances of orchestrator 50. Network controller manager 655 may represent an example implementation of Kubernetes cloud controller manager or Kube manager. Network controller 654 may represent an exemplary instance of SDN controller 70.

[0176] API server 650, scheduler 652, controller manager 656, and configuration storage can implement the master node of the cluster and are alternatively referred to as the "master component". The cluster can be a Kubernetes cluster, and the master node can be a Kubernetes master node, in which case the master component is a Kubernetes master component.

[0177] API server 650 includes code executable by microprocessor 610. API server 650 can be one or more computer processes. API server 650 authenticates and configures data for objects such as virtual execution elements (e.g., pods of containers), services, and replication controllers. A service can be an abstraction that defines a logical set of pods and policies for accessing those pods. The set of pods that implement the service is selected based on the service definition. A service can be partially implemented as a load balancer or otherwise incorporate a load balancer. API server 650 can implement a Representational State Transfer (REST) ​​interface to handle REST operations and provide a front-end to shared state of the appropriate cluster stored in configuration storage 658. API server 650 can authenticate and authorize requests. API server 650 communicates with other components to instantiate virtual execution elements in compute infrastructure 8. API server 650 can represent a Kubernetes API server.

[0178] Configuration store 658 is a backup store for all cluster data. Cluster data can include cluster state and configuration data. Configuration data can also provide a backend for service discovery and / or provide locking services. Configuration store 658 can be implemented as a key-value store. Configuration store 658 can be a central database or a distributed database. Configuration store 658 can represent an etcd storage. Configuration store 658 can represent the Kubernetes configuration store.

[0179] Scheduler 652 includes code executable by microprocessor 610. Scheduler 652 can be one or more computer processes. Scheduler 652 monitors newly created or requested virtual execution elements (such as pods of containers) and selects worker nodes on which the virtual execution elements will run. Scheduler 652 can select worker nodes based on resource requirements, hardware constraints, software constraints, policy constraints, location, etc. Scheduler 652 can represent a Kubernetes scheduler.

[0180] Typically, API server 650 can invoke scheduler 652 to schedule virtual execution elements, which can select worker nodes and return an identifier for the selected worker node to API server 650. API server 650 can then write this identifier to configuration store 658 associated with the virtual execution element. API server 650 can invoke orchestration agent 310 for the selected worker node. Figure 6A This allows the container engine 608 (used for selecting worker nodes) to be used. Figure 6AThe virtual execution element is obtained from the storage server and created on the worker node. The orchestration agent 310 for the selected worker node can update the state of the virtual execution element to the API server 650, which persists the new state to the configuration storage 658. Thus, the computing device 600 instantiates the computing infrastructure 100, 108 (…). Figure 1A and Figure 1B The new virtual execution element in ).

[0181] Controller Manager 656 includes code executable by microprocessor 610. Controller Manager 656 can be one or more computer processes. Controller Manager 656 can be embedded in a core control loop to monitor the shared state of the cluster by receiving notifications from API server 650. Controller Manager 656 can attempt to move the cluster state towards a desired state. Example controllers (not shown) managed by Controller Manager 656 may include a replication controller, endpoint controller, namespace controller, and service account controller. Controller Manager 656 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, pod termination garbage collection, cascading deletion garbage collection, node garbage collection, etc. Controller Manager 656 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0182] Network controller 654 includes code executable by microprocessor 610. Network controller 654 may include one or more computer processes. Network controller 654 may represent... Figure 1A , Figure 1B An exemplary instance of the SDN controller 70. The network controller 654 may be a logically centralized but physically distributed SDN controller responsible for providing management, control, and analysis functions for the virtualized network. Specifically, the network controller 654 may be a logically centralized control plane and management plane of the computing infrastructures 100, 108, and orchestrates virtual routers for one or more worker nodes.

[0183] Network controller 654 can provide cloud networking for computing architectures operating on network infrastructure. Cloud networking can include private clouds for enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) for cloud service providers (CSPs). Private cloud, VPC, and IaaS use cases can involve multi-tenant virtualized data centers, such as those related to... Figure 1AAs described, in such a scenario, multiple tenants in a data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is allocated its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; virtual networks). These logical resources are isolated from each other unless specifically permitted by security policies. Virtual networks in the data center may also be interconnected to physical IP VPNs or L2 VPNs.

[0184] The network controller 654 can provide network function virtualization (NFV) to networks such as business edge networks, broadband subscriber management edge networks, and mobile edge networks. NFV involves arranging and managing networking functions, such as firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, and wide area network (WAN) optimization, in virtual machines, containers, or other virtual execution elements rather than on physical hardware devices. In this market, the primary driver for network service virtualization is time-to-market and cost optimization.

[0185] The network controller 654 programs network infrastructure components to create virtual networks and can create interface configurations for virtual network interfaces used in virtual networks.

[0186] Additional information regarding exemplary network controllers can be found in International Application No. PCT / US2013 / 044378 and U.S. Patent Application No. 14 / 226,509, which are incorporated herein by reference.

[0187] Network controller manager 655 includes code executable by microprocessor 610. Network controller manager 655 can be one or more computer processes. Network controller manager 655 operates as an interface between orchestration-oriented components (e.g., scheduler 652, API server 650, controller manager 656, and configuration store 658) and network controller 654. Typically, network controller manager 655 monitors for new objects (e.g., pods and services) in the cluster. Network controller manager 655 can isolate pods in virtual networks and connect pods to services.

[0188] The Network Controller Manager 655 can be executed as a container on the master node of a cluster. In some cases, using the Network Controller Manager 655 makes it possible to disable the service broker on worker nodes (e.g., the Kubernetes Kube broker), allowing virtual routers to be used for connectivity across all pods, as described in this article.

[0189] Network Controller Manager 655 can use the orchestration platform's controller framework to listen for (or otherwise monitor) changes to objects defined in the API and add annotations to some of these objects. Annotations can be labels or other identifiers specifying object attributes (e.g., "virtual network green"). Network Controller Manager 655 can use the interface to Network Controller 654 to create network solutions for applications, defining network objects such as virtual networks, virtual network interfaces, and access control policies. Network Controller 654 can implement network solutions in the computing infrastructure, for example, by configuring one or more virtual networks and virtual network interfaces in a virtual router.

[0190] Figure 7 This is a block diagram of an example computing device (e.g., a server) including a hybrid data plane, based on the techniques described in this disclosure. Figure 7 In the example shown, server 700 includes network interfaces 721A-721N (collectively referred to as "network interface 721"), kernel 380, DPDK virtual router 206A, CNI 312, and cRPD 324. Server 700 can be... Figure 1A , Figure 1B , Figure 2 , Figure 3 , Figure 4 , Figure 6A and Figure 6B The implementation can be any of servers 12, 300, 400, and 600. Additionally, in this example, server 700 hosts pods 722A-722D (collectively referred to as "pod 722"). Kernel 380, DPDK Virtual Router 206A, CNI312, and cRPD 324 can be combined in the implementation of cloud-native router 702.

[0191] The cloud-local router 702 includes a hybrid data plane 734, which merges the kernel data plane 732 and the DPDK data plane 730. The hybrid data plane 734 can be... Figure 1B The implementation of the hybrid data plane 150. The kernel data plane 732 is implemented by the kernel 380 and the network stack 712. In some aspects, the kernel 380 may include a virtual router module as part of the kernel data plane 732. Figure 7 (Not shown in the image). The DPDK data plane 730 is implemented by the virtual router 206A.

[0192] NIC 721 can be NIC 321 ( Figure 3 and Figure 4 ) and / or NIC 630 ( Figure 6A and Figure 6BThe implementation of ) is as follows. In some respects, each of the NIC 721s is assigned to the DPDK data plane 734, and no NIC 721 is assigned to the kernel data plane 732.

[0193] In some respects, the Kubernetes nodes on server 700 are modeled as PE routers, and each pod is modeled as a CE router. cRPD 324 can be configured to run in "host mode". Pod 722 can be linked to the hybrid data plane 734 via one or both of the kernel data plane 732 and the DPDK data plane 734. Figure 7 In the example shown, pods 722A and 722B are linked to kernel data plane 732 via VRF 707A of CRPD 324. Pod 722D is linked to virtual router 206A of DPDK data plane 734. Pod 722 can be modeled as a multi-homed CE, meaning that a pod can connect to multiple VRFs. Furthermore, VRFs can exist in different data planes within the mixed data plane 734. Figure 7 In the example shown, pod 722C is multihomed and is linked to the kernel data plane 732 via VRF 707A and to the virtual router 206A of the DPDK data plane 734 via VRF 707B.

[0194] Responding to the orchestrator 50 ( Figure 1A and Figure 1B When pod 722 is created, cRPD 324 dynamically creates one or more virtualized network interfaces. For example, when orchestrator 50 creates a pod configured to use kernel data plane 732, cRPD 324 can create a veth pair that communicatively couples the pod to the VRF of cRPD 324. cRPD 324 terminates the veth link at one end of the pod, and the other end terminates within the VRF on cRPD 324. Figure 7In the example shown, when orchestrator 50 creates pods 722A-722C, cRPD 324 creates a veth pair between the corresponding pod and VRF 707A. Because cRPD 324 has access to the kernel 380 interface and network stack 712, various routing protocols can be used between pods 722A-722C and cRPD 324. Examples of such protocols include E-BGP, IS-IS, and OSPF. Additionally, cRPD 324 can leverage the kernel 380 and network stack 712 to apply various types of upper-layer network protocols. Examples of such upper-layer protocols include L3VPN using different lower-layer / tunneling methods, E-VPN (Type 2 / Type 5): MPLS, SR-MPLS, SRv6, MPLS over UDP / IPIP / GRE tunnels, etc. Furthermore, cRPD 324 can leverage the kernel 380 and network stack 712 to provide support for protocols such as IPsec, SRv6, and IP-IP. In addition, elements of kernel data plane 732 provide support for applications and workloads that utilize kernel 380’s native networking APIs (e.g., BSD sockets).

[0195] In response to orchestrator 50 creating a pod configured to use the DPDK data plane, cRPD 324 can create a vHost interface of the VRF of a DPDK-based virtual router that communicatively couples the pod to the DPDK data plane 730. Figure 7 In the example shown, when orchestrator 50 creates pods 722C and 722D, cRPD 324 creates a vhost interface between the corresponding pods and the VRF 707B of the DPDK-based virtual router 206A.

[0196] In some respects, cRPD 324 creates VRF / routing instances for interfaces in the kernel data plane 732, which are separate from the VRF / routing instances that cRPD 324 creates for interfaces in the DPDK data plane 730. For example, in Figure 7 In the example shown, cRPD 324 creates VRF 707A for kernel data plane 732 in DPDK data plane 730, and creates VRF 707B for virtual router 707B.

[0197] Additionally, cRPD 324 can create vhost interfaces for each network interface port that has been assigned to the virtual router 206A. As an example, the NIC 721N assigned to the virtual router 206A has two ports: port0 and port1. cRPD 324 creates two vhost interfaces (vhost0 and vhost1), corresponding to port0 and port1 respectively.

[0198] Network packets originating from a pod 722 can be referred to as "outbound packets". Outbound packets transmitted to VRF 707A (e.g., outbound packets from the veth interfaces of pods 722A, 722B, and pod 722C) are directed to kernel interfaces (e.g., the veth interface of cRPD 324). In this case, route lookup and encapsulation can be performed in kernel 380 and / or network stack 712. Kernel 380 and network stack 712 create fully formed L2 packets for the outbound packets, which are then handed over to DPDK data plane 730 via the vhost interface. The packet processor 710 of virtual router 206A is connected via NIC 721 ( Figure 7 In the example, NIC 721N sends packets. In some respects, apart from sending outbound packets via NIC 721, packet processor 710 does not perform any network processing on packets received from network stack 712 and / or kernel 380, and thus acts as a "pass-through" entity.

[0199] exist Figure 7 In the example shown, outbound packets sent from a pod configured to use DPDK are directly provided to the VRF 707B. In this example, each of pods 722C and 722D is configured to use DPDK. In this case, the virtual router 206A performs standard DPDK processing on the outbound packets and creates packets to be routed via NIC 721 (…). Figure 7 The example uses NIC721N for transmission.

[0200] Network packets originating from a network source outside of server 700, received by one of the network interfaces 721, can be referred to as "inbound packets." Because all network interfaces of server 700 are assigned to the DPDK data plane 730, inbound packets will be received at the DPDK-based virtual router. Figure 7 In the example, an inbound packet for one of pods 722 is received by virtual router 206A. The packet processor 710 of virtual router 206A can inspect the inbound packet and use various criteria to determine whether the packet requires kernel-based processing. Packet processor 710 operates as a "pathway" for packets it determines require kernel-based processing. Packet processor 710 delivers such packets to network stack 712 for delivery to the appropriate VRF. Figure 7 In the example shown, aside from passing inbound packets to network stack 712 for processing and passing them to VRF 707A, packet processor 710 does not perform any network processing on inbound packets destined for veth interfaces with pods 722A, 722B, and pod 722C.

[0201] cRPD 324 can install forwarding state for DPDK-based virtual routers in both the kernel data plane 732 and the DPDK data plane 730. For example, cRPD 324 creates a logical representation of a VRF that is created for a DPDK data plane with a routing table substantially the same as the corresponding DPDK VRF. Figure 7 In the example, VRF 707B' is a logical representation of VRF 707B and includes essentially the same routing table as VRF 707B, with minor differences. For instance, the routing tables of VRF 707B and 707B' may contain the same set of routing entries / prefixes, but will have different values ​​in the next-hop field of the route / prefix processed by the kernel data plane. For such entries, VRF 707B has a routing table entry with a next-hop value indicating that the packet is delivered to the kernel. For the same prefix, VRF 707B' has a routing table entry with a next-hop value indicating how the kernel 380 decapsulates the packet and performs a lookup in VRF 707B''s routing table. Therefore, for packets with a destination address indicating that the packet will be processed by kernel 380, cRPD 324 can program the routing table of VRF 707B to set the next hop to an interface of kernel 380 (e.g., vhost0 or vhost1), instructing packet processor 710 to forward the packet to kernel 380, for example, via the vhost interface. When kernel 380 receives the packet via the vhost interface, it can determine that the next hop field in the routing table entry associated with the destination address is a loopback address, instructing the kernel to process the inbound packet.

[0202] Packet processor 710 can perform standard DPDK processing on inbound packets with a destination indicating a VRF in the DPDK data plane 730. For example, packet processor 710 can perform DPDK processing on inbound packets with a destination having a vhost interface of pod 722C or pod 722D.

[0203] Figure 8A and Figure 8B This is a sequence diagram illustrating example processing of outbound and inbound data packets according to the technology of this disclosure. Figure 7 Discussed in the context of the cloud-local router 702 shown. Figure 8A and Figure 8B .exist Figure 8A and Figure 8BIn the example shown, two pods, 722A and 722D, send and receive network packets. Pod 722D is configured to utilize DPDK-based networking. Based on this configuration, cRPD 324 configures a vhost connection to communicatively couple pod 722D to VRF707B. Pod 722A is configured to use kernel-based networking. Based on this configuration, cRPD 324 configures a veth pair to communicatively couple pod 722A to VRF 707A.

[0204] Figure 8A This is an example sequence diagram illustrating example processing of outbound data packets according to the technology of this disclosure. Figure 8A In the example shown, each of pods 722A and 722D sends outbound packets. In operation 810, pod 722A sends an outbound packet. Pod 722A is configured to use the kernel data plane; therefore, the outbound packet will be referred to as a kernel-based outbound packet. VRF707A receives the outbound packet, and in operation 812, kernel 380 and / or network stack 712 process the outbound packet. Such processing may include a "start list". After processing in operation 812, the kernel-based outbound packet will be a fully formed L2 packet. In operation 814, network stack 712 forwards the packet to DPDK virtual router 206A via the vhost interface. The packet processor 710 of the virtual router 206A receives kernel-based outbound packets (such as those encapsulated by the network stack 712). Because the virtual router receives packets via the vhost interface between itself and the network stack 712, at operation 816, the virtual router 206A performs no packet processing other than forwarding the kernel-based outbound packets to the appropriate physical port. As an example, if the virtual router 206A receives a packet from the network stack 712 via the vhost1 interface, the packet processor 710 will act as a pass-through device and forward the packet to the NIC 721N for transmission via port1 (which corresponds to vhost1).

[0205] In operation 818, pod 722D sends outbound packets. Pod 722D is configured to use the DPDK data plane; therefore, the outbound packets will be referred to as DPDK-based outbound packets. Virtual router 206A's VRF 707B can receive outbound packets via the vhost interface or via the DPDK API. In operation 820, virtual router 206A performs DPDK-based processing on the packets and forwards the DPDK-based outbound packets to NIC 721N.

[0206] Operation 810-820 can be used with Figure 8AThe order shown differs from the execution order. For example, pod722D can send outbound packets before pod722A. Furthermore, Figure 8A The operations shown can occur simultaneously.

[0207] Figure 8B This is a technical illustration of an example sequence diagram of an inbound data packet processing example, based on the technical illustrations of this disclosure. Figure 8B In the example shown, NIC 721N receives inbound packets. As mentioned above, each of the NICs 721 is assigned to the DPDK data plane. Therefore, in operation 830, NIC 721N provides the inbound packets to virtual router 206A. Packet processor 710 performs packet inspection 832 to determine whether the inbound packet has a DPDK-based pod as its destination, or whether the inbound packet will be provided to kernel 380 for further processing, or whether the packet will be processed by virtual router 206A (384). Packet processor 710 can use various rules and heuristics to determine whether to provide the inbound packet to kernel 380 for further processing. For example, in some aspects, packet processor 710 can determine the packet encapsulation type and provide the packet to kernel 380 for processing if the encapsulation type is a type not supported by the DPDK-based virtual router. Examples of such encapsulations include IP-IP, VX-LAN, GRE, MPLS, EVPN, IPsec, SRV6, etc. As an example, packet processor 710 can establish a "delivery to host path," which specifies the vhost interface that packet processor 710 should use when packets have encapsulation that requires kernel-based processing (e.g., ...). Figure 7 (vhost0 or vhost1 in the example).

[0208] In some respects, the packet processor can determine where to forward packets to kernel 380 based on the destination address. As mentioned above, the routing table in virtual router 206A can be programmed to set the next-hop address for kernel 380. In this case, the packet is forwarded to kernel 380 for further processing.

[0209] In some respects, packet processor 710 can determine that it does not know how to handle packets. In this case, packet processor 710 can forward packets to kernel 380 under the assumption that kernel 380 will know how to handle them. As an example, packet processor 710 can establish an "exception-to-host path" that specifies the vhost interface (e.g., ...) to be used by packet processor 710 when it encounters a packet it does not know how to handle. Figure 7(e.g., vhost0 or vhost1 in the example). For instance, if packet processor 710 encounters a label in the packet header that it doesn't know how to handle, an "exception-to-host" path can be used to forward the packet for kernel-based processing.

[0210] If packet processor 710 determines that a packet will be provided to kernel 380, then in operation 836 (the "yes" branch of 834), packet processor 710 provides the inbound packet to the network stack 712 of kernel 380 for further processing by kernel 380 using network stack 712 (838). In this case, packet processor 710 does not perform any processing on the inbound packet other than forwarding the packet to kernel 380. For example, it bypasses the DPDK processing operation of virtual router 206A and does not modify any TTL value of the inbound packet.

[0211] If packet processor 710 determines that the packet has a DPDK pod as its destination and does not require kernel processing (the "No" branch of 834), then virtual router 206A performs standard DPDK packet processing and, in operation 842, provides the packet to the DPDK-based pod 722D using the vhost interface or the DPDK API.

[0212] Figure 9 This is a sample deployment specification for a pod based on the technology disclosed herein. Figure 9 The example shown is YAML, not a markup language (YAML) format. Figure 9 In the example shown, the pod named "odu-pod1" will be connected to two networks upon creation. First network definition 902 indicates that the first interface "net1" will be used to connect the pod to the network named "vswitch-pod1-bd100". The "dataplane" field has a "linux" value, indicating that this interface will be created for the kernel data plane (e.g., the Linux kernel data plane). Second definition 904 indicates that the second interface "net2" will be used to connect the pod to the network named "vswitch-pod1-bd200". The "dataplane" field has a "DPDK" value, indicating that the pod supports DPDK communication and that this interface will be created for the DPDK data plane.

[0213] Figure 10This is a flowchart illustrating the operation of a DPDK-based virtual router. The DPDK-based virtual router receives data packets destined for containerized applications from the physical interface (1005). The DPDK-based virtual router determines whether the kernel network stack, executed by the processing circuitry, should perform tunneling processing on the data packets (1010). In response to determining that the kernel network stack should perform tunneling processing on the first data packet, the DPDK-based virtual router forwards the data packet to the kernel network stack (1015).

[0214] The techniques described herein can be implemented using hardware, software, firmware, or any combination thereof. The various components, functional units, and / or modules shown or described in the figures and / or elsewhere in this disclosure can perform this operation using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and / or executing on one or more computing devices. For example, a computing device can utilize multiple processors or multiple devices to execute one or more such modules. A computing device can execute one or more such modules, such as a virtual machine executing on underlying hardware. One or more such modules can execute as one or more services of an operating system or computing platform. One or more such modules can execute as one or more executable programs at the application layer of a computing platform. In other examples, the functionality provided by the modules can be implemented by dedicated hardware devices. Although certain modules, data stores, components, programs, executables, data items, functional units, and / or other items included within one or more storage devices may be shown individually, one or more of these items can be combined and operate as a single module, component, program, executable, data item, or functional unit. For example, one or more modules or data stores can be combined or partially combined such that they operate or provide functionality as a single module. Furthermore, one or more modules can operate in combination with each other, such that, for example, one module acts as a service or extension of another module. Additionally, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may include multiple components, subcomponents, modules, submodules, data stores, and / or other components or modules or data stores not shown. Furthermore, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device can be implemented in various ways. For example, each module, data store, component, program, executable file, data item, functional unit, or other item shown within the storage device may be implemented as part of an operating system that executes on a computing device.

[0215] If implemented in hardware, the invention is applicable to a device, such as a processor or integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0216] Computer-readable media can form part of a computer program product, which may include packaging material. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0217] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that can change over time (e.g., in RAM or cache).

[0218] The code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in this invention can be provided within a software module or a hardware module.

Claims

1. A system for virtualizing computing infrastructure, the system comprising: Processing circuitry; Containerized applications; Physical interface; A virtual router based on the Data Plane Development Kit (DPDK) is configured to execute on the processing circuitry to send and receive data packets via the physical interface, wherein the physical interface is included in a plurality of physical interfaces of the system, and each of the plurality of physical interfaces is assigned to the DPDK-based virtual router. as well as The kernel network stack is configured to execute on the processing circuitry to perform tunneling processing on the packets destined for the containerized application and received by the DPDK-based virtual router via the physical interface. The DPDK-based virtual router is further configured to: in response to determining that the kernel network stack needs to perform the tunneling process on the data packet, forward the data packet to the kernel network stack without any network layer processing.

2. The system of claim 1, wherein, The kernel network stack is also configured to perform the tunneling process on packets initiated by the containerized application and sent by the DPDK-based virtual router via the physical interface.

3. The system according to claim 1, wherein, For each port among the multiple physical interfaces, a corresponding vhost interface is configured between the DPDK-based virtual router and the kernel network stack.

4. The system according to claim 3, wherein, The DPDK-based virtual router is configured to send data packets received via the vhost interface via the port corresponding to the vhost interface.

5. The system according to claim 3, wherein, The DPDK-based virtual router is configured to send the data packets received via the port to the kernel network stack via the vhost interface corresponding to the port.

6. The system according to claim 1, wherein, The DPDK-based virtual router is configured to determine whether to send the inbound network data packet to the kernel network stack for the tunneling process based on the encapsulation type of the inbound network data packet.

7. The system according to claim 1, wherein, The DPDK-based virtual router is configured to determine whether to send the inbound network packets to the kernel network stack for tunneling processing based on the labels of the inbound network packets.

8. The system according to claim 1, wherein, The DPDK-based virtual router is configured to determine whether to send the inbound network data packet to the kernel network stack for tunnel transmission processing based on the determination that the DPDK-based virtual router cannot process the inbound network data packet.

9. The system according to claim 1, wherein, The DPDK-based virtual router is configured to determine whether to send inbound network packets to the kernel network stack for tunnel transmission processing based on routing information in the routing table of the DPDK-based virtual router.

10. A method for virtualizing computing infrastructure, the method comprising: A virtual router based on the Data Plane Development Kit (DPDK), executed by processing circuitry, receives a first data packet destined for a containerized application from a physical interface, wherein the physical interface is included in a plurality of physical interfaces of the system, and each of the plurality of physical interfaces is assigned to the DPDK-based virtual router. The DPDK-based virtual router, executed by the processing circuitry, determines whether the kernel network stack should perform tunneling processing on the first data packet; as well as In response to determining that the kernel network stack needs to perform the tunneling process on the first data packet, the DPDK-based virtual router forwards the first data packet to the kernel network stack without any network layer processing.

11. The method according to claim 10, wherein, The method further includes: The kernel network stack receives the second data packet from the containerized application; The kernel network stack performs the tunneling process on the second data packet; The kernel network stack forwards the processed second data packet to the DPDK-based virtual router; and The processed second data packet is sent by the DPDK-based virtual router to the destination network device via the physical interface.

12. The method of claim 10, further comprising configuring a corresponding vhost interface between the DPDK-based virtual router and the kernel network stack for each port of the plurality of physical interfaces.

13. The method of claim 12, further comprising: The third data packet is received from the kernel network stack via the vhost interface by the DPDK-based virtual router; as well as The third data packet is sent by the DPDK-based virtual router via the port corresponding to the vhost interface.

14. The method of claim 12, further comprising: The third data packet is received by the DPDK-based virtual router via one of the multiple ports; as well as The DPDK-based virtual router sends the third data packet to the kernel network stack via the vhost interface corresponding to the port.

15. The method according to claim 10, wherein, Determining whether the kernel network stack should perform the tunneling process for the first data packet includes determining whether the kernel network stack should perform the tunneling process based on one or more of the encapsulation type of the first data packet, the label of the first data packet, and the routing table.

16. The method of claim 10, wherein, Determining whether the kernel network stack should perform the tunneling process for the first data packet includes determining whether the kernel network stack should perform the tunneling process based on the DPDK-based virtual router's determination that the DPDK-based virtual router cannot process inbound network data packets.

17. A computer-readable storage medium having computer-executable instructions, which, when executed, cause processing circuitry executing a DPDK-based virtual router and kernel network stack to perform the method according to any one of claims 10-16.

Images