Computing device, method for computer network and computer readable storage medium

Abstract

The invention relates to a computing device, a method for a computer network, and a computer readable storage medium. An example computing device includes a containerized router and processing circuitry. In one embodiment, a computing device includes a containerized virtual router configured to execute on a processing circuit and configured to implement a data plane for the containerized router. In one embodiment, a computing device includes a containerized routing protocol process configured to execute on a processing circuit and configured to implement a control plane for a containerized router, where the containerized routing protocol process is configured to execute a routing protocol. The computing device includes a process configured to execute on a processing circuit and configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by a containerized routing protocol process via a routing protocol.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 345,707, filed on June 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to virtualized computing infrastructure, and more particularly, to a containerized router for use in a cloud computing environment. Background Art

[0003] In a typical cloud data center environment, a large number of interconnected servers provide computing and / or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide user support equipment located in various physical hosting facilities for infrastructures located around the world.

[0004] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have widely utilized virtualized environments, wherein virtual hosts (also referred to herein as virtual execution elements, e.g., virtual machines or containers) are deployed on and executed on the underlying computing platform of physical computing devices.

[0005] Virtualization in a data center, or any environment containing one or more servers, can provide several advantages. One advantage is that virtualization can significantly improve efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. A second advantage is that virtualization provides efficient control over the computing infrastructure. As physical computing resources become fungible, for example, in cloud-based computing environments, the provisioning and management of computing infrastructure becomes easier. Therefore, in addition to the efficiency and higher return on investment (ROI) provided by virtualization, enterprise IT staff often prefer virtualized computing clusters in the data center because of their management advantages.

[0006] Containerization is a virtualization scheme based on operating system-level virtualization. Containers are lightweight and portable execution elements for applications that are isolated from each other and from the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and executed as a single lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers offer the promise of consistent operation from one computing environment to another (virtual or physical).

[0007] Due to the inherent lightweight nature of containers, a single host can typically support many more container instances than regular virtual machines (VMs). Containers are typically short-lived, can be created and moved more efficiently than VMs, and can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the requirements of container networking solutions: the network should be agile and scalable. VMs, containers, and bare metal servers may need to coexist in the same computing environment to enable communication between different application deployments. Container networking should also be agnostic to the multiple orchestration platforms used to deploy containerized applications.

[0008] Managing the deployment and infrastructure of computing infrastructure for application execution can involve two main roles: (1) Orchestration - used to automate the deployment, scaling, and operation of applications across a cluster of hosts and to provide computing infrastructure, which may include container-centric computing infrastructure; and (2) Network management - used to create virtual networks in the network infrastructure to enable packetized communication between applications running on virtual execution environments (e.g., containers or VMs) and between applications running on conventional (e.g., physical) environments. Software-defined networks facilitate network management.

[0009] A cloud data center environment (e.g., a virtualized data center) can provide services to any number of customers. For example, a cloud data center environment can provide services to different customers via a virtual private cloud (VPC). A VPC can be viewed as an extension of a customer-operated local (on-prem) network, which can be located on the premises of the customer site. Therefore, by using a VPC, a customer can quickly and efficiently expand their local network as needed by increasing or decreasing resources residing in the cloud data center through the VPC, rather than changing the resources of the local network itself. VPC implementations may include the use of a virtual private cloud gateway device. Summary of the invention

[0010] A VPC is a virtual network of computing nodes that can be similar to a conventional network that an enterprise can operate in a local (on-prem) data center or an extension of such a conventional network. A VPC can be a service provided by a service provider in a cloud computing infrastructure. Since a cloud computing infrastructure can include many resources that can be allocated to various customers, customers can quickly and efficiently expand their own network resources by using a VPC, avoiding the need to physically expand a local data center. A VPC may include one or more subnets. A subnet may include a range of IP addresses for the VPC. A VPC gateway may connect the VPC to a network outside the VPN. A VPC can provide a local data center with a connection to a virtual computing instance (e.g., pods, virtual machines, and / or other types of deployable virtual computing instances) running in a cloud computing environment, thereby expanding the computing power of the local data center without physically adding additional resources to the local data center.

[0011] A VPC gateway is a physical or virtual device that a cloud service provider can use to provide customers with access to a VPC. In some examples, a VPC gateway may not provide a mechanism to interact with the underlying network using standard network protocols. For example, a VPC gateway may not be able to read or learn BGP prefixes. In other words, a VPC gateway may not be a BGP speaker. Therefore, the VPC gateway itself may not be able to route or exchange BGP traffic to or from BGP-speaking virtual computing instances in a data center environment. Therefore, it may be necessary to provide seamless connectivity between customer workloads in a VPC and on-premises infrastructure.

[0012] A containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged and deployed as a container. cRPD can be executed in the user space of a host as a containerized process. Thus, cRPD makes available the rich routing software pedigree of physical routers on compute nodes. cRPD provides control plane functionality for containerized routers. Existing implementations of cRPD (running on the host) use forwarding provided by the kernel. The control plane is therefore containerized. In some examples, the compute nodes and kernels can be Linux-based.

[0013] A virtual router is a software entity that provides data plane functionality on a compute node, while cRPD can provide control plane functionality. Compute nodes can host centrally orchestrated and provisioned virtual compute instances. In some cases, a virtual router can further interact with an SDN controller to create an overlay network by exchanging routing, configuration, and other data. For example, a virtual router can run as either a kernel module or a DPDK-based process. DPDK allows a virtual router to process more packets per second than when running as a kernel module. The virtual router data plane can be containerized. Therefore, in combination, a containerized cRPD and a containerized virtual router can be a fully functional containerized router.

[0014] In general, the present disclosure describes techniques for providing routing control plane functionality for VPC workloads reachable via a VPC gateway using a containerized router operating in a public cloud environment. The containerized router can execute a routing protocol to learn routes associated with the VPC workloads, and can program a VPC routing table in a VPC gateway device so that the VPC gateway device routes traffic to the containerized router for further routing. Such traffic can include traffic from one or more containers (also referred to as pods). This arrangement can be beneficial because the pods and local routers can include a Border Gateway Protocol (BGP) speaker, but the VPC gateway may not support BGP and therefore cannot process BGP advertisements or BGP-advertised routing information to determine how to route overlay traffic originating from or destined for workloads in the VPC, such as BGP traffic.

[0015] The present disclosure also describes techniques for utilizing redundant containerized routers in a public cloud environment. The redundant containerized routers may be connected to each other, for example, via a virtual router redundancy protocol (VRRP) session. A first containerized router of the redundant containerized routers may determine, via the VRRP session, that the first containerized router is an active containerized router. The first containerized router may be connected to a virtual private cloud (VPC) gateway and program a VPC routing table in the VPC gateway.

[0016] In one example, a computing device includes a containerized router, the computing device including: processing circuitry; a containerized virtual router configured to execute on the processing circuitry and configured to implement a data plane for the containerized router; a containerized routing protocol process configured to execute on the processing circuitry and configured to implement a control plane for the containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol; and a process configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by the containerized routing protocol process via the routing protocol.

[0017] In one example, a method includes: connecting to a virtual private cloud gateway by a process, the process being configured to program a virtual private cloud (VPC) routing table in the VPC gateway based on routing information received by a containerized routing protocol process via a routing protocol; and programming a VPC routing table in the VPC gateway by a process configured to program the VPC routing table. In one example, a non-transitory computer-readable medium stores instructions that, when executed, cause a processing circuit to perform the method.

[0018] In another example, a non-transitory computer-readable medium stores instructions that, when executed, cause a processing circuit to connect to a virtual private cloud gateway through a process configured to program a virtual private cloud (VPC) routing table in the virtual private cloud gateway based on routing information received by a containerized routing protocol process via a routing protocol; and program a VPC routing table in the VPC gateway through the process configured to program the VPC routing table.

[0019] In another example, a computing device includes a first containerized router, the computing device including: processing circuitry; a containerized virtual router configured to execute on the processing circuitry and configured to implement a data plane for the containerized router; and a containerized routing protocol process configured to execute on the processing circuitry and configured to implement a control plane for the first containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol, wherein the first containerized router is configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by the containerized routing protocol process via the routing protocol, and wherein the first containerized router is communicatively coupled to a second containerized router via a virtual router redundancy protocol (VRRP) session.

[0020] In another example, a method includes: connecting, by a first containerized router, to a second containerized router via a virtual router redundancy protocol (VRRP) session; determining, via the VRRP session, that the first containerized router is an active containerized router among the first containerized router and the second containerized router; connecting to a VPC gateway; and programming, by a process configured to program a VPC routing table and based on the first containerized router being the active containerized router, a VPC routing table in the VPC gateway.

[0021] In another example, a non-transitory computer-readable medium stores instructions that, when executed, cause a processing circuit to connect to a second containerized router via a virtual router redundancy protocol (VRRP) session; determine, via the VRRP session, that a first containerized router is an active containerized router among the first containerized router and the second containerized router; connect to a VPC gateway; and program a VPC routing table in the VPC gateway based on the first containerized router being the active containerized router.

[0022] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a block diagram illustrating an example computing infrastructure in accordance with one or more aspects of the present disclosure.

[0024] Figure 2 is a block diagram of an example cloud computing environment including a CNR that programs a VPC routing table of a VPC gateway according to one or more aspects of the present disclosure.

[0025] Figure 3 is a block diagram of an example cloud computing environment with redundant CNRs according to one or more aspects of the present disclosure.

[0026] Figure 4 is a block diagram of an example computing device (eg, host) in accordance with techniques described in this disclosure.

[0027] Figure 5 is a flow chart illustrating an example CNR VPC technique according to one or more aspects of the present disclosure.

[0028] Figure 6 is a flow chart illustrating an example redundant CNR VPC technique according to one or more aspects of the present disclosure.

[0029] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0030] A cloud-native approach can include functional blocks that are decomposed into microservices. Microservices are deployed as containers on compute nodes and orchestrated by an orchestration system, such as Kubernetes (abbreviated as "K8s"). Kubernetes deploys containers using pods. K8s networking between pods is implemented via a plugin called the Container Network Interface (CNI) (also known as the Container Network Interface plugin). However, the networking capabilities of a typical CNI are fairly basic and are not appropriate when the containerized networking functions of the CNI service play a critical role in networks such as telecommunications networks. As described in this article, CNR is more suitable for these situations. CNR is a containerized router that allows compute nodes (e.g., x86 or ARM-based hosts) to become first-level members of the network routing system, participate in protocols such as Intermediate System to Intermediate System (IS-IS) and Border Gateway Protocol (BGP), and provide transport and multi-tenancy based on Multiprotocol Label Switching / Segment Routing (MPLS / SR). In other words, the platform is not an adjunct to the network (e.g., a customer edge (CE) router) and can operate as a provider edge (PE) router or other device within a provider network, for example, in a cloud computing environment.

[0031] The CNR has one or more advantages over conventional routers. A router has a control plane and a forwarding plane. The control plane participates in dynamic routing protocols and exchanges routing information with other routers in the network. The results are downloaded to the forwarding plane in the form of prefixes, next hops, and associated SR / MPLS labels. The implementation described in this article is modular in the sense that the control plane is unaware of the exact details of how the forwarding plane is implemented. In a hardware router, the forwarding plane may be based on a custom ASIC. In contrast, the CNR is a virtualized router. However, in both cases, the functionality of the routing protocol software is similar. This means that the CNR benefits from the same highly comprehensive and powerful protocol implementations as the hardware-based routers that power some of the world's largest networks.

[0032] CNR uses a containerized routing protocol daemon (cRPD) control plane and a virtual router (vrouting) forwarding plane to deliver high-performance networking in a small package size that is functionally similar to a non-virtualized router, a physical network function (PNF). The forwarding plane can be implemented via a choice of DPDK, Linux kernel, or Smart-NIC. A K8s CNI-compatible package is fully integrated and can be deployed within a K8s environment (e.g., supporting Multus).

[0033] Since CNR itself is a cloud-native application, it supports installation using K8s manifests or Helm Charts. These include the initial configuration of the router, including routing protocols and Layer 3 VPNs that support slicing. CNR can be orchestrated and configured in seconds, with all routing protocol adjacencies established and running with the rest of the network. Ongoing configuration changes during the lifecycle of CNR (for example, adding or removing network slices) can be implemented via a choice of CLI, K8s manifests, NetConf, or Terraform.

[0034] By adopting the K8s CNI framework, CNR can reduce the conventional operational overhead that occurs when using containerized devices instead of physical devices. By exposing the appropriate device interfaces, CNR can standardize the operating model of virtual devices to physical devices, removing barriers to adoption in operators' network operation environments. CNR can present a familiar routing device look and feel to any well-trained operations team. CNR has similar features and capabilities and a similar operating model to hardware-based platforms. Similarly, domain controllers can communicate with and control CNR using the protocols they use with any other Junos router, such as Netconf / OpenConfig, gRPC, Path Computation Element Protocol (PCEP), and Programmable Routing Daemon (pRPD) API.

[0035] Nodes executing CNR can participate in IS-IS, Open Shortest Path First (OSPF), BGP, and / or other internal or external routing protocols. In addition, MPLS can be used, typically based on Segment Routing (SR). There are two reasons for this: to allow traffic engineering if needed, and to support multi-tenancy by using MPLS-based Layer 3 VPNs. As an alternative, SRv6 can be used to meet these requirements. Having comprehensive routing capabilities is also necessary to implement network slicing. Each slice tenant is placed in its own Layer 3 VPN. From the perspective of the Layer 3 VPN, the CNR acts as a provider edge (PE) router. Therefore, the CNR exchanges Layer 3 VPN prefixes with other PE routers in the network via BGP, regardless of whether these other PEs are physical routers or CNRs residing on other hosts. Each tenant can be placed in a separate VRF table on each PE, providing an appropriate degree of isolation and security between tenants, just like a regular Layer 3 VPN service. This neatly solves the problem that K8s does not provide this isolation natively. Layer 3 VPN is a tried and tested method to achieve multi-tenancy in the network, trusted by many large companies around the world who purchase this service from their network service providers.

[0036] A VPC gateway, which may be deployed and used by a cloud service provider to provide customers with access to a VPC, may not be able to process routing protocol messages using conventional routing protocols (e.g., BGP) to obtain routing information (e.g., advertised routes and prefixes). For example, the VPC gateway may not be a BGP speaker. Therefore, the VPC gateway may not itself be able to process BGP routes, and therefore may not be able to perform exchange or routing of BGP messages or other traffic to or from BGP-speaking pods in a data center environment based on BGP-advertised routes. Because, in contrast, CNR is able to learn BGP routes and is able to exchange and route BGP and other traffic based on BGP-advertised prefixes, it may be desirable to utilize CNR to program the VPC routing table of the VPC gateway so that the VPC gateway can forward traffic to the CNR via the underlying network for further routing. In this way, the CNR directly programs the underlying elements of the cloud infrastructure to forward traffic to the CNR itself for further routing.

[0037] Figure 1 is a block diagram illustrating an example computing infrastructure according to one or more aspects of the present disclosure. The computing infrastructure 8 may represent a software defined network (SDN) architecture system for virtual networks. Example use cases for the SDN architecture include 5G mobile networks and cloud and enterprise cloud native use cases. The SDN architecture may include data plane elements implemented in computing nodes (e.g., servers 12) and network devices such as routers or switches, and the SDN architecture may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. Reference Figure 1 The described SDN architecture including the network controller 24 is only one example. The techniques of the present disclosure may be applied to other data center architectures and SDN controller deployments and implementations.

[0038] In this example, the SDN architecture components are microservices, and the SDN architecture uses a basic container orchestration platform to manage the lifecycle of the SDN architecture components. The SDN architecture can leverage Kubernetes constructs such as Multus, authentication and authorization, cluster APIs, Kube Alliance, Kube Virt, and Kata containers. The SDN architecture can support Data Plane Development Kit (DPDK) pods, and the SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies. The SDN architecture can be integrated with orchestration platforms such as Kubernetes, OpenShift, Mesos, OpenStack, VMware vSphere (e.g., orchestrator 23), and can also be integrated with service provider operations support systems / business support systems (OSS / BSS).

[0039] Typically, one or more data centers 10 provide an operating environment for applications and services at a customer site 11 (illustrated as "customer 11"), which has one or more customer networks coupled to the data center through a service provider (e.g., through a VPC) network 7. In some examples, the customer site 11 can be an on-prem customer network. For example, each data center 10 can host infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 15, which can represent one or more networks managed by other providers and can therefore form part of a large-scale public network infrastructure (e.g., the Internet). The public network 15 can represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0040] Although customer sites 11 and public networks 15 are primarily illustrated and described as edge networks of service provider network 7, in some examples, one or more of customer sites 11 and public networks 15 may be tenant networks within any of data centers 10. For example, data center 10 may host multiple tenants (customers), each tenant being associated with one or more virtual private networks (VPNs), each of which may implement one customer site 11.

[0041] The service provider network 7 provides packet-based connectivity to connected customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement multi-protocol label switching (MPLS) forwarding, and in this case may be referred to as an MPLS network or MPLS backbone. In some cases, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0042] In some examples, each of the data centers 10 may represent one of many geographically distributed network data centers, which may be connected to each other via a service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1As shown in the example of , data center 10 may include facilities for providing network services to customers. Customers of a service provider may be collective entities, such as businesses, governments, or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although illustrated as an independent edge network of a service provider network 7, elements of data center 10 (e.g., one or more physical network functions (PNFs) or virtualized network functions (VNFs)) may be included within the kernel of the service provider network 7.

[0043] In this example, data center 10 includes storage and / or computing servers (or "nodes") interconnected via a switch fabric 14 provided by one or more layers of physical network switches and routers, with servers 12A through 12X (herein "servers 12") depicted as coupled to top-of-rack switches 16A through 16N. Servers 12 are computing devices and may also be referred to herein as "computing nodes," "hosts," or "host devices." Although in Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10 may include many additional servers coupled to other TOR switches 16 in data center 10.

[0044] The data center can provide a virtual private cloud (VPC) to any customer 11. The data center 10 may include one or more VPC gateways, such as a VPC gateway 7, which can be used to connect the VPC to an external network (e.g., local, other clouds, external VPCs) to provide VPC services to customers 11. The VPC gateway 7 can be a physical device or a virtual device, such as a physical or virtual gateway, a router, a switch, etc. In some examples, the VPC gateway 7 can be implemented by one or more servers 12. In some examples, the CNR can be directly coupled to the VPC gateway. This document will refer to Figure 2 and Figure 3 Discuss VPC Gateway further.

[0045] The switch fabric 14 in the illustrated example includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A to 16N (collectively, "TOR switches 16") that are coupled to a distribution layer of chassis (or "spine" or "core") switches 18A to 18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as, for example, physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0046] In this example, the TOR switch 16 and the chassis switch 18 provide redundant (multi-homed) connections to the IP structure 20 and the service provider network 7 for the server 12. The chassis switch 18 aggregates traffic flows and provides connections between the TOR switches 16. The TOR switch 16 can be a network device that provides Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functions. The TOR switch 16 and the chassis switch 18 can each include one or more processors and memories, and can execute one or more software processes. The chassis switch 18 is coupled to the IP structure 20, and the IP structure can perform Layer 3 routing to route network traffic between the data center 10 and the customer site 11 through the service provider network 7. The switching architecture of the data center 10 is only an example. For example, other switching architectures can have more or fewer switching layers. The IP structure 20 can include one or more gateway routers.

[0047] The term "packet flow", "traffic flow", or simply "flow" refers to a group of packets originating from a specific source device or endpoint and sent to a specific destination device or endpoint. A single packet flow can be identified by a 5-tuple: for example, <source network address, destination network address, source port, destination port, protocol>. This 5-tuple generally identifies the packet flow to which the received packets correspond. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for the packet.

[0048] The servers 12 may each represent a computing, server, or storage server. For example, each server 12 may represent a computing device, such as an x86 processor-based server, configured to operate according to the techniques described herein. The servers 12 may provide a network function virtualization infrastructure (NFVI) for the NFV architecture.

[0049] Any of the servers 12 may configure virtual execution elements (e.g., pods or virtual machines) by virtualizing the server's resources to provide some measure of isolation between one or more processes (applications) executing on the server. "Hypervisor-based" or "hardware level" or "platform" virtualization refers to the creation of virtual machines, each of which includes a client operating system for executing one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are virtualized from the physical hardware of the host server, the executed applications are isolated from the hardware of the host and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0050] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., one of data centers 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly allowed by security policy.

[0051] Virtual networks can use the data center 10 gateway router ( Figure 1 The virtual network can be connected to and extended over physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and Ethernet virtual private network (EVPN) networks (not shown). The virtual network can also be used to implement network function virtualization (NFV) and service chaining.

[0052] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks, namely a physical underlying network consisting of an IP structure 20 and a switching structure 14 and a virtual overlay network. The role of the physical underlying network is to provide an "IP structure" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0053] As further described below with respect to virtual routers 21A to 21X (illustrated and also collectively referred to herein as "v-routers 21"), the virtual routers running in server 12 can create a virtual overlay network on top of the physical underlying network using a dynamic mesh of "tunnels" between them. For example, these overlay tunnels can be MPLS tunnels over GRE / UDP or VXLAN tunnels, or NVGRE tunnels. The underlying physical routers and switches may not store any per-tenant state for the virtual machines or other virtual execution elements, such as any media access control (MAC) addresses, IP addresses, or policies. The forwarding tables of the underlying physical routers and switches may, for example, contain only the IP prefix or MAC address of the physical server 12. (The gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0054] The virtual router 21 of the server 12 typically contains per-tenant state. For example, a separate forwarding table (routing instance) may be contained for each virtual network. The forwarding table contains the IP prefixes (in the case of a layer 3 overlay) or MAC addresses (in the case of a layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., container pods). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that exist locally on the server 12 (i.e., there is at least one virtual execution element on the server 12).

[0055] The control plane protocol between the control plane nodes of the network controller 24 or physical gateway routers (or switches) can be BGP (and can be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3VPN and MPLS EVPN. For example, the protocol between the network controller 24 and the virtual router 21 can be based on XMPP.

[0056] "Container-based" or "operating system" virtualization refers to virtualizing an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by the open source Docker container application or CoreOS Rkt ("Rocket"). Like a virtual machine, each container is virtualized and can remain isolated from the host machine and other containers. However, unlike a virtual machine, each container can omit a separate operating system and instead provide a suite of applications and application-specific libraries. Typically, a container is executed by a host machine as an isolated userspace instance and can share an operating system and common libraries with other containers executing on the host machine. Therefore, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks.

[0057] In some examples, containers are managed by their host kernel to allow limiting and prioritizing resources (CPU, memory, block I / O, network, etc.) without the need to launch any virtual machines, in some cases using a namespace isolation feature that allows complete isolation of an application's (e.g., a given container's) view of the operating environment, including process trees, networks, user identifiers, and mounted file systems. In some examples, containers can be deployed in accordance with Linux Containers (LXC), an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

[0058] Server 12 hosts virtual network endpoints for one or more virtual networks that operate on a physical network represented herein by IP fabric 20 and switch fabric 14. Although primarily described with respect to a data center based switched network, other physical networks such as service provider network 7 may serve as the basis for one or more virtual networks.

[0059] Each server 12 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a set of one or more containers (e.g., pods), or another virtual execution element, such as a layer 3 endpoint of a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" may also include one or more pods of containers. A virtual execution element may represent an application workload. As Figure 1As shown, server 12A hosts a virtual network endpoint in the form of a pod (POD) 22 having one or more containers. However, given the hardware resource limitations of server 12, server 12 can execute as many virtual execution elements as possible. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0060] Each server 12 includes at least one network interface card (NIC) 13, each NIC including at least one interface for exchanging packets with a TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) described in the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to present one or more virtual network interfaces as "virtual functions" for use by various endpoints executed on the server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and the virtual function is an example of a virtual hardware component 21. As another example, one or more servers 12 can implement Virtio, which is a paravirtualization framework that can be used in, for example, a Linux operating system, which provides emulated NIC functions as a virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement an open v switch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) for hosted virtual machines, wherein such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some cases, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, the virtual hardware component is an SR-IOV virtual function. In some examples, any server in the server 12 may implement a Linux bridge that simulates a hardware bridge and forwards packets between the virtual network interfaces of the server or between the virtual network interfaces of the server and the physical network interfaces of the server. For a Docker implementation of a container hosted by a server, a Linux bridge or other operating system bridge executed on the server that exchanges packets between containers may be referred to as a "Docker bridge". The term "virtual router" used herein may include a Contrail or Tungsten Fabric virtual router, an open v switch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software located on a host device and performing switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, wherein the virtual network endpoints are hosted by one or more servers 12.

[0061] One or more servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) for an Internet Protocol virtual private network (IP-VPN). For example, in some examples, a packet received by the virtual router 21 of the server 12A from the underlying physical network structure of the data center 10 (i.e., the IP structure 20 and the switching structure 14) may include an outer header to allow the physical network structure to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the network interface card 13A of the server, but also a virtual network identifier, such as a VxLAN label or a multi-protocol label switching (MPLS) label that identifies a virtual network and a corresponding routing instance executed by the virtual router 21. The inner data packet includes an inner header having a destination network address that conforms to a virtual network addressing space of the virtual network identified by the virtual network identifier.

[0062] The virtual router 21 terminates the virtual network overlay tunnel, determines the virtual network of the received data packet based on the tunnel encapsulation header of the data packet, and forwards the data packet to the appropriate destination virtual network endpoint of the data packet. In the following, the virtual router is mainly described with respect to the virtual router 21A, but this description is relevant to other instances of the virtual router 21.

[0063] In some examples, for example, for each data packet outbound from a virtual network endpoint hosted by server 12A (e.g., any of pods 22A-22C), virtual router 21A appends a tunnel encapsulation header indicating the virtual network of the data packet to generate an encapsulated or "tunnel" data packet, and virtual router 21A outputs the encapsulated data packet to a physical destination computing device, e.g., another of servers 12, via an overlay tunnel of the virtual network. As used herein, virtual router 21A may perform operations of a tunnel endpoint to encapsulate an internal data packet initiated by a virtual network endpoint to generate a tunnel data packet, and decapsulate the tunnel data packet to obtain an internal data packet for routing to other virtual network endpoints.

[0064] In some examples, virtual router 21A may be kernel-based and executed as part of the kernel of the operating system of server 12A.

[0065] In some examples, virtual router 21A can be a virtual router that supports the Data Plane Development Kit (DPDK). In such an example, virtual router 21A uses DPDK as a data plane. In this mode, virtual router 21A runs as a user space application linked to a DPDK library (not shown). This is a performance version of a virtual router and is typically used by telecommunications companies, where VNFs are typically DPDK-based applications. The performance of virtual router 21A as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the polling mode driver (PMD) of the DPDK instead of the interrupt-based driver of the Linux kernel.

[0066] Server 12 includes and executes containerized routing protocol daemons 25A to 25X (collectively referred to as "cRPD 25"). A containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged into a container and can be run in a Linux-based environment. The cRPD can be executed in the user space of the host as a containerized process. Thus, the cRPD makes available a rich routing software pedigree of physical routers on Linux-based computing nodes (e.g., in some cases, server 12). The cRPD provides control plane functionality. The control plane is therefore containerized. For example, cRPD 25A implements the control plane of CNR 32A executed by server 12A.

[0067] Meanwhile, the virtual router 21 is a software entity that provides data plane functionality on the server 12. The cRPD 25A may use the forwarding (or data) plane provided by the Linux kernel of the server 12A for the kernel-based virtual router 21A. The cRPD 25A may alternatively use a DPDK-enabled or SmartNIC-executed instance of the virtual router 21. The virtual router 21A may work with an SDN controller (e.g., a network controller 24) to create an overlay network by exchanging routing, configuration, and other data. The virtual router 21A may be containerized. Thus, in some examples, in combination, the containerized cRPD and the containerized virtual router may be a fully functional containerized CNR 32A.

[0068] The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be an orchestration system. In some cases, the orchestration system is a container orchestration system that provides a container-centric infrastructure for automating the deployment, expansion, and operation of containers to provide a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provision, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements to host servers available to the orchestration platform. Specifically, container orchestration allows container orchestration and refers to the deployment, management, expansion, and configuration of containers to host servers via a container orchestration platform. Examples of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0069] Elements of the automation platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework, wherein a cluster master node of the cluster manages the deployment and operation of containers to one or more cluster child nodes of the cluster. The terms "master node" and "child node" used herein include different orchestration platform terms used to distinguish between the primary management element of the cluster and similar devices of the primary container hosting device of the cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "child node", while the Docker Swarm platform refers to the cluster manager and cluster nodes. A Kubernetes cluster is an example of an orchestration system cluster.

[0070] The orchestrator 23 and the network controller 24 may be executed on separate computing devices, on the same computing device. Each of the orchestrator 23 and the network controller 24 may be a distributed application executed on one or more computing devices. The orchestrator 23 and the network controller 24 may implement a corresponding master node for one or more clusters, each having one or more child nodes (also referred to as "computing nodes") implemented by a corresponding server 12.

[0071] In general, network controller 24 controls the network configuration of the data center 10 structure, for example, to establish one or more virtual networks for packetized communications between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator.

[0072] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides a computing infrastructure, which may include a container-centric computing infrastructure. The orchestrator 23 and, in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualized infrastructure for the container management platform.

[0073] Kubernetes operates with various Kubernetes objects - entities that represent the state of a Kubernetes cluster. Kubernetes objects can include any combination of names, namespaces, labels, annotations, field selectors, and recommended labels. For example, a Kubernetes cluster may contain one or more "namespace" objects. Each namespace of a Kubernetes cluster is isolated from other namespaces of the Kubernetes cluster. Namespace objects may include at least one of the organization, security, and performance of the Kubernetes cluster. As an example, a pod can be associated with a namespace, thereby associating the pod with a feature of the namespace (e.g., a virtual network). This feature enables multiple newly created pods to be organized by associating the pod with a set of common features. A namespace can be created according to namespace specification data that defines the namespace features, including a namespace name. In one example, a namespace can be named "namespace A", and each newly created pod can be associated with a set of features represented by "namespace A". In addition, Kubernetes includes a "default" namespace. If a newly created pod does not specify a namespace, the newly created pod can be associated with the features of the "default" namespace.

[0074] Namespaces enable a Kubernetes cluster to be used by multiple users, multiple teams of users, or one user using multiple applications. In addition, each user, team of users, or application can be isolated from every other user of the cluster within a namespace. Therefore, each user of a Kubernetes cluster in a namespace operates like a unique user of the Kubernetes cluster. Multiple virtual networks can be associated with a single namespace. In this way, pods belonging to a specific namespace have the ability to access every virtual network of the virtual networks associated with that namespace, including other pods that act as virtual network endpoints for that set of virtual networks.

[0075] In one example, pod 22A is a Kubernetes pod, which is an example of a virtual network endpoint. A pod is a group of one or more logically related containers ( Figure 1 pods 22A are instantiated for execution, shared storage for containers, and options for how to run containers. Pods may also be referred to as "pod copies" when instantiated for execution. Each container of a pod 22A is an example of a virtual execution element. The containers of a pod are always located on the same server, co-scheduled, and run in a shared context. The shared context of a pod can be a set of Linux namespaces, cgroups, and other isolation aspects. Within the context of a pod, separate applications may have further sub-isolations applied. Typically, containers in a pod have a common IP address and port space and are able to detect each other via the local host. Because of the shared context, containers in a pod also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different pods have different IP addresses and cannot communicate via IPC without a configuration that enables this feature. Containers that are members of different pods typically communicate with each other via the pod IP address.

[0076] Server 12A includes container platform 19A for running containerized applications, such as applications in pods 22. Container platform 19A receives a request from orchestrator 23 to obtain and host a container in server 12A. Container platform 19A obtains and executes the container.

[0077] A container network interface (CNI) 17A configures a virtual network interface for a virtual network endpoint. The orchestrator 23 and the container platform 19A use the CNI 17A to manage the network of pods, including the pod 22. For example, the CNI 17A creates a virtual network interface to connect the pod 22 to the virtual router 21A and enables containers of such a pod to communicate with other virtual network endpoints through the virtual network via the virtual network interface. For example, the CNI 17A may insert a virtual network interface of a virtual network into the network namespace of a container in the pod 22A and configure (or request configuration) a virtual network interface of a virtual network in the virtual router 21A so that the virtual router 21A is configured to send packets received from the virtual network via the virtual network interface to the containers of the pod 22A and to send packets received from the containers of the pod 22A on the virtual network via the virtual network interface. The CNI 17A may assign a network address (e.g., a virtual IP address of the virtual network) and may establish routes for the virtual network interface. In Kubernetes, by default, all pods can communicate with all other pods without using network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a pod virtual network shared by all namespaces, from which service and pod network addresses are allocated, respectively. In some cases, all pods in all namespaces generated in the Kubernetes cluster are able to communicate with each other, and the network addresses of all pods can be allocated from the pod subnet specified by the orchestrator 23. When a user creates an isolated namespace for a pod, the orchestrator 23 and the network controller 24 can create a new pod virtual network and a new shared service virtual network for the new isolated namespace. Pods in the isolated namespace generated in the Kubernetes cluster extract network addresses from the new pod virtual network, and the corresponding services of such pods extract network addresses from the new service virtual network.

[0078] CNI 17A may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17A may at least partially conform to the Container Network Interface (CNI) specification or the rkt networking proposal. CNI 17A may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNI. CNI 17A may also be referred to as a network plugin or CNI plugin or CNI instance. Separate CNIs may be called by, for example, Multus CNI to establish different virtual network interfaces for pod 202A.

[0079] CNI 17A can be called by the orchestrator 23. For the purposes of the CNI specification, a container can be considered a synonym for a Linux network namespace. What unit this corresponds to depends on the specific container runtime implementation: for example, in implementations of the application container specification (e.g., rkt), each pod runs in a unique network namespace. However, in Docker, a network namespace typically exists within each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and can communicate with each other. This can be an individual container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can be conceptually added to or removed from one or more networks. The CNI specification specifies some considerations for conformance plugins ("CNI plugins").

[0080] By adopting the Kubernetes CNI framework in some examples, CNR 32A can reduce the normal operational overhead incurred when using containerized devices instead of their physical counterparts. By exposing the appropriate device interfaces, CNR 32A can standardize the operating model of virtual devices to physical devices, eliminating barriers to adoption in an operator's network operating environment. CNR 32A (and more specifically the interface of cRPD 25A) can present a familiar look and feel of routing devices to any well-trained operations team. CNR has similar features and capabilities and a similar operating model to hardware-based platforms. Similarly, in some cases, the domain controller can communicate with and control CNR 32A using the protocols it uses with any other router, for example, Netconf / OpenConfig, gRPC, Path Computation Element Protocol (PCEP), or other interfaces.

[0081] CNR 32A is actually a virtualized router and can participate in IS-IS, Open Shortest Path First (OSPF), BGP, and / or other internal or external routing protocols and exchange routing protocol messages with other routers residing on other hosts, whether these routers are physical routers or CNR 32B-32X (CNR 32A-32X are collectively referred to as "CNR32" herein). In addition, MPLS can be used, typically based on segment routing (SR). There are two reasons for doing this: to allow traffic engineering if needed, and to support multi-tenancy by using VPNs, such as Layer 3 VPNs or EVPNs based on MPLS.

[0082] Each pod 22 includes one or more containers. In some examples, one or more pods 22 include a containerized DPDK workload designed to use DPDK to accelerate packet processing, for example, by exchanging data with other components using a DPDK library. In some examples, a virtual router 21 can be executed as a containerized DPDK workload.

[0083] Each pod 22 is configured with one or more virtual network interfaces (VNI) 26 for sending and receiving packets through the virtual router 21. The virtual network interface 26 can be the default interface of the pod 22. The pod 22 can implement the virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while the virtual router 21 can implement the virtual network interface 26 as a tap interface, a virtio user interface, or other type of interface. Although described with respect to pods, the VNI 26 can be attached to other virtual execution elements, such as virtual machines. The virtual network interface may also be referred to as a virtual machine interface (VMI), even when referring to the VNI of a pod / container.

[0084] The pod 22 and the virtual router 21 use a virtual network interface 26 to exchange data packets. The virtual network interface 26 may be a DPDK interface. The pod 22 and the virtual router 21 may use the v-host to establish the virtual network interface 26. The pod 22 may operate according to the aggregation model. The pod 22 may use a virtual device, such as a virtio device with a v-host user adapter, for user space container process communication of the virtual network interface 26. The following mainly describes the pod 22A, but may be applied to other pods 22 or other virtual execution elements.

[0085] CNI 17A can be combined with Figure 1 One or more other components are shown, configuring a virtual network interface 26 for the pod 22A. Any container of the pod 22A can utilize, i.e., share, the virtual network interface 26 of the pod 22A.

[0086] In some examples, cRPD 25A includes a default VRF 28 (shown as "D.VRF 28") and VRFs 29A to 29B (collectively referred to as "VRF 29"). Default VRF 28 stores a global routing table. cRPD 25A programs forwarding information derived from the routing table of VRF 29 into virtual router 21A as a forwarding table. In this way, virtual router 21A implements the VPN of VRF 29, which is shown as being included in virtual router 21A and cRPD 25A.

[0087] In some examples, cRPD 25A is configured to operate in host network mode, also known as local networking. cRPD 25A therefore uses the network namespace and IP address of its host (i.e., server 12A). cRPD 25A can see and access network interfaces 30A-30B of NIC 13A, which are inserted into default VRF 28 and are viewed by cRPD 25A as core-facing interfaces. Interfaces 30A-30B are connected to switch fabric 14 and can be Ethernet interfaces. Interface 30 is viewed and used by cRPD 25A as a core-facing interface for providing VPN because interface 30 can be used to transmit VPN service traffic through a Layer 3 network consisting of one or more of switch fabric 14, IP fabric 20, service provider network 7, or public network 15.

[0088] exist Figure 1 In the example server 12A of FIG. 1 , pod 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployments that specify a virtual network and that pod 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. Orchestrator 23 may receive configuration data from, for example, a user, an operator / administrator, or other machine system.

[0089] For example, as part of the process of creating pod 22A, orchestrator 23 requests network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). Pod 22A may have different virtual network interfaces for each virtual network to which it belongs. For example, virtual network interface 26 may be a virtual network interface for a particular virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks. Network controller 24 processes the request to generate interface configuration data for the virtual network interface of pod 22A. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0090] In some examples, a pod may have multiple interfaces to different cRPD VRFs 29, e.g., one for management traffic and another for data traffic. Figure 1 In the example, pod 22B can use VRF 29B for management traffic and VRF 29A for data traffic.

[0091] A pod 22 (e.g., pod 22A) may be connected to some physical interfaces, where the pod 22 learns the IP addresses of other devices, such as a user device of a mobile network gateway implemented by the pod or a customer network subnet of a CE router or gateway implemented by the pod. In some examples, to advertise these IP addresses into the network, the pod 22A treats the virtual network interface 26 with VRF 29 as an IP link and advertises the routes of these IP addresses to the cRPD 25A. The cRPD 25A may then advertise the reachability of these IP addresses through the cRPD 25A and the pod 22A as the next hop, which also complies with the VPN service model. The cRPD 25A programs the virtual router 21 with the corresponding forwarding information derived from the VRF 29 and the default VRF 28, and the virtual router 21 forwards the VPN traffic according to the VPN service model to implement the VPN.

[0092] CRPD 25A can apply many different types of overlay networks / VPNs, including L3VPN or EVPN (Type 2 / Type 5), using various underlying tunnel types including, for example, MPLS, SR-MPLS, SRv6, MPLSoUDP, MPLSoGRE, or IP-in-IP.

[0093] CNI 17A can be combined with Figure 1 One or more other components shown configure virtual network interfaces 26 for pods 22s. Any container of a pod 22 can utilize (i.e., share) any virtual network interface of the pod. Orchestrator 23 can store or manage virtual network interface configuration data for application deployment. For example, orchestrator 23 can receive a specification (a "pod specification" in the Kubernetes context) and network attachment definitions for a containerized application from a user, operator / administrator, or other machine system, and network controller 24 can derive configuration data from this information for configuring VRF 29 and default VRF 28; as well as configuring virtual network interfaces 26.

[0094] A conventional CNI plugin is invoked by a container platform / runtime to receive an Add command from the container platform to add a container to a single virtual network, and such a plugin may subsequently be invoked to receive a Del command from the container / runtime and remove the container from the virtual network. The term "invoke" may refer to instantiating a software component or module in memory into executable code for execution by processing circuitry.

[0095] exist Figure 1In the example computing infrastructure of , the network controller 24 is a cloud native distributed network controller for SDN, which is implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself can be implemented using one or more cloud native component microservices. Each control node 32 itself can be implemented using one or more cloud native component microservices.

[0096] The network controller 24 can be multi-tenant aware and support multi-tenancy of the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined network structures and advanced network features such as virtual networks, BGPaaS, network policies, service chains, and other telecommunications features. The network controller 24 can support network isolation using virtual network structures and support layer 3 networking.

[0097] In order to interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual router 21) a network policy, which is called a virtual network policy (VNP) and is interchangeably referred to as a virtual network router or a virtual network topology in this article. VNP defines the connection policy between virtual networks. A single network controller 24 can support multiple Kubernetes clusters, so VNP allows multiple virtual networks to be connected in a namespace, in a Kubernetes cluster, and across Kubernetes clusters. VNP can also be extended to support virtual network connections across multiple instances of the network controller 24.

[0098] The network controller 24 can use network policies to implement multiple layers of security. The default behavior of Kubernetes is to let pods communicate with each other. To apply network security policies, the SDN architecture implemented by the network controller 24 and the virtual router 21 can run as the CNI of Kubernetes through CNI 17A. For Layer 3, isolation occurs at the network level and virtual networks operate at Layer 3. Virtual networks are connected through policies. Kubernetes native network policies provide security at Layer 4. The SDN architecture may support Kubernetes network policies. Kubernetes network policies operate on Kubernetes namespace boundaries. The SDN architecture can add customized resources for enhanced network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta-tags to apply granular security policies in an extensible manner.) For Layer 4+, in some examples, the SDN architecture can support integration with containerized security devices and / or Istio, and can provide encryption support.

[0099] CNRs (e.g., CNR 32A) are described in more detail in U.S. Patent Publication 2022-0279420A1, entitled “Containerized Router with Virtual Network,” published on September 1, 2022, the entire contents of which are incorporated herein by reference.

[0100] According to the technology of the present disclosure, a containerized router operating in a public cloud environment can provide routing control plane functions for VPC workloads reachable via a VPC gateway. The containerized router can execute a routing protocol to learn the routes associated with the VPC workloads, and can program the VPC routing table in the VPC gateway device so that the VPC gateway device routes traffic to the containerized router for further routing. This traffic can include traffic from one or more containers (also known as pods). This arrangement may be beneficial because the pods and local routers may include a Border Gateway Protocol (BGP) speaker, but the VPC gateway may not support BGP and therefore cannot process BGP advertisements or routing information of BGP advertisements to determine how to route overlay traffic originating from or going to workloads in the VPC, such as BGP traffic.

[0101] According to the technology of the present disclosure, redundant containerized routers can be used in a public cloud environment. The redundant containerized routers can be connected to each other, for example, via a virtual router redundancy protocol (VRRP) session. A first containerized router among the redundant containerized routers can determine that the first containerized router is an active containerized router via the VRRP session. The first containerized router can be connected to a VPC gateway and program a VPC routing table in the VPC gateway.

[0102] Figure 2 is a block diagram of an example cloud computing environment including a CNR for programming a VPC routing table of a VPC gateway according to one or more aspects of the present disclosure. The cloud computing environment 100 may be Figure 1 The cloud computing environment 100 may be configured to provide VPC services to customers of the operator of the cloud computing environment 100. The VPC may operate as an extension of a local network (e.g., local network 130) of a customer of the operator of the cloud computing environment 100, or as a virtual network similar to a conventional (or physical) network of a customer of the operator of the cloud computing environment 100. The VPC may include one or more subnets and one or more gateways, each subnet having its own IP address range, and the gateway may be referred to as a VPC gateway herein. A gateway may be a device (physical or virtual) that connects the VPC to other networks. For example, an Internet gateway (e.g., VPC gateway 116) may connect the VPC to the Internet 140.

[0103] Cloud computing environment 100 may include CNR 150, VPC gateways 110A to 110N (collectively, VPC gateways 110) that may provide access to clusters 102A to 102N (collectively, clusters 102). VPC gateways 110 may be Figure 1 Example of VPC Gateway 7. Cluster 102A may include multiple worker (compute) nodes 104A to 104N, which may include Kubernetes worker nodes. Similarly, cluster 102N may include multiple worker nodes 106A to 106N. Each of worker nodes 104A to 104N and worker nodes 106A to 106N may include a BGP speaker (e.g., BGPS1 to BGPSN) and one or more pods, such as pods P1 and P2. In some examples, P1 and P2 may not have BGP enabled. As Figure 2 As shown, pod P1 represents a pod of a first service and pod P2 represents a pod of a second service. Pods P1 and P2 may be part of the same subnet or different subnets. Each service may have a unique IP address, which may be a virtual IP or a service IP. The IP address of pod P1 may be extracted from the service subnet of the first service. A given worker node may include pods for one or more services. Cluster 102 may represent a cluster of a VPC, which may be implemented as an extension of another network (e.g., local network 130).

[0104] BGP speakers BGPS1 through BGPSN, which may be pods (e.g., pods of cluster 102A), may know or be aware of the IP addresses associated with pods P1 and P2. In some examples, BGP speakers BGPS1 through BGPSN may be part of the CNI that assigns IP addresses to pods P1 and P2, and therefore will know the IP addresses of pods P1 and P2. In some examples, BGP speakers BGPS1 through BGPSN may learn the IP addresses, for example, through interaction with Kubernetes elements, and may advertise such IP addresses via a data protocol such as BGP. However, as described in further detail below, VPC gateway 110 may be unable to process routing protocol (e.g., BGP) messages from BGP speakers BGPS1 through BGPSN, and therefore may be unable to learn routes to pods P1 or P2.

[0105] Cloud computing environment 100 may also include VPC gateway 112, VPC gateway 114, and VPC gateway 116. VPC gateway 112 may provide access to cloud native network functions (CNFs) 120. CNFs 120 may be software services that provide network functions to cloud computing environment 100. In some examples, CNFs 120 may be implemented by pods or containers. VPC gateway 114 may provide access to local network 130, and VPC gateway 116 may provide access to Internet 140. Although in Figure 2 Although not shown, cloud computing environment 100 may include additional devices (which may be physical or virtual) that may be used to access Figure 2 100, such as additional local networks. The VPC gateway 116 can be used as an Internet gateway to provide access to the Internet 140 to the various clusters 102 of the cloud computing environment 100. The VPC gateway 114 can be used as a gateway for the local network 130, providing access between the local network 130 and the cluster 102 of the VPC of the local network 130. In some examples, each of the cluster 102, VPC 110, and CNR 150 can act as a layer 3 router so that no tunnel is required between these devices. For example, CNR 150 can use MAC learning to direct traffic from, for example, the local network 130 to the cluster 102.

[0106] Although cloud computing environment 100 may have a VPC including VPC gateway 110, such VPC gateway may provide limited functionality due to limitations of the underlying hardware platform (e.g., in terms of scale, functionality, etc.) or software implementation (e.g., in terms of equal-cost multipath routing (ECMP), coverage, signaling, protocols, etc.), scale (e.g., in terms of a limited number of routes, tables / instances, number of protocol sessions such as BGP, etc.) of workloads deployed on these environments. In some examples, VPC gateway 110 may not operate using conventional routing protocols and / or may need to use custom APIs and non-portable protocols. For example, devices of cloud environment 100 may not be able to talk to VPC gateway 110 using any standard protocols. VPC gateway 110 may not know the IP address of local network 130. As such, a BGP speaker (e.g., BGPS1 of workload 104A) may not be able to advertise workload 104A or pods P1 and P2 of workload 104A to VPC gateway 110A.

[0107] To address this issue, CNR 150 may use an API to program VPC routing table 118 of VPC gateway 110. Such programming may not introduce a new tunnel between VPC 110 and CNR 150, but rather instructs VPC gateway 110 how to forward packets (e.g., to workloads 104 and 106) using the same IP header information used by CNR 150. In some aspects, CNR 150 may view VPC gateway 110 as a forwarding plane—instructing VPC gateway 110 how to forward packets from different routes learned by CNR 150 via, for example, routing protocol messages advertised by BGPS1 to BGPSN of cluster 102. In these aspects, CNR 150 may be viewed as a control plane for VPC gateway 110 because CNR 150 may program VPC routing table 118 of VPC 110. When programming VPC routing table 118, CNR 150 may use cRPD 25A ( Figure 1 ) policy engine and apply different routing policies to each VPC routing table.

[0108] For example, CNR 150 can use routing policies as filters to affect the routing of data packets. The routing policies can be based on source addresses and destination addresses. In this way, for example, in the case where local network 130 includes multiple tenants, data packets can be routed within an appropriate VPN (e.g., VPC).

[0109] In some examples, CNR 150 abstracts and hides all VPC network details from the hosted service, while also making the solution portable across different cloud environments and providing easy migration to an on-premises environment. CNR 150 can provide this by using a network overlay or by interacting directly with cloud network elements.

[0110] For example, CNR 150 can interact with the cloud environment bottom layer, learn the bottom layer network topology, and then distribute routes into the bottom layer to provide seamless connectivity between applications deployed in the cloud environment and a local or another cloud environment (e.g., a hybrid cloud).

[0111] For example, CNR 150 can dynamically establish (BGP) routing adjacencies on worker nodes or pods and distribute routing information to the cloud environment underlay and / or advertise to remote BGP peers. These prefixes are not (automatically) learned by VPC gateway 110, but in some cases, these prefixes are required to achieve, for example, traffic forwarding from VPC gateway 110 to, for example, local network 130.

[0112] When deploying an IP router in a cloud environment with a VPC, routing may not be provided for most cases because such cloud environments have their own layer 3-based network (underlying) VPC infrastructure that prevents the use of protocols such as IS-IS, OSPF, etc. to establish routing adjacencies due to lack of support for multicast and broadcast traffic. The present disclosure describes a CNR that provides routing functionality in a cloud environment with one or more VPCs. Although shown separately, in some examples, CNR 150 can be implemented in a cluster of clusters 102.

[0113] exist Figure 2 In the example of FIG. 1 , CNR 150 performs a query on VPC routing table 118 (in Figure 1 In the example of FIG. 1 , a standard router (shown as VPCRTs 118A to 118N) is programmed so that VPC Gateway 110 forwards traffic to CNR 150, whereas a standard router would simply program its own forwarding information base (FIB) to forward incoming traffic out of itself. Since CNR 150 can learn the routes to, for example, local network 130, CNR 150 can route traffic destined for local network 130 from VPC Gateway 110 to local network 130 after receiving the traffic from VPC Gateway 110.

[0114] CNR 150 may program a VPC routing table 118 of VPC 110 using an API of VPC 110. CNR 150 may program multiple VPC tables 118, each with a corresponding (e.g., different) set of routes and next hops. In some examples, CNR 150 may map cRPD routes and interfaces to VPC routing tables 118 and interfaces based on operator-defined labels. In some examples, an operator may define routing tables, next hops, route selection criteria, etc. through routing policies that may be applied by CNR 150. Administrators may also statically program VPC routing tables, but this approach is not scalable and does not take advantage of the high scalability of VPC for customers.

[0115] For example, a VPC routing table (e.g., VPC routing table 118A) may include an indication that for a particular IP address, the next hop IP address should be zero or a worker node (e.g., worker node 104A). cRPD 25A may use an interface ID, such as a virtual network interface on a VPC gateway and an interface name on CNR 150.

[0116] VPC 110 may know the IP addresses of worker nodes 104 and 106, but may not know the IP addresses assigned to application pods (e.g., P1 and P2) within worker nodes 104 and 106. For example, VPC 110 may not know how to transmit packets from pods P1 and P2 to local network 130, or vice versa.

[0117] An administrator may determine different routing policies that CNR 150 uses to program each VPC routing table 118. For example, policies may be used to determine which routes are programmed into a VPC routing table. In this way, policies may act as filters so that a given VPC routing table (e.g., VPC routing table 118A) may include a subset of available routes.

[0118] In some examples, there may be overlap in destination IP addresses between different intended destinations. In this case, using policy-based routing, CNR 150 can use both the source IP address and the destination IP address to classify packets into different VPNs to route packets from all pods sharing the same interface to the same VPC, such that packets destined for local network 130 are sent to VPC 114 instead of a VPC associated with another local network.

[0119] In some examples, the BGP speaker can be a CNI that assigns pod IP addresses, so the BGP speaker (e.g., BGPS1) knows the IP addresses of the relevant pods (P1 and P2 of worker node 104A). In some examples, the BGP speaker can be a plugin.

[0120] Some wireless service providers may want to migrate part of their core network to the public cloud when expanding their existing infrastructure to accommodate 5G. For example, the cloud-native core functions of 5G (e.g., unified data management (UDM), network slice selection function (NSSF), and network repository function (NRF)) can be containerized and migrated to the public cloud. Because wireless service providers can provide slice functions and provide slices based on factors such as functionality, performance, user equipment (UE) groups, etc., these CNFs should be dynamically elastically scaled up or down in the public cloud environment using load balancer services. Service providers can also leverage the inherent redundancy capabilities provided by the public cloud to distribute their functions in different geographical regions in an active-standby mode. Service providers want seamless connectivity, low latency, and high throughput from their local data centers to the core functions running in the cloud. The main requirement for service providers with geographically distributed networks may be seamless switchover to different geographical regions in the event of natural disasters / failures. Conventional public cloud services may not support the network services required for such use cases. Therefore, it may be desirable to provide a cloud-native software router to address these requirements.

[0121] The CNR 150 can leverage DPDK v-Routing as its data plane to accelerate packets processed per second (PPS) to deliver the required packet performance in cloud environments.

[0122] CNR 150 can be used to extend a 5G core infrastructure running locally (e.g., local network 130) to, for example, VPC 110A. In the 5G example, different services (e.g., NSSF, NRF, and UDM, each with their own service IP addresses) can run in an EKS cluster (e.g., cluster 102A), each running in their own VRF (e.g., SIGVRF, OAM VRF), and CNR 150 can, for example, provide seamless communication with VRF counterparts (e.g., CU) running in local network 130 in complete isolation. For example, cluster 102A can include a node group with a maximum and minimum size that is automatically scaled to 2. 5G CNF pods and CNR 150 can run in different clusters. For example, a 5G CNF pod can run in cluster 102A, while CNR 150 can run in another cluster of clusters 102A to 102N. The service IP address may be a cluster endpoint that can only be accessed within the VPC. The cluster network can be provided by a cloud service provider VPC container network interface (CNI) plugin. The CNI plugin allows Kubernetes pods to have the same IP address as the Kubernetes pods on the VPC network. All containers within the pods that share the network namespace can communicate with each other using ports.

[0123] To attract traffic from these 5G CNFs and local CNFs, CNR 150 can program internal and external VPC routing tables via the VPC API. For example, CNR 150 can include a complete BGP stack through which CNR 150 communicates with 5G CNFs, transit gateways (TGWs) (such as Figure 3 100) and any local edge gateway peers running in the cloud computing environment 100 to exchange routes. The 5G CNF may have a BGP capability to advertise the associated service virtual IP address to CNR 150. In some examples, CNR 150 includes a BGP auto-configuration function, where CNR 150 automatically establishes a BGP session when a new CNF workload is orchestrated.

[0124] As described above, the routes in VPC routing table 118 are programmed by CNRs (e.g., CNR 150) so that VPC gateway 110 forwards traffic to the CNRs. In a deployment with a single CNR 150, a failure of CNR 150 will interrupt traffic to / from services hosted on worker nodes in the cluster. Traffic disruption may be temporary or short-lived. Running two instances of CNRs in active / standby mode can significantly reduce traffic disruption by shortening recovery time.

[0125] Figure 3 2 is a block diagram of an example cloud computing environment with redundant CNRs according to one or more aspects of the present disclosure. Cloud computing environment 200 may be similar to cloud computing environment 100, but includes redundant CNRs (CNR 250A and CNR 250B), which may be Figure 2 Example of CNR 150. Figure 3 In the example of FIG. 1 , the cloud computing environment 200 is shown coupled to a local network 330, which may be Figure 2 The local network 330 may be an example of a local network 130. The local network 330 may be a wireless wide area network, such as a 5G network.

[0126] CNR 250A and CNR 250B may include virtual router redundancy protocol (VRRP) components 214A and 214B, respectively, which may be used to establish a VRRP session 201 between CNR 250A and CNR 250B. In some examples, one or more of the VRRP components 214A or 214B may be implemented as a pod, for example, pod P1 or P2. In order to determine which of CNR 250A and CNR 250B should be the active node and which should be the standby node, CNR 250A and CNR 250B may run a VRRP session 201. VRRP is a protocol that may be used to provide redundancy in a network. The VRRP session 201 may select one of CNR 250A or CNR 250B as the active node (sometimes referred to as the primary node) and select the other of CNR 250A or CNR 250B as the standby node (sometimes referred to as the backup node). The active CNR is responsible for operating as a CNR, similar to Figure 2 If a problem occurs with the active CNR or the link to the active CNR that prevents the active CNR from functioning as expected, the standby CNR can still assume the function of the active CNR. In some examples, VRRP session 201 may include a unicast VRRP session. Private cloud environments with VPCs may not support L2-based VRRP because these private cloud environments may not support IP broadcast and / or multicast traffic. As such, VRRP session 201 may be a unicast VRRP session.

[0127] In some examples, CNR 250A and CNR 250B can share a virtual IP address on an interface facing worker node 204 (e.g., an interface with VPC 210). In some examples, the shared virtual IP address will only be used by the active nodes of CNR 250A and CNR 250B. The active nodes can forward traffic to / from worker nodes 204, while the standby nodes cannot. In addition, the active nodes can update the VPC routing table 218, while the standby nodes cannot. In this way, the VPC engaged with CNR 250A and CNR 250B (e.g., VPC 210) can only forward traffic to the active nodes.

[0128] The activated CNR node may have an L2 MAC address that is different from the L2 MAC address of the standby CNR node. The activated CNR node may advertise the L2 MAC address of the activated CNR node, while the standby CNR node may not advertise the L2 MAC address of the standby CNR node. For example, if CNR 250A is the active node, CNR 250A may advertise the L2 MAC address of CNR 250A, while the standby node CNR 250B may not advertise the L2 MAC address of CNR 250B.

[0129] For example, CNR 250A may continue to operate as an active node while CNR 250B operates as a standby node until a failure of CNR 250A is detected. When a failure of CNR 250A is detected via VRRP 201, CNR 250B (the standby node) will become the active node. In this case, in order to continue to attract traffic to the active node (which has now changed from CNR 250A to CNR 250B), CNR 250B may learn routes from neighboring devices, and the routing agent application of CNR 250B may update VPC routing table 218 to update the forwarding next hop to CNR 250B (instead of CNR 250A), and the routing agent application of CNR 250A may stop updating VPC routing table 218. In this way, traffic from worker 204 may be forwarded by VPC 210 to CNR 250B, the new active node. For example, in an L2-based segment, CNR 250B may use gratuitous ARP to update the ARP entry corresponding to the virtual IP address so that traffic is now forwarded to CNR 250B, the newly activated VRRP master.

[0130] CNR 250B may begin advertising the L2 MAC address of CNR 250B, and CNR 250A may stop advertising the L2 MAC address of CNR 250A. In this manner, traffic from, for example, TGW 220 may be directed to CNR 250B instead of CNR 250A.

[0131] In some examples, VRRP control state may be used to modify BGP routing attributes, such as local-pref (I-BGP) or multi-exit discriminator (MED) (e-BGP), so that a remote, local device (virtual or physical) (e.g., a BGP peer) will prefer CNR 250B (a VRRP-activated node) over CNR 250A as an egress PE.

[0132] The upstream router (e.g., TGW 220) may have a virtual IP address that may be used by the activated CNR as a forwarding next-hop IP address. In some examples, both CNR 250A and CNR 250B (active and standby nodes) advertise BGP prefixes to local network 330 to establish BGP sessions with BGP peers in local network 330. In some examples, the BGP speakers (BGPS1 to BGPSN) of worker node 204 advertise BGP prefixes to CNR 250A and CNR 250B.

[0133] In some examples, if the virtual IP address is in the same subnet as the worker nodes, the worker nodes 204 may use the virtual IP address shared by CNR 250A and CNR 250B as a default route. This approach will result in fewer updates to the VPC routing table 218, but will require the activated VRRP node to obtain the virtual IP address from the VPC infrastructure. In such an example, the virtual IP address should be known to all worker nodes.

[0134] exist Figure 3 In the example of , worker node 204 is part of cluster 302, which can host 5G core network functions (CNFs) for service providers associated with local network 330, such as NSSF, UDM, etc. Each of these CNFs can have its own virtual IP address. In some examples, these services can operate in different virtual routing and forwarding functions (VRFs) and include traffic isolation between these services. For example, there may be a virtual IP address for operations, administration and maintenance (OAM) VRF (10.10.10.10 / 32) and another virtual IP address for secure internet gateway (SIG) VRF (10.10.11.10 / 32). Each worker of worker node 204 can have a BGP speaker from BGPS1 to BGPSN, which can be used to peer with the activated CNRs of CNRs 250A and 250B and advertise the virtual IP address.

[0135] In some examples, CNRs 250A and 250B may be in a cluster ( Figure 3), for example, cluster 102N ( Figure 2 ). For example, the auto-scaling size of such a cluster can be 2. One of CNR 250A or 250B can be in an active mode, while the other of CNR 250A or 250B can be in a standby mode, providing redundancy for the functionality of the activated CNR. In some examples, CNR 250A and CNR 250B can both be located in different geographic locations (e.g., cloud computing environment 200 is a geographically dispersed cloud computing environment) to provide geographic redundancy. CNR 250A and 250B can peer with TGW 220 to reach a local network 330, which can include a telecommunications data center 322, in which a core network, a network function (CN NF 226), a centralized unit (CU) 228, and / or a distributed unit (DU) 234 can be located. It should be understood that the local network 330 may also include a cellular radio / antenna 332.

[0136] exist Figure 3 In the example of , an activated CNR can support multiple interfaces (e.g., up to 8) and can route traffic between these interfaces. For example, an Ethernet interface in CNR 250A and / or CNR 250B can be used to send traffic such as http, https, etc. to worker nodes 204 of all virtual IP addresses.

[0137] In some examples, CNR 250A and CNR 250B may include BGP auto-configuration so that when a new BGP speaker (e.g., BGPSN) starts, no new configuration needs to be added. In some examples, CNR 250A and CNR 250B may include bidirectional forwarding detection (BFD), which can be used to detect sub-second failures of 5G workloads on worker node 204 and reroute traffic. In some examples, CNR 250A and CNR 250B can utilize VRF-lite. For example, a virtual routing domain with route leakage can be configured to route traffic across different VRFs. For example, prefixes in SIG VRF and OAM VRF may be leaked in an Ethernet interface. In some examples, CNR 250A and CNR 250B can support equal-cost multipath (ECMP) routing. For example, a new worker (e.g., worker node 104N) can be added or deleted instantly without affecting existing flows. This can be referred to as flow affinity or flow stickiness. In some examples, CNR 250A and CNR 250B can be configured to interact with VPC gateway 210 via APIs (e.g., APIs 216A and 216B) that are programmed to populate external and internal routing tables (e.g., routing table 218) based on BGP advertisements (e.g., from BGP speakers BGPS1 to BGPSN) to attract traffic to the activated one of CNR 250A or CNR 250B.

[0138] In some examples, CNR 250A and CNR 250B can provide asymmetric routing. For optimization and cost savings, there may be some flows from the activated CNR to the worker node 204, but the reverse flow from the worker node 204 can be routed directly from the VPC gateway 210. In some examples, in order to support active-standby switching between CNR 250A and CNR 250B, CNR 250A and CNR 250B can use one or more protocols, such as VRRP. In some examples, because most public clouds do not support multicast or broadcast, CNR 250A and CNR 250B can use unicast VRRP. In some examples, CNR 250A and CNR 250B can support tunnels, such as MPLS tunnels over UDP (MPLSoUDP). For example, CNR250A and CNR 250B can use MPLSoUDP encapsulation to tunnel traffic to the local network 330. In some examples, CNR 250A and CNR 250B may provide control plane features, such as EVPN-VXLAN.

[0139] Figure 4 is a block diagram of an example computing device (eg, host) in accordance with techniques described in this disclosure. Figure 4 The computing device 300 may represent a physical or virtual server and may represent Figure 1 344 . In some examples, the bus 342 may couple the storage device 344, the microprocessor 320, and the NIC 330. The bus 342 may represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller may control DMA transfers between components coupled to the bus 342. In some examples, components coupled to the bus 342 control DMA transfers between components coupled to the bus 342.

[0140] The microprocessor 320 may include one or more processors, each processor including an independent execution unit to execute instructions that conform to an instruction set architecture, the instructions being stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined in one or more multi-core processors (or "many-core" processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0141] Disk 346 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information, such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD) or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 320.

[0142] The main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM), such as DDR2 / DDR3 SDRAM, or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to transmit or store the required program code and program data in the form of instructions or data structures and can be accessed by the computer. The main memory 344 provides a physical address space consisting of addressable storage locations.

[0143] The network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange data packets using the links of the underlying physical network. The interface 332 may include a port interface card having one or more network ports. The NIC 330 may also include, for example, on-card memory for storing data packets. Direct memory access transmissions between the NIC 330 and other devices coupled to the bus 342 may read / write from / to the NIC memory.

[0144] Memory 344, NIC 330, storage disk 346 and microprocessor 320 can provide an operating environment for a software stack, which includes an operating system kernel 380 executed in kernel space. Kernel 380 can represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel or Windows server operating system kernel available from Microsoft. In some examples, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can include a virtual machine manager (VMM). The operating system including kernel 380 provides an execution environment for one or more processes in user space 345.

[0145] Kernel 380 includes a physical driver 325 that uses a network interface card 330. Network interface card 330 may also implement SR-IOV to enable sharing of physical network functions (I / O) between one or more virtual execution elements, such as containers 329A and 329B or one or more virtual machines ( Figure 2 330). A shared virtual device (e.g., a virtual function) may provide dedicated resources so that each virtual execution element may access dedicated resources of the NIC 330, and thus, the NIC 330 appears as a dedicated NIC to each virtual execution element. A virtual function may represent a lightweight PCIe function that shares physical resources with a physical function used by the physical driver 325 and other virtual functions. For a NIC 330 that supports SR-IOV, the NIC 330 may have thousands of virtual functions available according to the SR-IOV standard, but for I / O intensive applications, the number of configured virtual functions is typically much smaller.

[0146] Computing device 300 may be coupled to a physical network switch fabric that includes an overlay network that extends the switch fabric from physical switches to software or "virtual" routers, including virtual router 306A, that are physical servers coupled to the switch fabric. A virtual router may be a physical server (e.g., Figure 1 A process or thread or component thereof executed by a server 12) that dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints, which can include VPCs. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to separate the virtual address of the endpoint from the physical address (e.g., IP address) of the server on which the endpoint executes. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and between virtual networks on a physical network. The term "virtual router" as used herein may include an Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software that resides on a host device and performs switching, bridging, or routing of packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. In Figure 4 In the example computing device 300 of , the virtual router 306A executes as a virtual router in user space, but in various implementations, the virtual router 306A may execute within a hypervisor, a host operating system, a host application, or a virtual machine.

[0147] Virtual router 306A can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used for Kubernetes deployments of pods 302. Virtual router 306A can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 306A can perform network services such as applying security policies, NAT, multicast, mirroring, and load balancing. In some examples, virtual router 306A can perform traffic to / from one or more VPCs (e.g., Figure 2 Routing of packets to any VPC 110).

[0148] Virtual router 306A may be executed as a kernel module or a user space DPDK process (virtual router 306A is shown here in user space 345). Virtual router agent 314 may also be executed in user space. Figure 4In the example computing device 300 of , the virtual router 306A is executed in user space as a DPDK-based virtual router, but in various implementations, the virtual router 306A can be executed within a hypervisor, a host operating system, a host application, or a virtual machine. In some examples, the virtual router agent 314 connects to the network controller 24 using a channel, which is used to download configuration and forwarding information. The virtual router agent 314 programs the forwarding state into the virtual router data (or "forwarding") plane represented by the virtual router 306A. The virtual router 306A and the virtual router agent 314 can be processes.

[0149] The virtual router 306A can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used for Kubernetes deployments of pods 302. The virtual router 306A can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 306A can perform network services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0150] Virtual router 306A can be multithreaded and executed on one or more processor cores. Virtual router 306A can include multiple queues. Virtual router 306A can implement a packet processing pipeline. Depending on the operation to be applied to the packet, virtual router agent 314 can stitch the pipeline from the simplest to the most complex. Virtual router 306A can maintain multiple instances of the forwarding basis. Virtual router 300 can use RCU (read copy update) locks to access and update tables.

[0151] To send packets to other computing nodes or switches, virtual router 306A uses one or more physical interfaces 332. Typically, virtual router 306A communicates with workloads 104 or 106 (eg, Figure 2 ) to exchange packets. In some examples, virtual router 306A has multiple virtual network interfaces (e.g., vifs). These interfaces may include a kernel interface vhost0 for exchanging packets with the host operating system; an interface with the virtual router agent 314 (pkt0) for obtaining forwarding state from the network controller and sending exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 332. In some examples, virtual router 306A shares a virtual interface with another virtual router facing one or more VPCs 110 (facing workloads 104 or 106). In this case, virtual router 306A and another virtual router can be in an active / standby configuration, for example, Figure 3 Configuration of the CNR 250A and CNR 250B.

[0152] In a kernel-based deployment of virtual router 306A (not shown), virtual router 306A is installed as a kernel module in the operating system. Virtual router 306A records itself to the TCP / IP stack to receive packets from any desired operating system interface it wants. The interface can be binding, physical, tap (for VM), veth (for container), etc. The virtual router 306A in this mode sends and receives packets from different interfaces according to the operating system. For example, the operating system can display a tap interface supported by a vhost-net driver to communicate with the VM. Once virtual router 306A records the packet from the tap interface, the TCP / IP stack sends all packets to it. Virtual router 306A sends packets via the operating system interface. In addition, NIC queues (physical or virtual) are handled by the operating system. Packet processing may run in interrupt mode, which will generate interrupts and may cause frequent context switching. When the packet rate is high, the overhead of frequent interrupts and context switching may overwhelm the operating system, resulting in performance degradation.

[0153] In a DPDK-based deployment of virtual router 306A, virtual router 306A is installed as a user space 345 application linked to the DPDK library. This can result in faster performance than a kernel-based deployment, particularly in the presence of high packet rates. The DPDK's polling mode driver (PMD) uses the physical interface 332 instead of the kernel's interrupt-based driver. The registers of the physical interface 332 can be exposed in user space 345 so that the PMD can access them; the physical interface 332 bound in this way is no longer managed or visible to the host operating system, and the DPDK-based virtual router 306A manages the physical interface 332. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the virtual router 306A DPDK data plane. When the packet rate is high, this "polling mode" nature makes the virtual router 306A DPDK data plane packet processing / forwarding more efficient than interrupt mode. Compared to kernel mode virtual router 306A, there are relatively fewer interrupts and context switches during packet I / O, and in some cases interrupts and context switches during packet I / O can be avoided entirely.

[0154] In general, each pod 302A-202B can be assigned one or more virtual network addresses for use in a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by virtual router 306A. Pod 302B can be assigned its own virtual layer 3 (L3) IP address, e.g., for sending and receiving communications, but may not be aware of the IP address of the computing device 300 executing pod 302B. The virtual network address can therefore be different from the logical address of the underlying physical computer system (e.g., computing device 300).

[0155] The computing device 300 includes a virtual router agent 314 that controls the virtual network overlay of the computing device 300 and orchestrates the routing of packets within the computing device 300. Typically, the virtual router agent 314 communicates with a network controller 24 for a virtualized infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 300, and more specifically, the virtual router 306A and the virtual network interface 212. By configuring the virtual router 306A based on information received from the network controller 24, the virtual router agent 314 can support configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0156] As described above, the network controller 24 can provide a logical centralized controller to facilitate the operation of one or more virtual networks. The network controller 24 can, for example, maintain a routing information base, for example, one or more routing tables storing routing information of a physical network and one or more overlay networks. The virtual router 306A implements one or more virtual routing and forwarding instances (VRFs), for example, VRF 322A, for the corresponding virtual network for which the virtual router 306A operates as a corresponding tunnel endpoint. Typically, each VRF 322 stores forwarding information for the corresponding virtual network or VPC, and identifies where the data packet will be forwarded and whether the data packet will be encapsulated in a tunnel protocol, for example, using a tunnel header, which may include one or more headers of different layers of the virtual network protocol stack. Each VRF 322 may include a network forwarding table storing routing and forwarding information for the virtual network. For example, the virtual router 306A may learn routes from the cRPD 324 via a virtual router agent 314 that may include an API, and use the routes learned in this way to build a forwarding table for each VRF 322.

[0157] cRPD 324, acting as a control plane for virtual router 306A, can learn BGP routes for packets flowing through computing device 300. In this way, cRPD 324 can program the routing table of virtual router 306 (e.g., via virtual router agent 314). VPC table programmer 326 can communicate with cRPD 324 to learn BGP routes for packets flowing through computing device 300. VPC table programmer 326 can be a table programming agent configured to program a VPC routing table (e.g., VPC routing table 118) of VPC gateway 110. Although shown as part of pod 302B, in some examples, cRPD 324 and VPC table programmer can be part of different pods rather than part of the same pod. For example, VPC table programmer 326 can be executed in a separate pod from pod 302. In some examples, VPC table programmer 326 can be configured to utilize an API, such as an API of a VPC gateway, to interface with the VPC gateway when programming the VPC routing table.

[0158] The VPC routing table 118 can change from situation to situation. For example, if a VPC is restarted, the associated VPC routing table may change. In some examples, the VPC table programmer 326 can abstract the VPC routing table from a configuration perspective. For example, the VPC table programmer 326 can enable administrators to interact with the VPC routing table using tags (which may be more descriptive), e.g., "vpc-table-tag": "cnr-aws-vpc-internal-table", rather than names, e.g., "RouteTableId": "rtb-08fdf3cc822baa22c". The VPC table programmer 326 can perform any transformations, such as applying appropriate policies to the appropriate VPC routing table, or updating the appropriate VPC routing table.

[0159] An example mapping that VPC table programmer 326 may perform for VPC gateway 118 is as follows: [

[0161] {

[0162] "cnr-table-name":"inet.0",

[0163] "cnr-policy-name":"inet-aws-export",

[0164] "cnr-nexthop-interface-name":"eth3",

[0165] "vpc-table-tag": "cnr-aws-vpc-internal-table"

[0166] },

[0167] {

[0168] "cnr-table-name": "oam.inet.0",

[0169] "cnr-policy-name": "oam-aws-export",

[0170] "cnr-nexthop-interface-name": "eth5",

[0171] "vpc-table-tag": "cnr-aws-vpc-external-table-oam"

[0172] },

[0173] {

[0174] "cnr-table-name": "sig.inet.0",

[0175] "cnr-policy-name": "sig-aws-export",

[0176] "cnr-nexthop-interface-name": "eth4",

[0177] "vpc-table-tag": "cnr-aws-vpc-external-table-sig"

[0178] },

[0180] The example policy statement is as follows:

[0181] policy-statement inet-aws-export{

[0182] term 1{

[0183] from{

[0184] protocol bgp;

[0185] as-path[as-path-infra-cluster aws-onsite-cluster];

[0186] ​}

[0187] then accept;

[0188] }

[0189] then reject;

[0190] }

[0191] An example of programming a routing table to local network 130 is as follows:

[0192] "RouteTables":[

[0193] {

[0194] "RouteTableId":"rtb-08fdf3cc822baa22c",

[0195] "Routes":[

[0196] {

[0197] "DestinationCidrBlock":"10.10.10.4 / 32",

[0198] "InstanceId":"i-04d12991242918ed8",

[0199] "InstanceOwnerId":"956845549706",

[0200] "NetworkInterfaceId":"eni-0b91bb9d332b6529e",

[0201] "Origin":"CreateRoute",

[0202] "State":"active"

[0203] },

[0204] {

[0205] "DestinationCidrBlock":"10.10.10.17 / 32",

[0206] "InstanceId":"i-04d12991242918ed8",

[0207] "InstanceOwnerId":"956845549706",

[0208] "NetworkInterfaceId":"eni-0b91bb9d332b6529e",

[0209] "Origin":"CreateRoute",

[0210] "State":"active"

[0211] }

[0212] ],

[0213] "Tags":[

[0214] {

[0215] "Key":"cnr-aws-simulated-vpc-table-sig",

[0216] "Value":""

[0217] }

[0218] ],

[0219] "VpcId":"vpc-09f812bf56bdee948",

[0220] "OwnerId":"956845549706"

[0221] } ]

[0223] For traffic flowing to workloads 104 and 106, VPC table programmer 326 may program the L2D MAC to identify the appropriate node.

[0224] NIC 330 may receive a packet with a BGP header. Virtual router 306A processes the packet to determine from the BGP header which BGP speaker the packet is to be routed to. For example, a first packet from local network 130 may be destined for BGPS2 of workload 104B, or a second packet from BGPS1 of workload 106B may be destined for Internet 140 ( Figure 2 ). Virtual router 306A may route the first received packet to VPC 110A and the second received packet to VPC 116.

[0225] Pod 302A includes one or more application containers 329A. Pod 302B includes an instance of cRPD 324. Container platform 304 includes container runtime 308, orchestration agent 310, service broker 311, and CNI 312.

[0226] The container engine 308 includes code that can be executed by the microprocessor 310. The container runtime 308 can be one or more computer processes. The container engine 308 runs containerized applications in the form of containers 329A to 329B. The container engine 308 can represent Dockert, rkt, or other container engines for managing containers. Typically, the container engine 308 receives, requests, and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 310, the container engine 308 can obtain images and instantiate them into executable containers in pods 302A to 302B.

[0227] The service agent 311 includes code that can be executed by the microprocessor 320. The service agent 311 can be one or more computer processes. The service agent 311 monitors the addition and removal of services and endpoint objects, and maintains the network configuration of the computing device 300 to ensure communication between pods and containers, for example, using services. The service agent 311 can also manage iptables to capture traffic to the virtual IP address and port of the service and redirect the traffic to the proxy port of the pod supported by the agent. The service agent 311 can represent a kube agent of a subsidiary node of a Kubernetes cluster. In some examples, the container platform 204 does not include the service agent 311, or disables the service agent 311, to facilitate the CNI 312 to configure the virtual router 306A and the pod 302.

[0228] Orchestration agent 310 includes code that can be executed by microprocessor 320. Orchestration agent 310 can be one or more computer processes. Orchestration agent 310 can represent a kubelet of a node attached to a Kubernetes cluster. Orchestration agent 310 is an orchestrator (e.g., Figure 1 The orchestration agent 310 is an agent of the orchestrator 23 of the orchestration engine 23 that receives container specification data of the container and ensures that the container is executed by the computing device 300. The container specification data can be in the form of a manifest file sent from the orchestrator 23 to the orchestration agent 310, or is received indirectly via a command line interface, an HTTP endpoint, or an HTTP server. The container specification data can be a pod specification (e.g., a pod space (PodSpec) - a YAML (yet another markup language) or JSON object that describes the pod) of a pod 302 of the container 329. Based on the container specification data, the orchestration agent 310 instructs the container engine 308 to obtain and instantiate a container image of the container 329 for the computing device 300 to execute the container 329.

[0229] Orchestration agent 310 instantiates or calls CNI 312 to configure one or more virtual network interfaces for each pod 302. For example, orchestration agent 310 receives container specification data for pod 302A and directs container engine 308 to create pod 302A with container 329A based on the container specification data for pod 302A. Orchestration agent 310 also calls CNI 312 to configure a virtual network interface for pod 302A corresponding to a virtual network of VRF 322A. In this example, pod 302A is a virtual network endpoint for a virtual network corresponding to VRF 322A.

[0230] CNI 312 may obtain interface configuration data for configuring a virtual network interface of pod 302. Virtual router agent 314 may operate as a virtual network control plane module for enabling network controller 24 to configure virtual router 306A. Unlike an orchestration control plane (including container platform 304 for satellite nodes and master nodes, e.g., orchestrator 23) that manages provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 and virtual router agent 314 for satellite nodes) manages the configuration of a virtual network implemented in part in the data plane by virtual router 306A of the satellite node. Virtual router agent 314 transmits interface configuration data of a virtual network interface to CNI 312 to enable the orchestration control plane element (i.e., CNI 312) to configure the virtual network interface according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Additionally, this may enable CNI 312 to obtain interface configuration data for multiple virtual network interfaces of a pod and configure the multiple virtual network interfaces, which may reduce the communication and resource overhead inherent in calling a separate CNI 312 to configure each virtual network interface.

[0231] Figure 5 is a flow diagram illustrating an example CNR VPC technique according to one or more aspects of the present disclosure. A VPC table programmer 326 of a containerized router (e.g., CNR 150) may be connected to a VPC gateway 110A (400). The VPC table programmer 326 may include a process (e.g., an application) to a VPC gateway configured to program a virtual private cloud (VPC) routing table (e.g., VPTRT 118A) in a VPC gateway (e.g., VPC 110A) based on routing information received by a containerized routing protocol process (e.g., cPRD 25A) via a routing protocol. For example, for programming purposes, CNR 150 may connect to VPC gateway 110A using API 116A. CNR 150 may program VPC routing table 118A in VPC gateway 110A (402).

[0232] In some examples, CNR 150 may program VPC routing table 118A so that VPC gateway 110A directs network traffic from VPC gateway 110A to CNR 150. For example, CNR 150 may program VPC routing table 118A to indicate that the forwarding next hop is CNR 150. For example, the forwarding next hop may be a virtual IP address associated with CNR 150, a MAC address associated with CNR 150, an interface address associated with CNR 150, etc.

[0233] In some examples, CNR 150 is configured to receive network traffic from VPC gateway 110A and forward the network traffic to local 130. In some examples, the routing protocol includes BGP.

[0234] In some examples, VPC gateway 110A is coupled to one or more Border Gateway Protocol (BGP) virtual appliances (e.g., BGP speakers BGPS1 through BGPSN) via one or more corresponding links. In some examples, the one or more BGP virtual appliances include one or more Kubernetes pods, each pod being associated with a corresponding Kubernetes worker (e.g., workers 104 and / or 106). In some examples, each corresponding Kubernetes worker includes at least one service pod (e.g., P1 and / or P2) for services other than the BGP protocol. In some examples, the one or more corresponding links include one or more corresponding Ethernet links.

[0235] In some examples, VPC gateway 110A cannot learn routes directly from BGP speakers BGPS1 to BGPSN. In some examples, VPC gateway 110A includes a physical device or a virtual device. In some examples, VPC gateway 110A includes a first VPC gateway, VPC routing table 118A includes a first VPC routing table, and CNR 150 is further configured to program a second VPC routing table (e.g., VPC routing table 118N) in a second VPC gateway (e.g., VPC gateway 110N), wherein the second VPC routing table is different from the first VPC routing table. In some examples, as part of programming VPC routing table 118A in VPC gateway 110A, CNR 150 is configured to apply a policy to the routes.

[0236] Figure 62 is a flow diagram illustrating an example redundant CNR VPC technique according to one or more aspects of the present disclosure. CNR 250B may connect to CNR 250A via VRRP session 302 (500). For example, for redundancy purposes, CNR 250A and CNR 250B may establish VRRP session 302. CNR 250B may determine via the VRRP session that CNR 250B is the active CNR among CNR 250A and CNR 250B (502). For example, based on the VRRP session, CNR 250B may become the active CNR, for example, through an election or through CNR 250A becoming the deactivated active CNR. CNR 250B may connect to VPC gateway 210 via API 216A (504). For example, for programmatic purposes, CNR 250B may connect to VPC gateway 210 using API 216A. It should be noted that CNR 250B may connect to VPC gateway 210 via API 216A before, during, or after CNR 250B becomes an activated CNR. In some examples, based on CNR 250B being an activated CNR, CNR 250B may connect to VPC gateway 210. Based on being an activated CNR, CNR 250B may program (506) VPC routing table 218 in VPC gateway 210. In some examples, VRRP session 302 includes a unicast session.

[0237] In some examples, CNR 250B includes VRRP component 214B. For example, VRRP component 214B may be implemented in a processing circuit (e.g., microprocessor 320 ( Figure 4 )). In some examples, one of CNR 250A or CNR 250B is configured as an active computing device, and the other of CNR 250A or CNR 250B is configured as a standby computing device. In some examples, CNR 250B is configured to Figure 1 ) shares a virtual IP address with CNR 250A on the interface of one or more worker nodes 204.

[0238] In some examples, CNR 250B is configured as a standby computing device, wherein CNR 250B is further configured to determine a failure of CNR 250A via a VRRP session, and in response to the determination of the failure, assume the role of an activated computing device and update VPC routing table 218 of VPC gateway 210. In some examples, as part of updating VPC routing table 218, CNR 250B is configured to update VPC routing table 218 to direct network traffic from VPC gateway 210 to CNR 250B, wherein prior to updating VPC routing table 218, VPC routing table 218 directed network traffic from VPC gateway 210 to CNR 250A. In some examples, as part of updating VPC routing table 218, CNR 250B is configured to update VPC routing table 218 to indicate that the forwarding next hop is CNR 250B.

[0239] In some examples, CNR 250B can be configured to send a gratuitous Address Resolution Protocol (ARP) message to VPC gateway 210 as part of updating VPC routing table 218. In some examples, CNR 250B is further configured to modify at least one of a BGP Local Preference attribute or a BGP MED attribute in response to assuming the role of an activated computing device to direct traffic from a local router to a route advertised by CNR 250B.

[0240] The technology described herein can be implemented with hardware, software, firmware, or any combination thereof. The various components, functional units, and / or modules shown in the drawings and / or shown or described elsewhere in this disclosure can perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware that resides and / or is executed in one or more computing devices. For example, a computing device can execute one or more such modules with multiple processors or multiple devices. A computing device can execute one or more such modules as a virtual machine executed on underlying hardware. One or more such modules can be executed as one or more services of an operating system or computing platform. One or more such modules can be executed as one or more executable programs at the application layer of a computing platform. In other examples, the functions provided by the modules can be implemented by dedicated hardware devices. Although some modules, data storage devices, components, programs, executable programs, data items, functional units, and / or other items contained in one or more storage devices can be shown separately, one or more of these items can be combined and operated as a single module, component, program, executable program, data item, or functional unit. For example, one or more modules or data storage devices can be combined or partially combined so that they operate or provide functions as a single module. In addition, one or more modules can be operated in conjunction with each other so that, for example, one module serves as a service or extension of another module. In addition, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device may include multiple components, subcomponents, modules, submodules, data storage and / or other components or modules or data storage not shown. In addition, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device can be implemented in various ways. For example, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device can be implemented as a part of an operating system executed on a computing device.

[0241] If implemented in hardware, the present disclosure may be directed to an apparatus, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions that are executed by a processor.

[0242] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0243] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may mean that the storage medium is not contained in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0244] The code or instructions may be software and / or firmware executed by a processing circuit that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the present disclosure may be provided in software modules or hardware modules.

Claims

1. A computing device configured with a containerized router, the computing device comprising: Processing circuit; a containerized virtual router configured to execute on the processing circuitry and configured to implement a data plane for the containerized router; a containerized routing protocol process configured to execute on the processing circuit and configured to implement a control plane for the containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol; as well as A process is configured to program a virtual private cloud routing table in a virtual private cloud gateway based on routing information received by the containerized routing protocol process via the routing protocol.

2. The computing device of claim 1, wherein: As part of programming the virtual private cloud routing table in the virtual private cloud gateway, the process configured to program the virtual private cloud routing table is configured to program the virtual private cloud routing table to direct all network traffic from the virtual private cloud gateway to the computing device.

3. The computing device of claim 2, wherein: As part of programming the virtual private cloud routing table, the process configured to program the virtual private cloud routing table is configured to program the virtual private cloud routing table to indicate that a forwarding next hop is the computing device.

4. The computing device according to any one of claims 1 to 3, wherein: The containerized virtual router is configured to receive network traffic from the virtual private cloud gateway and forward the network traffic to a local network.

5. The computing device according to any one of claims 1 to 3, wherein: The routing protocol includes the Border Gateway Protocol.

6. The computing device according to any one of claims 1 to 3, wherein: The process configured to program the virtual private cloud routing table is configured to program the virtual private cloud routing table via an API of the virtual private cloud gateway.

7. The computing device according to any one of claims 1 to 3, wherein: The virtual private cloud gateway is coupled to one or more Border Gateway Protocol virtual devices via one or more corresponding links.

8. The computing device of claim 7, wherein: The one or more Border Gateway Protocol virtual devices include one or more Kubernetes pods, each pod being associated with a corresponding Kubernetes worker.

9. The computing device of claim 8, wherein: Each respective Kubernetes worker includes at least one service pod associated with a service outside of the Border Gateway Protocol.

10. The computing device of claim 7, wherein: The one or more corresponding links include one or more corresponding Ethernet links.

11. The computing device according to any one of claims 1 to 3, wherein: The virtual private cloud gateway cannot learn routes directly from the border gateway protocol.

12. The computing device according to any one of claims 1 to 3, wherein: The virtual private cloud gateway includes a physical device or a virtual device.

13. The computing device according to any one of claims 1 to 3, wherein: The virtual private cloud gateway includes a first virtual private cloud gateway, the virtual private cloud routing table includes a first virtual private cloud routing table, and wherein the process is configured to program the first virtual private cloud routing table and is further configured to program a second virtual private cloud routing table in a second virtual private cloud gateway, wherein the second virtual private cloud routing table is different from the first virtual private cloud routing table.

14. The computing device according to any one of claims 1 to 3, wherein: As part of programming the virtual private cloud routing table in the virtual private cloud gateway, the containerized routing protocol process is configured to apply a policy to routing.

15. A method for a computer network, comprising: connecting to the virtual private cloud gateway by a process, the process being configured to program a virtual private cloud routing table in the virtual private cloud gateway based on routing information received by the containerized routing protocol process via the routing protocol; as well as The virtual private cloud routing table in the virtual private cloud gateway is programmed by the process configured to program the virtual private cloud routing table.

16. The method according to claim 15, wherein: Programming the virtual private cloud routing table in the virtual private cloud gateway includes programming the virtual private cloud routing table to direct all network traffic from the virtual private cloud gateway to a containerized virtual router of a computing device.

17. The method according to claim 16, wherein: Programming the virtual private cloud routing table includes programming the virtual private cloud routing table to indicate that a forwarding next hop is the computing device.

18. The method according to any one of claims 15 to 17, further comprising: receiving, by a containerized virtual router, network traffic from the virtual private cloud gateway based on the virtual private cloud routing table; as well as The network traffic is forwarded by the containerized virtual router toward a local network.

19. The method according to any one of claims 15 to 17, wherein: The routing protocol includes the Border Gateway Protocol.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 15 to 19 or to be configured as a computing device according to any one of claims 1 to 14.

Images