Redundant containerized virtual routers for use with virtual private clouds

Abstract

The invention relates to a redundant containerized virtual router for use with a virtual private cloud. The computing device includes a first containerized router and processing circuitry. In one embodiment, a computing device includes a containerized virtual router configured to execute on a processing circuit and the containerized virtual router configured to implement a data plane of a first containerized router. In one embodiment, a computing device includes a containerized routing protocol process configured to execute on a processing circuit and the containerized routing protocol process configured to implement a control plane of a first containerized router, where the containerized routing protocol process is configured to execute a routing protocol. The first containerized router is configured to program a virtual private cloud (VPC) routing table in the VPC gateway based on routing information received via a routing protocol through a containerized routing protocol process. The first containerized router is communicatively coupled to the second containerized router via a virtual router redundancy protocol (VRRP) session.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 18 / 345,737, filed on June 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to virtualized computing infrastructure, and more particularly, to a containerized router for use in a cloud computing environment. Background Art

[0004] In a typical cloud data center environment, a large number of interconnected servers provide computing and / or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide user support equipment located in various physical hosting facilities for infrastructures spread across the globe.

[0005] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. In particular, modern data centers have widely utilized virtualized environments, wherein virtual hosts (also referred to herein as virtual execution elements, such as virtual machines or containers) are deployed and executed on an underlying computing platform of a physical computing device.

[0006] Virtualization in a data center or any environment containing one or more servers can provide multiple advantages. One advantage is that virtualization can significantly improve efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. The second advantage is that virtualization provides effective control over the computing infrastructure. As physical computing resources become fungible resources (such as in cloud-based computing environments), the provisioning and management of computing infrastructure becomes easier. Therefore, in addition to the efficiency and higher return on investment (ROI) provided by virtualization, enterprise IT personnel often prefer virtualized computing clusters in data centers because of their management advantages.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable application execution elements that are isolated from each other and from the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and executed as a single lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers provide a guarantee of consistent operation from one computing environment to another (virtual or physical).

[0008] Due to the inherent lightweight nature of containers, a single host can typically support many more container instances than traditional virtual machines (VMs). Containers are typically short-lived, can be created and moved more efficiently than VMs, and can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms (e.g., Kubernetes). These container characteristics affect the requirements of container networking solutions: the network should be agile and scalable. VMs, containers, and bare metal servers may need to coexist in the same computing environment, with communication between different application deployments. Container networks should also be agnostic to the multiple orchestration platforms used to deploy containerized applications.

[0009] Managing the deployment and infrastructure of computing infrastructure for application execution can involve two main roles: (1) orchestration - used to automate the deployment, scaling, and operation of applications across a cluster of hosts and to provide computing infrastructure, which can include container-centric computing infrastructure; and (2) network management - used to create virtual networks in the network infrastructure to enable packetized communication between applications running on virtual execution environments (such as containers or VMs) and between applications running on traditional (e.g., physical) environments. Software-defined networking facilitates network management.

[0010] A cloud data center environment (such as a virtualized data center) can provide services to any number of customers. For example, a cloud data center environment can provide services to different customers via a virtual private cloud (VPC). A VPC can be viewed as an extension of a customer-operated on-prem network, which can be located on the premises of the customer site. Therefore, by using a VPC, a customer can quickly and efficiently expand its internal network as needed by increasing or decreasing resources residing in a cloud data center through the VPC (rather than changing the resources of the internal network itself). VPC implementations may include the use of a virtual private cloud gateway device. Summary of the invention

[0011] A VPC is a virtual network of computing nodes that can be similar to a traditional network that an enterprise can operate in an internal (on-prem) data center or an extension of such a traditional network. A VPC can be a service provided by a service provider in a cloud computing infrastructure. Since a cloud computing infrastructure can include many resources that can be allocated to various customers, customers can quickly and efficiently expand their own network resources by using a VPC, avoiding the need to physically expand an internal data center. A VPC may include one or more subnets. A subnet may include a range of IP addresses for the VPC. A VPC gateway may connect the VPC to a network outside the VPN. A VPC can provide an internal data center with a connection to a virtual computing instance (e.g., a pod, a virtual machine, and / or other types of deployable virtual computing instances) running in a cloud computing environment, thereby expanding the computing power of the internal data center without physically adding additional resources to the internal data center.

[0012] A VPC gateway is a physical or virtual device that a cloud service provider can use to provide customers with access to a VPC. In some examples, a VPC gateway may not provide a mechanism to interact with the underlying network using standard networking protocols. For example, a VPC gateway may not be able to read or learn from BGP prefixes. In other words, a VPC gateway may not be a BGP speaker. Therefore, the VPC gateway itself may not be able to route or exchange BGP traffic to or from BGP-speaking virtual computing instances in a data center environment. Therefore, it may be necessary to provide seamless connectivity between customer workloads in a VPC and on-premises infrastructure.

[0013] The containerized routing protocol daemon (cRPD) is a routing protocol process packaged and deployed as a container. The cRPD can be executed in the user space of the host as a containerized process. Therefore, the cRPD makes the rich routing software pedigree of the physical routers on the compute nodes available. The cRPD provides control plane functionality for containerized routers. Existing implementations of cRPD (running on the host) use forwarding provided by the kernel. The control plane is therefore containerized. In some examples, the compute nodes and kernels can be Linux-based.

[0014] A virtual router is a software entity that provides data plane functionality on a compute node, while cRPD can provide control plane functionality. Compute nodes can host virtual compute instances that are centrally orchestrated and provisioned. In some cases, the virtual router can further interact with the SDN controller to create an overlay network by exchanging routing, configuration, and other data. For example, a virtual router can run as a kernel module or a DPDK-based process. DPDK allows a virtual router to process more packets per second than when running as a kernel module. The virtual router data plane can be containerized. Therefore, in combination, a containerized cRPD and a containerized virtual router can be a fully functional containerized router.

[0015] In general, the present disclosure describes techniques for providing routing control plane functionality for VPC workloads reachable via a VPC gateway using a containerized router operating in a public cloud environment. The containerized router can execute a routing protocol to learn routes associated with the VPC workloads, and can program a VPC routing table in a VPC gateway device so that the VPC gateway device routes traffic to the containerized router for further routing. Such traffic can include traffic from one or more containers (also referred to as pods). This arrangement can be beneficial because the pods and internal routers may include a Border Gateway Protocol (BGP) speaker, but the VPC gateway may not support BGP and therefore cannot process BGP advertisements or BGP-advertised routing information, such as BGP traffic, to determine how to route overlay traffic originating from or going to workloads in the VPC.

[0016] The present disclosure also describes techniques for utilizing redundant containerized routers in a public cloud environment. The redundant containerized routers may be connected to each other, for example, via a virtual router redundancy protocol (VRRP) session. A first containerized router of the redundant containerized routers may determine, via the VRRP session, that the first containerized router is an active containerized router. The first containerized router may be connected to a VPC gateway and program a VPC routing table in the VPC gateway.

[0017] In one example, a computing device includes a containerized router, the computing device comprising: processing circuitry; a containerized virtual router configured to execute on the processing circuitry and configured to implement a data plane of the containerized router; a containerized routing protocol process configured to execute on the processing circuitry and configured to implement a control plane of the containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol; and a process configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by the containerized routing protocol process via the routing protocol.

[0018] In one example, a method includes: connecting to a virtual private cloud (VPC) gateway through a process that programs a VPC routing table in the VPC gateway based on routing information received by a containerized routing protocol process via a routing protocol; and programming a VPC routing table in the VPC gateway through a process configured to program the VPC routing table. In one example, a non-transitory computer-readable storage medium stores instructions that, when executed, cause a processing circuit

[0019] In another example, a non-transitory computer-readable medium stores instructions that, when executed, cause a processing circuit to connect to a virtual private cloud (VPC) gateway by programming a VPC routing table in the VPC gateway based on routing information received by a containerized routing protocol process via a routing protocol; and program a VPC routing table in the VPC gateway by a process configured to program the VPC routing table.

[0020] In another example, a computing device includes a first containerized router, the computing device including: processing circuitry; a containerized virtual router configured to execute on the processing circuitry, and the containerized virtual router is configured to implement a data plane for the containerized router; and a containerized routing protocol process configured to execute on the processing circuitry, and the containerized routing protocol process is configured to implement a control plane for the first containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol, wherein the first containerized router is configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by the containerized routing protocol process via the routing protocol, and wherein the first containerized router is communicatively coupled to a second containerized router via a virtual router redundancy protocol (VRRP) session.

[0021] In another example, a method includes: connecting, by a first containerized router, to a second containerized router via a virtual router redundancy protocol (VRRP) session; determining, via the VRRP session, that the first containerized router is an active containerized router among the first containerized router and the second containerized router; connecting to a virtual private cloud (VPC) gateway; and programming, by a process configured to program the VPC routing table and based on the first containerized router being the active containerized router, a VPC routing table in the VPC gateway.

[0022] In another example, a non-transitory computer-readable medium stores instructions that, when executed, cause a processing circuit to connect to a second containerized router via a virtual router redundancy protocol (VRRP) session; determine, via the VRRP session, that a first containerized router is an active containerized router among the first containerized router and the second containerized router; connect to a virtual private cloud (VPC) gateway; and program a VPC routing table in the VPC gateway based on the first containerized router being the active containerized router.

[0023] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a block diagram illustrating an example computing infrastructure in accordance with one or more aspects of the present disclosure.

[0025] Figure 2 is a block diagram of an example cloud computing environment including a CNR that programs a VPC routing table for a VPC gateway according to one or more aspects of the present disclosure.

[0026] Figure 3 is a block diagram of an example cloud computing environment with redundant CNRs according to one or more aspects of the present disclosure.

[0027] Figure 4 is a block diagram of an example computing device (eg, host) according to techniques described in this disclosure.

[0028] Figure 5 is a flow chart illustrating an example CNR VPC technique according to one or more aspects of the present disclosure.

[0029] Figure 6 is a flow diagram illustrating an example redundant CNR VPC technique in accordance with one or more aspects of the present disclosure.

[0030] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0031] A cloud native approach can include functional blocks that are decomposed into microservices. Microservices are deployed as containers on computing nodes and orchestrated by an orchestration system such as Kubernetes (abbreviated as "K8s"). Kubernetes deploys containers using pods. K8s networking between pods is implemented via a plug-in called the container networking interface (CNI) (also known as the container network interface plug-in). However, the networking capabilities of a typical CNI are quite basic, and when the containerized network functions of the CNI service play a key role in a network (such as a telecommunications network), the network capabilities of a typical CNI are not suitable. As described in this article, CNR is more suitable for these situations. CNR is a containerized router that allows computing nodes (such as x86-based or ARM-based hosts) to become first-level members of the network routing system, participate in protocols such as Intermediate System to Intermediate System (IS-IS) and Border Gateway Protocol (BGP), and provide transport and multi-tenancy based on Multi-Protocol Label Switching / Segment Routing (MPLS / SR). In other words, the platform is not an adjunct to a network (e.g., a customer edge (CE) router) and may operate as a provider edge (PE) router or other device within a provider network (such as in a cloud computing environment).

[0032] CNR has one or more advantages over traditional routers. A router has a control plane and a forwarding plane. The control plane participates in dynamic routing protocols and exchanges routing information with other routers in the network. The results are downloaded to the forwarding plane in the form of prefixes, next hops, and associated SR / MPLS labels. The implementation described in this article is modular, in the sense that the control plane does not need to know the exact details of how the forwarding plane is implemented. In a hardware router, the forwarding plane can be based on a custom ASIC. In contrast, CNR is a virtualized router. However, in both cases, the functionality of the routing protocol software is similar. This means that CNR benefits from the same highly comprehensive and powerful protocol implementation as the hardware-based routers that power some of the world's largest networks.

[0033] CNR uses a containerized routing protocol daemon (cRPD) control plane and a virtual router (vRouter) forwarding plane to deliver high-performance networking in a small package size that is functionally similar to a non-virtualized router, a physical network function (PNF). The forwarding plane can be implemented via a choice of DPDK, Linux kernel, or Smart-NIC. A K8s CNI compatible package is fully integrated and can be deployed within a K8s environment (e.g., supporting Multus).

[0034] Since CNR itself is a cloud-native application, it supports installation using K8s manifests or Helm Charts. These include the initial configuration of the router, including routing protocols and Layer 3 VPNs that support slicing. CNR can be orchestrated and configured in seconds, with all routing protocol adjacencies established and running with the rest of the network. Continuous configuration changes during the lifecycle of CNR (for example, adding or removing network slices) can be implemented via the choice of CLI, K8s manifests, NetConf, or Terraform.

[0035] By adopting the K8s CNI framework, CNR can reduce the traditional operational overhead incurred when using containerized devices instead of physical devices. By exposing the appropriate device interfaces, CNR can standardize the operating model of virtual devices to physical devices, eliminating barriers to adoption in operators' network operation environments. CNR can present a familiar routing device look and feel to any well-trained operations team. CNR has similar features and capabilities and a similar operating model to hardware-based platforms. Similarly, domain controllers can communicate with and control CNR using the protocols they use with any other Junos router, such as Netconf / OpenConfig, gRPC, Path Computation Element Protocol (PCEP), and Programmable Routing Daemon (pRPD) API.

[0036] Nodes executing CNR can participate in IS-IS, Open Shortest Path First (OSPF), BGP, and / or other internal or external routing protocols. In addition, MPLS can be used, typically based on Segment Routing (SR). There are two reasons for this: to allow traffic engineering if needed, and to support multi-tenancy by using MPLS-based Layer 3 VPNs. As an alternative, SRv6 can be used to meet these requirements. Having comprehensive routing capabilities is also necessary to implement network slicing. Each slice tenant is placed in its own Layer 3 VPN. From the perspective of the Layer 3 VPN, the CNR acts as a provider edge (PE) router. Therefore, the CNR exchanges Layer 3 VPN prefixes with other PE routers in the network via BGP, regardless of whether these other PEs are physical routers or CNRs residing on other hosts. Each tenant can be placed in a separate VRF table on each PE, providing an appropriate degree of isolation and security between tenants, just like traditional Layer 3 VPN services. This neatly solves the problem that K8s does not provide this isolation natively. Layer 3 VPN is a tried and tested method to achieve multi-tenancy in networking, trusted by many large companies around the world who purchase this service from their network service providers.

[0037] A VPC gateway, which may be deployed and used by a cloud service provider to provide customers with access to a VPC, may not be able to use traditional routing protocols (such as BGP) to process routing protocol messages to obtain routing information (such as advertised routes and prefixes). For example, the VPC gateway may not be a BGP speaker. Therefore, the VPC gateway may not itself be able to process BGP routes, and therefore, may not be able to perform exchange or routing of BGP messages or other traffic to or from a BGP-speaking pod in a data center environment based on the routes advertised by BGP. Because, in contrast, CNR is able to learn BGP routes and is able to exchange and route BGP and other traffic based on prefixes advertised by BGP, it may be desirable to utilize CNR to program the VPC routing table of the VPC gateway so that the VPC gateway can forward traffic to the CNR via the underlying network for further routing. In this way, the CNR directly programs the underlying elements of the cloud infrastructure to forward traffic to the CNR itself for further routing.

[0038] Figure 1 is a block diagram illustrating an example computing infrastructure according to one or more aspects of the present disclosure. Computing infrastructure 8 may represent a software defined networking (SDN) architecture system for virtual networks. Example use cases for SDN architectures include 5G mobile networks and cloud and enterprise cloud native use cases. The SDN architecture may include data plane elements implemented in computing nodes (e.g., servers 12) and network devices (such as routers or switches), and the SDN architecture may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. Reference Figure 1 The described SDN architecture including the network controller 24 is merely an example. The techniques of the present disclosure may be applied to other data center architectures and SDN controller deployments and implementations.

[0039] In this example, the SDN architecture components are microservices, and the SDN architecture uses a basic container orchestration platform to manage the lifecycle of the SDN architecture components. The SDN architecture can utilize Kubernetes structures such as Multus, authentication and authorization (Authentication & Authorization), cluster (Cluster) API, KubeFederation, KubeVirt, and Kata containers. The SDN architecture can support data plane development toolkit (DPDK) pods, and the SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies. The SDN architecture can be integrated with orchestration platforms such as Kubernetes, OpenShift, Mesos, OpenStack, VMware vSphere (e.g., orchestrator 23), and can also be integrated with service provider operations support systems / business support systems (OSS / BSS).

[0040] Typically, one or more data centers 10 provide an operating environment for applications and services at a customer site 11 (illustrated as "Customer 11"), which has one or more customer networks coupled to the data center through a service provider (such as, through a VPC) network 7. In some examples, the customer site 11 can be an internal (on-site) customer network. For example, each data center 10 can host infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 15, which can represent one or more networks managed by other providers and can therefore form part of a large-scale public network infrastructure (e.g., the Internet). The public network 15 can represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0041] Although customer sites 11 and public networks 15 are primarily illustrated and described as edge networks of service provider network 7, in some examples, one or more of customer sites 11 and public networks 15 may be tenant networks within any of data centers 10. For example, data center 10 may host multiple tenants (customers), each tenant being associated with one or more virtual private networks (VPNs), each of which may implement one customer site 11.

[0042] The service provider network 7 provides packet-based connectivity to connected customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement multi-protocol label switching (MPLS) forwarding, and in this case may be referred to as an MPLS network or MPLS backbone. In some cases, the service provider network 7 represents multiple interconnected autonomous systems (such as the Internet) that provide services from one or more service providers.

[0043] In some examples, each of data centers 10 may represent one of many geographically distributed network data centers, which may be connected to each other via service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1 As shown in the example of , data center 10 may include facilities for providing network services to customers. Customers of a service provider may be collective entities, such as businesses, governments, or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although illustrated as an independent edge network of a service provider network 7, elements of data center 10 (such as one or more physical network functions (PNFs) or virtualized network functions (VNFs)) may be included within the kernel of the service provider network 7.

[0044] In this example, data center 10 includes storage and / or computing servers (or "nodes") interconnected via a switch fabric 14 provided by one or more layers of physical network switches and routers, with servers 12A-12X (herein "servers 12") being depicted as coupled to top-of-rack switches 16A-16N. Servers 12 are computing devices and may also be referred to herein as "computing nodes," "hosts," or "host devices." Although in Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10 may include many additional servers coupled to other TOR switches 16 in data center 10.

[0045] The data center can provide a virtual private cloud (VPC) to any customer 11. The data center 10 may include one or more VPC gateways (such as VPC gateway 7), which can be used to connect the VPC to an external network (e.g., internal, other clouds, external VPCs) to provide VPC services to customers 11. The VPC gateway 7 can be a physical device or a virtual device, such as a physical or virtual gateway, a router, a switch, etc. In some examples, the VPC gateway 7 can be implemented by one or more servers 12. In some examples, the CNR can be directly coupled to the VPC gateway. This document will refer to Figure 2 and Figure 3 Discuss VPC Gateway further.

[0046] The switch fabric 14 in the illustrated example includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") that are coupled to a distribution layer of chassis (or "spine" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices (such as cellular phones or personal digital assistants), wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as, for example, physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0047] In this example, the TOR switch 16 and the chassis switch 18 provide redundant (multi-homed) connections to the IP structure 20 and the service provider network 7 for the server 12. The chassis switch 18 aggregates traffic flows and provides connections between the TOR switches 16. The TOR switch 16 can be a network device that provides layer 2 (MAC) and / or layer 3 (e.g., IP) routing and / or switching functions. The TOR switch 16 and the chassis switch 18 can each include one or more processors and memories, and can execute one or more software processes. The chassis switch 18 is coupled to the IP structure 20, and the IP structure can perform layer 3 routing to route network traffic between the data center 10 and the customer site 11 through the service provider network 7. The switching architecture of the data center 10 is only an example. For example, other switching architectures can have more or fewer switching layers. The IP structure 20 can include one or more gateway routers.

[0048] The term "packet flow", "traffic flow", or simply "flow" refers to a group of packets originating from a specific source device or endpoint and sent to a specific destination device or endpoint. A single packet flow can be identified by a 5-tuple: for example, <source network address, destination network address, source port, destination port, protocol>. This 5-tuple generally identifies the packet flow to which the received packets correspond. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for the packet.

[0049] The servers 12 may each represent a computing, server, or storage server. For example, each server 12 may represent a computing device (such as an x86 processor-based server) that is configured to operate according to the techniques described herein. The servers 12 may provide a network function virtualization infrastructure (NFVI) for the NFV architecture.

[0050] Any of the servers 12 may configure a virtual execution element (such as a pod or virtual machine) by virtualizing the server's resources to provide some measure of isolation between one or more processes (applications) executing on the server. "Hypervisor-based" or "hardware-level" or "platform" virtualization refers to the creation of virtual machines, each of which includes a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are virtualized from the physical hardware of the host server, the executed applications are isolated from the hardware of the host and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0051] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., one of data centers 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly allowed by security policy.

[0052] Virtual networks can use the data center 10 gateway router ( Figure 1 The virtual network can be connected to and extended on physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and Ethernet virtual private network (EVPN) networks (not shown). The virtual network can also be used to implement network function virtualization (NFV) and service chaining.

[0053] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks, namely a physical underlying network consisting of an IP structure 20 and a switching structure 14 and a virtual overlay network. The role of the physical underlying network is to provide an "IP structure" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0054] As further described below with respect to virtual routers 21A-21X (illustrated as and also collectively referred to herein as "vRouter 21"), the virtual routers running in server 12 can create a virtual overlay network on top of the physical underlying network using a dynamic mesh of "tunnels" between them. For example, these overlay tunnels can be MPLS tunnels over GRE / UDP or VXLAN tunnels, or NVGRE tunnels. The underlying physical routers and switches may not store any per-tenant state for virtual machines or other virtual execution elements, such as any media access control (MAC) addresses, IP addresses, or policies. The forwarding tables of the underlying physical routers and switches may, for example, contain only the IP prefix or MAC address of the physical server 12. (A gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0055] The virtual router 21 of the server 12 typically contains per-tenant state. For example, a separate forwarding table (routing instance) may be included for each virtual network. The forwarding table contains the IP prefixes (in the case of a layer 3 overlay) or MAC addresses (in the case of a layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., container pods). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that exist locally on the server 12 (i.e., there is at least one virtual execution element on the server 12).

[0056] The control plane protocol between the control plane nodes of the network controller 24 or physical gateway routers (or switches) can be BGP (and can be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3VPN and MPLS EVPN. For example, the protocol between the network controller 24 and the virtual router 21 can be based on XMPP.

[0057] "Container-based" or "operating system" virtualization refers to virtualizing an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by the open source Docker container application or CoreOS Rkt ("Rocket"). Like a virtual machine, each container is virtualized and can remain isolated from the host and other containers. However, unlike a virtual machine, each container can omit a separate operating system and instead provide a suite of applications and application-specific libraries. Typically, a container is executed by a host as an isolated userspace instance and can share an operating system and common libraries with other containers executing on the host. Therefore, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks.

[0058] In some examples, containers are managed by their host kernel to allow limiting and prioritizing resources (CPU, memory, block I / O, network, etc.) without the need to launch any virtual machines, in some cases using a namespace isolation feature that allows complete isolation of an application's (e.g., a given container's) view of the operating environment, including process trees, networking, user identifiers, and mounted file systems. In some examples, containers can be deployed in accordance with Linux Containers (LXC), an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

[0059] Server 12 hosts virtual network endpoints for one or more virtual networks that operate over a physical network represented herein by IP fabric 20 and switch fabric 14. Although primarily described with respect to a data center based switched network, other physical networks such as service provider network 7 may serve as the basis for one or more virtual networks.

[0060] Each server 12 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a set of one or more containers (e.g., a pod), or another virtual execution element, such as a third-layer endpoint of a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" may also include one or more pods of containers. A virtual execution element may represent an application workload. As Figure 1As shown, server 12A hosts a virtual network endpoint in the form of a pod 22 having one or more containers. However, given the hardware resource limitations of server 12, server 12 can execute as many virtual execution elements as possible. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0061] Each server 12 includes at least one network interface card (NIC) 13, each NIC including at least one interface for exchanging packets with a TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) described in the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to present one or more virtual network interfaces as "virtual functions" for use by various endpoints executing on the server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and the virtual function is an example of a virtual hardware component 21. As another example, one or more servers 12 can implement Virtio, which is a paravirtualization framework that can be used in, for example, a Linux operating system, which provides emulated NIC functions as a virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement an open vSwitch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) for hosted virtual machines, where such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some cases, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, the virtual hardware component is an SR-IOV virtual function. In some examples, any server in the server 12 may implement a Linux bridge that simulates a hardware bridge and forwards packets between the virtual network interfaces of the server or between the virtual network interfaces of the server and the physical network interfaces of the server. For a Docker implementation of a container hosted by a server, a Linux bridge or other operating system bridge executed on the server that exchanges packets between containers may be referred to as a "Docker bridge". The term "virtual router" used herein may include a Contrail or Tungsten Fabric virtual router, an open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software located on a host device and performing switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12.

[0062] One or more servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) for an Internet Protocol virtual private network (IP-VPN). For example, in some examples, a packet received by the virtual router 21 of the server 12A from the underlying physical network structure of the data center 10 (i.e., the IP structure 20 and the switching structure 14) may include an outer header to allow the physical network structure to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the network interface card 13A of the server, but also a virtual network identifier, such as a VxLAN label or a multi-protocol label switching (MPLS) label that identifies a virtual network and a corresponding routing instance executed by the virtual router 21. The inner packet includes an inner header having a destination network address that conforms to a virtual network addressing space of the virtual network identified by the virtual network identifier.

[0063] Virtual router 21 terminates the virtual network overlay tunnel, determines the virtual network of the received packet based on the tunnel encapsulation header of the packet, and forwards the packet to the appropriate destination virtual network endpoint of the packet. In the following, virtual routers are mainly described with respect to virtual router 21A, but this description is relevant to other instances of virtual router 21.

[0064] In some examples, for example, for each packet outbound from a virtual network endpoint hosted by server 12A (e.g., any of pods 22A-22C), virtual router 21A appends a tunnel encapsulation header indicating the virtual network of the packet to generate an encapsulated or "tunnel" packet, and virtual router 21A outputs the encapsulated packet to a physical destination computing device, e.g., another one of servers 12, via the overlay tunnel of the virtual network. As used herein, virtual router 21A may perform operations of a tunnel endpoint to encapsulate internal packets initiated by a virtual network endpoint to generate tunnel packets, and decapsulate tunnel packets to obtain internal packets for routing to other virtual network endpoints.

[0065] In some examples, virtual router 21A may be kernel-based and executed as part of the kernel of the operating system of server 12A.

[0066] In some examples, virtual router 21A can be a virtual router that supports the Data Plane Development Kit (DPDK). In such an example, virtual router 21A uses DPDK as a data plane. In this mode, virtual router 21A runs as a user space application linked to a DPDK library (not shown). This is a performance version of a virtual router and is typically used by telecommunications companies, where VNFs are typically DPDK-based applications. The performance of virtual router 21A as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the polling mode driver (PMD) of the DPDK instead of the interrupt-based driver of the Linux kernel.

[0067] Server 12 includes and executes containerized routing protocol daemons 25A-25X (collectively referred to as "cRPD 25"). A containerized routing protocol daemon (cRPD) is a routing protocol process that is packaged into a container and can run in a Linux-based environment. The cRPD can be executed in the user space of the host as a containerized process. Thus, the cRPD makes available a rich routing software pedigree of physical routers on Linux-based computing nodes (e.g., in some cases, server 12). The cRPD provides control plane functionality. The control plane is therefore containerized. For example, cRPD 25A implements the control plane of CNR 32A executed by server 12A.

[0068] Meanwhile, the virtual router 21 is a software entity that provides data plane functionality on the server 12. The cRPD 25A may use the forwarding (or data) plane provided by the Linux kernel of the server 12A for the kernel-based virtual router 21A. The cRPD 25A may alternatively use a DPDK-enabled or SmartNIC-executed instance of the virtual router 21. The virtual router 21A may work with an SDN controller (e.g., a network controller 24) to create an overlay network by exchanging routing, configuration, and other data. The virtual router 21A may be containerized. Thus, in some examples, in combination, the containerized cRPD and the containerized virtual router may be a fully functional containerized CNR 32A.

[0069] The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be an orchestration system. In some cases, the orchestration system is a container orchestration system that provides a container-centric infrastructure for automating the deployment, expansion, and operation of containers to provide a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provision, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements to host servers available to the orchestration platform. Specifically, container orchestration allows container orchestration and refers to the deployment, management, expansion, and configuration of containers to host servers via a container orchestration platform. Examples of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0070] Elements of the automation platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework, wherein a cluster master node of the cluster manages the deployment and operation of containers to one or more cluster child nodes of the cluster. The terms "master node" and "child node" used herein include different orchestration platform terms used to distinguish between the primary management element of the cluster and similar devices of the primary container hosting device of the cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "child node", while the Docker Swarm platform refers to the cluster manager and cluster nodes. A Kubernetes cluster is an example of an orchestration system cluster.

[0071] The orchestrator 23 and the network controller 24 may be executed on separate computing devices or on the same computing device. Each of the orchestrator 23 and the network controller 24 may be a distributed application executed on one or more computing devices. The orchestrator 23 and the network controller 24 may implement a corresponding master node for one or more clusters, each having one or more child nodes (also referred to as "computing nodes") implemented by a corresponding server 12.

[0072] In general, network controller 24 controls the network configuration of the data center 10 structure, for example, to establish one or more virtual networks for packetized communications between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator.

[0073] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides a computing infrastructure, which may include a container-centric computing infrastructure. The orchestrator 23 and, in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualized infrastructure for the container management platform.

[0074] Kubernetes operates using various Kubernetes objects - entities that represent the state of a Kubernetes cluster. Kubernetes objects can include any combination of names, namespaces, labels, annotations, field selectors, and recommended labels. For example, a Kubernetes cluster may contain one or more "namespace" objects. Each namespace of a Kubernetes cluster is isolated from other namespaces of the Kubernetes cluster. Namespace objects may include at least one of the organization, security, and performance of the Kubernetes cluster. As an example, a pod can be associated with a namespace, thereby associating the pod with a feature of the namespace (e.g., a virtual network). This feature enables multiple newly created pods to be organized by associating the pod with a set of common features. A namespace can be created according to namespace specification data that defines the namespace features, including a namespace name. In one example, a namespace can be named "namespace A", and each newly created pod can be associated with a set of features represented by "namespace A". In addition, Kubernetes includes a "default" namespace. If a newly created pod does not specify a namespace, the newly created pod can be associated with the features of the "default" namespace.

[0075] Namespaces enable a Kubernetes cluster to be used by multiple users, multiple teams of users, or one user using multiple applications. In addition, each user, team of users, or application can be isolated from every other user of the cluster within a namespace. Therefore, each user of a Kubernetes cluster in a namespace operates like a unique user of the Kubernetes cluster. Multiple virtual networks can be associated with a single namespace. In this way, pods belonging to a specific namespace have the ability to access every virtual network of the virtual networks associated with that namespace, including other pods that act as virtual network endpoints for that set of virtual networks.

[0076] In one example, pod 22A is a Kubernetes pod, which is an example of a virtual network endpoint. A pod is a group of one or more logically related containers ( Figure 1 ), shared storage for containers, and options on how to run containers. When instantiated for execution, a pod may also be referred to as a "pod copy". Each container of a pod 22A is an example of a virtual execution element. The containers of a pod are always located on the same server, co-scheduled, and run in a shared context. The shared context of a pod can be a set of Linux namespaces, cgroups, and other isolation aspects. Within the context of a pod, separate applications may apply further sub-isolations. Typically, containers in a pod have a common IP address and port space and are able to detect each other via the local host. Because of the shared context, containers in a pod also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different pods have different IP addresses and cannot communicate via IPC without a configuration that enables this feature. Containers that are members of different pods typically communicate with each other via the pod IP address.

[0077] Server 12A includes container platform 19A for running containerized applications, such as applications in pods 22. Container platform 19A receives a request from orchestrator 23 to obtain and host a container in server 12A. Container platform 19A obtains and executes the container.

[0078] The container network interface (CNI) 17A configures a virtual network interface for a virtual network endpoint. The orchestrator 23 and the container platform 19A use the CNI 17A to manage the networking of pods, including the pod 22. For example, the CNI 17A creates a virtual network interface to connect the pod 22 to the virtual router 21A and enables the containers of such a pod to communicate with other virtual network endpoints through the virtual network via the virtual network interface. For example, the CNI 17A may insert a virtual network interface of a virtual network into the network namespace of a container in the pod 22A, and configure (or request configuration) a virtual network interface of a virtual network in the virtual router 21A, so that the virtual router 21A is configured to send packets received from the virtual network via the virtual network interface to the container of the pod 22A, and to send packets received from the container of the pod 22A on the virtual network via the virtual network interface. The CNI 17A may assign a network address (e.g., a virtual IP address of the virtual network) and may establish a route for the virtual network interface. In Kubernetes, by default, all pods can communicate with all other pods without using network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a pod virtual network shared by all namespaces, from which service and pod network addresses are allocated, respectively. In some cases, all pods in all namespaces generated in the Kubernetes cluster are able to communicate with each other, and the network addresses of all pods can be allocated from the pod subnet specified by the orchestrator 23. When a user creates an isolated namespace for a pod, the orchestrator 23 and the network controller 24 can create a new pod virtual network and a new shared service virtual network for the new isolated namespace. Pods in the isolated namespace generated in the Kubernetes cluster extract network addresses from the new pod virtual network, and the corresponding services of such pods extract network addresses from the new service virtual network.

[0079] CNI 17A may represent a library, plugin, module, runtime, or other executable code of server 12A. CNI 17A may at least partially conform to the Container Network Interface (CNI) specification or rkt Networking Proposal. CNI 17A may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNI. CNI 17A may also be referred to as a network plugin or CNI plugin or CNI example. Separate CNIs may be called by, for example, Multus CNI to establish different virtual network interfaces for pod 202A.

[0080] CNI 17A can be called by the orchestrator 23. For the purposes of the CNI specification, a container can be considered a synonym for a Linux network namespace. What unit this corresponds to depends on the specific container runtime implementation: for example, in implementations of the application container specification (e.g., rkt), each pod runs in a unique network namespace. However, in Docker, a network namespace typically exists within each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and can communicate with each other. This can be an individual container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can be conceptually added to or removed from one or more networks. The CNI specification specifies some considerations for conformance plugins ("CNI plugins").

[0081] By adopting the Kubernetes CNI framework in some examples, CNR 32A can reduce the traditional operational overhead incurred when using containerized devices instead of their physical counterparts. By exposing the appropriate device interfaces, CNR 32A can standardize the operating model of virtual devices to physical devices, eliminating barriers to adoption in an operator's network operating environment. CNR 32A (and more specifically the interface of cRPD 25A) can present a familiar look and feel of routing devices to any well-trained operations team. CNR has similar features and capabilities and a similar operating model to hardware-based platforms. Similarly, in some cases, the domain controller can communicate with and control CNR 32A using the protocols it uses with any other router, for example, Netconf / OpenConfig, gRPC, Path Computation Element Protocol (PCEP), or other interfaces.

[0082] CNR 32A is actually a virtualized router and can participate in IS-IS, Open Shortest Path First (OSPF), BGP, and / or other internal or external routing protocols and exchange routing protocol messages with other routers residing on other hosts, whether these routers are physical routers or CNR 32B–32X (CNR 32A–32X are collectively referred to as "CNR32" herein). In addition, MPLS can be used, typically based on segment routing (SR). There are two reasons for doing this: to allow traffic engineering if needed, and to support multi-tenancy through the use of VPNs, such as Layer 3 VPNs or EVPNs based on MPLS.

[0083] Each pod 22 includes one or more containers. In some examples, one or more pods 22 include a containerized DPDK workload designed to use DPDK to accelerate packet processing, for example, by exchanging data with other components using a DPDK library. In some examples, virtual router 21 can be executed as a containerized DPDK workload.

[0084] The pods 22 are each configured with one or more virtual network interfaces (VNIs) 26 for sending and receiving packets through the virtual router 21. The virtual network interface 26 can be the default interface of the pod 22. The pod 22 can implement the virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while the virtual router 21 can implement the virtual network interface 26 as a tap interface, a virtio user interface, or other type of interface. Although described with respect to pods, the VNI 26 can be attached to other virtual execution elements, such as virtual machines. The virtual network interface may also be referred to as a virtual machine interface (VMI), even when referring to the VNI of a pod / container.

[0085] The pod 22 and the virtual router 21 use a virtual network interface 26 to exchange data packets. The virtual network interface 26 may be a DPDK interface. The pod 22 and the virtual router 21 may use vhost to establish a virtual network interface 26. The pod 22 may operate according to an aggregation model. The pod 22 may use a virtual device, such as a virtio device with a vhost user adapter, for user space container process intercommunication of the virtual network interface 26. The following mainly describes the pod 22A, but may be applied to other pods 22 or other virtual execution elements.

[0086] CNI 17A can be combined with Figure 1 One or more other components are shown configuring a virtual network interface 26 for the pod 22A. Any container of the pod 22A may utilize, ie, share, the virtual network interface 26 of the pod 22A.

[0087] In some examples, cRPD 25A includes a default VRF 28 (shown as "D.VRF 28") and VRFs 29A-29B (collectively referred to as "VRF 29"). Default VRF 28 stores a global routing table. cRPD 25A programs forwarding information derived from the routing table of VRF 29 into virtual router 21A as a forwarding table. In this way, virtual router 21A implements the VPN of VRF 29, which is shown as being included in virtual router 21A and cRPD 25A.

[0088] In some examples, cRPD 25A is configured to operate in host network mode, also known as internal networking. cRPD 25A therefore uses the network namespace and IP address of its host (i.e., server 12A). cRPD 25A can see and access network interfaces 30A-30B of NIC 13A, which are inserted into default VRF 28 and are viewed by cRPD 25A as core-facing interfaces. Interfaces 30A-30B are connected to switch fabric 14 and can be Ethernet interfaces. Interface 30 is viewed and used by cRPD 25A as a core-facing interface for providing VPN because interface 30 can be used to transmit VPN service traffic through a third layer network consisting of one or more of switch fabric 14, IP fabric 20, service provider network 7, or public network 15.

[0089] exist Figure 1 In the example server 12A of FIG. 1 , pod 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployments that specify a virtual network and that pod 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. Orchestrator 23 may receive configuration data from, for example, a user, an operator / administrator, or other machine system.

[0090] For example, as part of the process of creating pod 22A, orchestrator 23 requests network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). Pod 22A may have different virtual network interfaces for each virtual network to which it belongs. For example, virtual network interface 26 may be a virtual network interface for a particular virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks. Network controller 24 processes the request to generate interface configuration data for the virtual network interface of pod 22A. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. Below is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0091] In some examples, a pod may have multiple interfaces to different cRPD VRFs 29, e.g., one for management traffic and another for data traffic. Figure 1 Pod 22B may use VRF 29B for management traffic and VRF 29A for data traffic.

[0092] Pods 22 (e.g., pods 22A) may be connected to some physical interfaces, where pods 22 learn IP addresses of other devices, such as user devices of a pod implementing a mobile network gateway or customer network subnets of a pod implementing a CE router or gateway. In some examples, to advertise these IP addresses into the network, pods 22A treat virtual network interfaces 26 with VRFs 29 as IP links and advertise routes for these IP addresses to cRPD 25A. CRPD 25A may then advertise the reachability of these IP addresses through cRPD 25A and pods 22A as next hops, which also complies with the VPN service model. cRPD 25A programs virtual router 21 with corresponding forwarding information derived from VRF 29 and default VRF 28, and virtual router 21 forwards VPN traffic according to the VPN service model to implement VPN.

[0093] CRPD 25A can apply many different types of overlay networks / VPNs, including L3VPN or EVPN (Type 2 / Type 5), using various underlying tunnel types including, for example, MPLS, SR-MPLS, SRv6, MPLSoUDP, MPLSoGRE, or IP-in-IP.

[0094] CNI 17A can be combined with Figure 1 One or more other components shown configure virtual network interfaces 26 for pods 22s. Any container of pod 22 can utilize (i.e., share) any virtual network interface of the pod. Orchestrator 23 can store or manage virtual network interface configuration data for application deployment. For example, orchestrator 23 can receive a specification of a containerized application (a "pod specification" in the Kubernetes context) and network attachment definitions from a user, operator / administrator, or other machine system, and network controller 24 can derive configuration data from this information for configuring VRF 29 and default VRF 28; as well as configuring virtual network interfaces 26.

[0095] A traditional CNI plugin is called by a container platform / runtime, receives an Add command from the container platform to add a container to a single virtual network, and such a plugin may subsequently be called to receive a Del(ete) command from the container / runtime and remove the container from the virtual network. The term "call" may refer to instantiating a software component or module in memory into executable code for execution by a processing circuit.

[0096] exist Figure 1In the example computing infrastructure of , the network controller 24 is a cloud native distributed network controller for SDN, which is implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself can be implemented using one or more cloud native component microservices. Each control node 32 itself can be implemented using one or more cloud native component microservices.

[0097] The network controller 24 can be multi-tenant aware and support multi-tenancy of the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined networking structures and advanced networking features, such as virtual networking, BGPaaS, networking policies, service chains, and other telecommunications features. The network controller 24 can support network isolation using virtual network structures and support third-layer networking.

[0098] In order to interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual router 21) a network policy, which is referred to as a virtual network policy (VNP) and is interchangeably referred to as a virtual network router or virtual network topology in this article. VNP defines the connection policy between virtual networks. A single network controller 24 can support multiple Kubernetes clusters, so VNP allows multiple virtual networks to be connected in namespaces, Kubernetes clusters, and across Kubernetes clusters. VNP can also be extended to support virtual network connections across multiple instances of the network controller 24.

[0099] The network controller 24 can use network policies to implement multiple layers of security. The default behavior of Kubernetes is to let pods communicate with each other. To apply network security policies, the SDN architecture implemented by the network controller 24 and the virtual router 21 can run as the CNI of Kubernetes through CNI 17A. For Layer 3, isolation occurs at the network level and virtual networks operate at Layer 3. Virtual networks are connected through policies. Kubernetes native network policies provide security at Layer 4. The SDN architecture may support Kubernetes network policies. Kubernetes network policies operate on Kubernetes namespace boundaries. The SDN architecture can add customized resources for enhanced network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta-tags to apply granular security policies in an extensible manner.) For Layer 4+, in some examples, the SDN architecture can support integration with containerized security devices and / or Istio, and can provide encryption support.

[0100] The CNR (e.g., CNR 32A) is described in more detail in U.S. Patent Publication 2022-0279420A1, entitled “CONTAINERIZED ROUTER WITH VIRTUAL NETWORKING,” published on September 1, 2022, the entire contents of which are incorporated herein by reference.

[0101] According to the technology of the present disclosure, a containerized router operating in a public cloud environment can provide routing control plane functions for VPC workloads reachable via a VPC gateway. The containerized router can execute a routing protocol to learn routes associated with the VPC workloads, and can program the VPC routing table in the VPC gateway device so that the VPC gateway device routes traffic to the containerized router for further routing. This traffic can include traffic from one or more containers (also referred to as pods). This arrangement may be beneficial because the pods and internal routers may include a Border Gateway Protocol (BGP) speaker, but the VPC gateway may not support BGP and therefore cannot process BGP advertisements or routing information of BGP advertisements to determine how to route overlay traffic originating from or going to workloads in the VPC, such as BGP traffic.

[0102] According to the technology of the present disclosure, redundant containerized routers can be used in a public cloud environment. The redundant containerized routers can be connected to each other, for example, via a virtual router redundancy protocol (VRRP) session. A first containerized router in the redundant containerized routers can determine that the first containerized router is an active containerized router via the VRRP session. The first containerized router can be connected to a VPC gateway and program a VPC routing table in the VPC gateway.

[0103] Figure 2 is a block diagram of an example cloud computing environment including a CNR for programming a VPC routing table of a VPC gateway according to one or more aspects of the present disclosure. The cloud computing environment 100 may be Figure 1The cloud computing environment 100 may be configured to provide VPC services to customers of the operator of the cloud computing environment 100. The VPC may operate as an extension of an internal network (e.g., internal network 130) of a customer of the operator of the cloud computing environment 100, or as a virtual network similar to a traditional (or physical) network of a customer of the operator of the cloud computing environment 100. The VPC may include one or more subnets and one or more gateways, each subnet having its own IP address range, and the gateway may be referred to as a VPC gateway herein. A gateway may be a device (physical or virtual) that connects the VPC to other networks. For example, an Internet gateway (e.g., VPC gateway 116) may connect the VPC to the Internet 140.

[0104] Cloud computing environment 100 may include CNR 150, VPC gateways 110A-110N (collectively, VPC gateways 110) that may provide access to clusters 102A-102N (collectively, clusters 102). VPC gateways 110 may be Figure 1 Example of VPC Gateway 7. Cluster 102A may include multiple worker (compute) nodes 104A-104N, which may include Kubernetes worker nodes. Similarly, cluster 102N may include multiple worker nodes 106A-106N. Each of worker nodes 104A-104N and worker nodes 106A-106N may include a BGP speaker (e.g., BGPS1-BGPSN) and one or more pods, such as pods P1 and P2. In some examples, P1 and P2 may not have BGP enabled. As Figure 2 As shown, pod P1 represents a pod of a first service and pod P2 represents a pod of a second service. Pods P1 and P2 may be part of the same subnet or different subnets. Each service may have a unique IP address, which may be a virtual IP or a service IP. The IP address of pod P1 may be extracted from the service subnet of the first service. A given worker node may include pods for one or more services. Cluster 102 may represent a cluster of a VPC, which may be implemented as an extension of another network, such as internal network 130.

[0105] BGP speakers BGPS1-BGPSN, which may be pods (such as, pods of cluster 102A), may know or understand the IP addresses associated with pods P1 and P2. In some examples, BGP speakers BGPS1-BGPSN may be part of the CNI that assigns IP addresses to pods P1 and P2, and therefore will know the IP addresses of pods P1 and P2. In some examples, BGP speakers BGPS1-BGPSN may learn the IP addresses, for example, through interaction with Kubernetes elements, and may advertise such IP addresses via a data protocol such as BGP. However, as described in further detail below, VPC gateway 110 may be unable to process routing protocol (e.g., BGP) messages from BGP speakers BGPS1-BGPSN and therefore may be unable to learn routes to pods P1 or P2.

[0106] Cloud computing environment 100 may also include VPC gateway 112, VPC gateway 114, and VPC gateway 116. VPC gateway 112 may provide access to cloud native network functions (CNFs) 120. CNFs 120 may be software services that provide network functions to cloud computing environment 100. In some examples, CNFs 120 may be implemented by pods or containers. VPC gateway 114 may provide access to internal network 130, and VPC gateway 116 may provide access to Internet 140. Although in Figure 2 Although not shown, cloud computing environment 100 may include additional devices (which may be physical or virtual) that may be used to access Figure 2 The VPC gateway 116 may be used as an Internet gateway to provide access to the Internet 140 to the various clusters 102 of the cloud computing environment 100. The VPC gateway 114 may be used as a gateway for the internal network 130 to provide access between the internal network 130 and the cluster 102 of the VPC of the internal network 130. In some examples, each of the cluster 102, VPC 110, and CNR 150 may act as a layer 3 router so that no tunnel is required between these devices. For example, the CNR 150 may employ MAC learning to direct traffic from, for example, the internal network 130 to the cluster 102.

[0107] Although the cloud computing environment 100 may have a VPC including a VPC gateway 110, such a VPC gateway may provide limited functionality due to the limitations of the underlying hardware platform (e.g., in terms of scale, functionality, etc.) or software implementation (e.g., in terms of equal-cost multipath routing (ECMP), coverage, signaling, protocols, etc.), scale (e.g., in terms of a limited number of routes, tables / instances, number of protocol sessions such as BGP, etc.) of the workloads deployed on these environments. In some examples, the VPC gateway 110 may not operate using traditional routing protocols and / or may need to use custom APIs and non-portable protocols. For example, the devices of the cloud environment 100 may not be able to talk to the VPC gateway 110 using any standard protocol. The VPC gateway 110 may not know the IP address of the internal network 130. As such, a BGP speaker (e.g., BGPS1 of the workload 104A) may not be able to advertise the workload 104A or the pods P1 and P2 of the workload 104A to the VPC gateway 110A.

[0108] To address this issue, CNR 150 may use an API to program VPC routing table 118 of VPC gateway 110. Such programming may not introduce a new tunnel between VPC 110 and CNR 150, but rather instructs VPC gateway 110 how to forward packets (e.g., to workloads 104 and 106) using the same IP header information used by CNR 150. In some aspects, CNR 150 may view VPC gateway 110 as a forwarding plane—instructing VPC gateway 110 how to forward packets from different routes learned by CNR 150 via, for example, routing protocol messages advertised by BGPS1-BGPSN of cluster 102. In these aspects, CNR 150 may be viewed as a control plane for VPC gateway 110 because CNR 150 may program VPC routing table 118 of VPC 110. When programming VPC routing table 118, CNR 150 may use cRPD 25A ( Figure 1 ) policy engine and apply different routing policies to each VPC routing table.

[0109] For example, CNR 150 can use routing policies as filters to affect the routing of packets. The routing policies can be based on source addresses and destination addresses. In this way, for example, in the case where internal network 130 includes multiple tenants, packets can be routed within an appropriate VPN (e.g., VPC).

[0110] In some examples, CNR 150 abstracts and hides all VPC network details from the hosted service, while also making the solution portable across different cloud environments and providing easy migration to an on-premises environment. CNR 150 can provide this by using a network overlay or by interacting directly with cloud network elements.

[0111] For example, CNR 150 can interact with the cloud environment bottom layer, learn the bottom layer network topology, and then distribute routes into the bottom layer to provide seamless connectivity between applications deployed in the cloud environment and on-premises or another cloud environment (e.g., a hybrid cloud).

[0112] For example, CNR 150 can dynamically establish (BGP) routing adjacencies on worker nodes or pods and distribute routing information to the cloud environment underlay and / or advertise to remote BGP peers. These prefixes are not learned (automatically) by VPC Gateway 110, but in some cases, these prefixes are required to achieve, for example, traffic forwarding from VPC Gateway 110 to, for example, internal network 130.

[0113] When deploying an IP router in a cloud environment with a VPC, routing may not be provided for most cases because such cloud environments have their own layer 3 based network (underlying) VPC infrastructure that prevents the use of protocols such as IS-IS, OSPF to establish routing adjacencies due to lack of support for multicast and broadcast traffic. The present disclosure describes a CNR that provides routing functionality in a cloud environment with one or more VPCs. Although shown separately, in some examples, CNR 150 can be implemented in a cluster of clusters 102.

[0114] exist Figure 2 In the example of FIG. 1 , CNR 150 performs a query on VPC routing table 118 (in Figure 1 In the example of FIG. 1 , a standard router (shown as VPCRT 118A-118N) is programmed so that VPC gateway 110 forwards traffic to CNR 150, while a standard router would simply program its own forwarding information base (FIB) to forward incoming traffic out of itself. Since CNR 150 can learn the routes of internal network 130, for example, CNR 150 can route traffic from VPC gateway 110 to internal network 130 after receiving the traffic from VPC gateway 110 that is destined for internal network 130.

[0115] CNR 150 may program VPC routing table 118 of VPC 110 using an API of VPC 110. CNR 150 may program multiple VPC tables 118, each with a corresponding (e.g., different) set of routes and next hops. In some examples, CNR 150 may map cRPD routes and interfaces to VPC routing tables 118 and interfaces based on operator-defined labels. In some examples, an operator may define routing tables, next hops, route selection criteria, etc. through routing policies that may be applied by CNR 150. Administrators may also statically program VPC routing tables, but this approach is not scalable and does not take advantage of the high scalability of VPC for customers.

[0116] For example, a VPC routing table (e.g., VPC routing table 118A) may include an indication that for a particular IP address, the next hop IP address should be zero or a worker node (e.g., worker node 104A). cRPD 25A may use an interface ID, such as a virtual network interface on a VPC gateway and an interface name on CNR 150.

[0117] VPC 110 may know the IP addresses of worker nodes 104 and 106, but may not know the IP addresses assigned to application pods (e.g., P1 and P2) within worker nodes 104 and 106. For example, VPC 110 may not know how to transmit packets from pods P1 and P2 to internal network 130, or vice versa.

[0118] An administrator may determine different routing policies that CNR 150 uses to program each VPC routing table 118. For example, policies may be used to determine which routes are programmed into a VPC routing table. In this way, policies may act as filters so that a given VPC routing table (e.g., VPC routing table 118A) may include a subset of available routes.

[0119] In some examples, there may be overlap in destination IP addresses between different intended destinations. In this case, using policy-based routing, CNR 150 can use both the source IP address and the destination IP address to classify packets into different VPNs to route packets from all pods sharing the same interface to the same VPC, such that packets destined for internal network 130 are sent to VPC 114 instead of a VPC associated with another internal network.

[0120] In some examples, the BGP speaker may be a CNI that assigns pod IP addresses, so the BGP speaker (eg, BGPS1) knows the IP addresses of the associated pods (P1 and P2 of worker node 104A). In some examples, the BGP speaker may be a plugin.

[0121] Some wireless service providers may want to migrate parts of their core network to the public cloud as they expand their existing infrastructure to accommodate 5G. For example, the cloud-native core functions of 5G (e.g., unified data management (UDM), network slice selection function (NSSF), and network repository function (NRF)) can be containerized and migrated to the public cloud. Since wireless service providers can provide slicing functions and provide slices based on factors such as functionality, performance, user equipment (UE) groups, etc., these CNFs should be dynamically elastically scaled up or down in the public cloud environment using load balancer services. Service providers can also leverage the inherent redundancy capabilities provided by the public cloud to distribute their functions in different geographical regions in an active-standby mode. Service providers want seamless connectivity, low latency, and high throughput from their internal data centers to the core functions running in the cloud. The main requirement for service providers with geographically distributed networks may be seamless switchover to different geographical regions in case of natural disasters / failures. Traditional public cloud services may not support the network services required for such use cases. Therefore, it may be desirable to provide cloud-internal software routers to address these requirements.

[0122] The CNR 150 may utilize the DPDK vrouter as its data plane to accelerate packets processed per second (PPS) to deliver the required packet performance in a cloud environment.

[0123] CNR 150 can be used to extend a 5G core infrastructure running internally (e.g., internal network 130) to, for example, VPC 110A. In the 5G example, different services (e.g., NSSF, NRF, and UDM, each with their own service IP addresses) can run in an EKS cluster (e.g., cluster 102A), each running in their own VRF (e.g., SIGVRF, OAM VRF), and CNR 150 can, for example, provide seamless communication with VRF counterparts (e.g., CU) running in internal network 130 in complete isolation. For example, cluster 102A can include a node group with a maximum and minimum size that is automatically scaled to 2. 5G CNF pods and CNR 150 can run in different clusters. For example, a 5G CNF pod can run in cluster 102A, while CNR 150 can run in another cluster of clusters 102A-102N. The service IP address may be a cluster endpoint that can only be accessed within the VPC. Cluster networking can be provided by a cloud provider VPC container networking interface (CNI) plugin. The CNI plugin allows Kubernetes pods to have the same IP address as the Kubernetes pods on the VPC network. All containers within a pod that shares a network namespace can communicate with each other using ports.

[0124] To attract traffic from these 5G CNFs and internal CNFs, CNR 150 can program internal and external VPC routing tables via the VPC API. For example, CNR 150 can include a complete BGP stack through which CNR 150 communicates with 5G CNFs, transit gateways (TGWs) (such as Figure 3 150) and any internal edge gateway peers running in the cloud computing environment 100 to exchange routes. The 5G CNF may have a BGP capability to advertise the associated service virtual IP address to CNR 150. In some examples, CNR 150 includes a BGP auto-configuration function, where CNR 150 automatically establishes a BGP session when a new CNF workload is orchestrated.

[0125] As described above, the routes in VPC routing table 118 are programmed by a CNR (such as CNR 150) so that VPC gateway 110 forwards traffic to the CNR. In a deployment with a single CNR 150, a failure of CNR 150 will interrupt traffic to / from services hosted on worker nodes in the cluster. Traffic disruption may be temporary or short-lived. Running two instances of a CNR in active / standby mode can significantly reduce traffic disruption by shortening recovery time.

[0126] Figure 3 2 is a block diagram of an example cloud computing environment with redundant CNRs according to one or more aspects of the present disclosure. Cloud computing environment 200 may be similar to cloud computing environment 100, but includes redundant CNRs (CNR 250A and CNR 250B), which may be Figure 2 Example of CNR 150. Figure 3 In the example of FIG. 1 , the cloud computing environment 200 is shown coupled to an internal network 330, which may be Figure 2 The internal network 330 may be an example of an internal network 130. The internal network 330 may be a wireless wide area network, such as a 5G network.

[0127] CNR 250A and CNR 250B may include virtual router redundancy protocol (VRRP) components 214A and 214B, respectively, which may be used to establish a VRRP session 201 between CNR 250A and CNR 250B. In some examples, one or more of VRRP components 214A or 214B may be implemented as a pod, such as pod P1 or P2. In order to determine which of CNR 250A and CNR 250B should be the active node and which should be the standby node, CNR 250A and CNR 250B may run a VRRP session 201. VRRP is a protocol that may be used to provide redundancy in a network. VRRP session 201 may select one of CNR 250A or CNR 250B as the active node (sometimes referred to as the master node) and select the other of CNR 250A or CNR 250B as the standby node (sometimes referred to as the backup node). The active CNR is responsible for operating as a CNR, similar to Figure 2 If a problem occurs with the active CNR or the link to the active CNR that prevents the active CNR from functioning as expected, the standby CNR can still assume the function of the active CNR. In some examples, VRRP session 201 may include a unicast VRRP session. Private cloud environments with VPCs may not support L2-based VRRP because these private cloud environments may not support IP broadcast and / or multicast traffic. As such, VRRP session 201 may be a unicast VRRP session.

[0128] In some examples, CNR 250A and CNR 250B can share a virtual IP address on an interface facing worker node 204 (e.g., an interface with VPC 210). In some examples, the shared virtual IP address will only be used by the active nodes of CNR 250A and CNR 250B. The active node can forward traffic to / from worker node 204, while the standby node cannot. In addition, the active node can update the VPC routing table 218, while the standby node cannot. In this way, a VPC coupled to CNR 250A and CNR 250B (e.g., VPC 210) can only forward traffic to the active node.

[0129] The active CNR node may have an L2 MAC address that is different from the L2 MAC address of the standby CNR node. The active CNR node may advertise the L2 MAC address of the active CNR node, while the standby CNR node may not advertise the L2 MAC address of the standby CNR node. For example, if CNR 250A is the active node, CNR 250A may advertise the L2 MAC address of CNR 250A, while the standby node CNR 250B may not advertise the L2 MAC address of CNR 250B.

[0130] For example, CNR 250A may continue to operate as an active node while CNR 250B operates as a standby node until a failure of CNR 250A is detected. When the failure of CNR 250A is detected via VRRP 201, CNR 250B (standby node) will become the active node. In this case, in order to continue to attract traffic to the active node (which has now changed from CNR 250A to CNR 250B), CNR 250B may learn routes from neighboring devices, and the routing agent application of CNR 250B may update VPC routing table 218 to update the forwarding next hop to CNR 250B (instead of CNR 250A), and the routing agent application of CNR 250A may stop updating VPC routing table 218. In this way, traffic from worker 204 may be forwarded by VPC 210 to CNR 250B, the new active node. For example, in an L2-based segment, CNR 250B may use gratuitous ARP to update the ARP entry corresponding to the virtual IP address so that traffic is now forwarded to CNR 250B, the new active VRRP master.

[0131] CNR 250B may begin advertising the L2 MAC address of CNR 250B, and CNR 250A may stop advertising the L2 MAC address of CNR 250A. In this manner, traffic from, for example, TGW 220 may be directed to CNR 250B instead of CNR 250A.

[0132] In some examples, VRRP control state may be used to modify BGP routing attributes, such as local-pref (I-BGP) or multi-exit discriminator (MED) (e-BGP), so that a remote, internal device (virtual or physical), such as a BGP peer, will prefer CNR 250B (the VRRP active node) over CNR 250A as the egress PE.

[0133] The upstream router (such as TGW 220) may have a virtual IP address that may be used by the active CNR as a forwarding next-hop IP address. In some examples, both CNR 250A and CNR 250B (active and standby nodes) advertise BGP prefixes to internal network 330 to establish BGP sessions with BGP peers in internal network 330. In some examples, the BGP speakers (BGPS1-BGPSN) of worker node 204 advertise BGP prefixes to CNR 250A and CNR 250B.

[0134] In some examples, if the virtual IP address is in the same subnet as the worker nodes, the worker nodes 204 may use the virtual IP address shared by CNR 250A and CNR 250B as a default route. This approach will result in fewer updates to the VPC routing table 218, but will require the active VRRP node to obtain the virtual IP address from the VPC infrastructure. In such an example, the virtual IP address should be known to all worker nodes.

[0135] exist Figure 3 In the example of , worker node 204 is part of cluster 302, which can host 5G core network functions (CNFs) such as NSSF, UDM, etc. for service providers associated with internal network 330. Each of these CNFs can have its own virtual IP address. In some examples, these services can operate in different virtual routing and forwarding functions (VRFs) and include traffic isolation between these services. For example, there may be a virtual IP address for operations, administration and maintenance (OAM) VRF (10.10.10.10 / 32) and another virtual IP address for secure internet gateway (SIG) VRF (10.10.11.10 / 32). Each worker of worker node 204 can have a BGP speaker of BGPS1-BGPSN, which can be used to peer with the active CNRs of CNRs 250A and 250B and advertise the virtual IP address.

[0136] In some examples, CNRs 250A and 250B may be in a cluster ( Figure 3 ), such as cluster 102N ( Figure 2 ). For example, the auto-scaling size of such a cluster can be 2. One of CNR 250A or 250B can be in active mode, while the other of CNR 250A or 250B can be in standby mode, providing redundancy for the functionality of the active CNR. In some examples, CNR 250A and CNR 250B can both be located in different geographic locations (such as, cloud computing environment 200 is a geographically dispersed cloud computing environment) to provide geographic redundancy. CNR 250A and 250B can peer with TGW 220 to reach an internal network 330, which can include a telecommunications data center 322, in which a core network, network functions (CN NF 226), centralized units (CU) 228 and / or distributed units (DU) 234 can be located. It should be understood that the internal network 330 may also include a cellular radio / antenna 332.

[0137] exist Figure 3In the example of , the active CNR can support multiple interfaces (e.g., up to 8) and can route traffic between these interfaces. For example, the Ethernet interface in CNR 250A and / or CNR 250B can be used to send traffic such as http, https, etc. to the worker nodes 204 of all virtual IP addresses.

[0138] In some examples, CNR 250A and CNR 250B may include BGP auto-configuration so that when a new BGP speaker (such as, BGPSN) is started, no new configuration needs to be added. In some examples, CNR 250A and CNR 250B may include bidirectional forwarding detection (BFD), which can be used to detect sub-second failures of 5G workloads on worker node 204 and reroute traffic. In some examples, CNR 250A and CNR 250B can utilize VRF-lite. For example, a virtual routing domain with route leakage can be configured to route traffic across different VRFs. For example, prefixes in SIG VRF and OAM VRF may be leaked in an Ethernet interface. In some examples, CNR 250A and CNR 250B can support equal cost multipath (ECMP) routing. For example, a new worker (e.g., worker node 104N) can be added or deleted instantly without affecting existing flows. This can be referred to as flow affinity or flow viscosity. In some examples, CNR 250A and CNR 250B can be configured to interact with VPC gateway 210 via APIs (such as, APIs 216A and 216B) that are programmed to populate external and internal routing tables (e.g., routing table 218) based on BGP advertisements (e.g., from BGP speakers BGPS1-BGPSN) to attract traffic to the active one of CNR 250A or CNR 250B.

[0139] In some examples, CNR 250A and CNR 250B can provide asymmetric routing. For optimization and cost savings, there may be some flows from active CNR to worker node 204, but reverse flows from worker node 204 can be routed directly from VPC gateway 210. In some examples, to support active-standby switching between CNR 250A and CNR 250B, CNR 250A and CNR 250B can use one or more protocols, such as VRRP. In some examples, because most public clouds do not support multicast or broadcast, CNR 250A and CNR 250B can use unicast VRRP. In some examples, CNR 250A and CNR 250B can support tunnels, such as MPLS tunnels over UDP (MPLSoUDP). For example, CNR 250A and CNR 250B can tunnel traffic to internal network 330 using MPLSoUDP encapsulation. In some examples, CNR 250A and CNR 250B may provide control plane features, such as EVPN-VXLAN.

[0140] Figure 4 is a block diagram of an example computing device (eg, host) in accordance with techniques described in this disclosure. Figure 4 The computing device 300 may represent a physical or virtual server and may represent Figure 1 344 . In some examples, the bus 342 may couple the storage device 344, the microprocessor 320, and the NIC 330. The bus 342 may represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller may control DMA transfers between components coupled to the bus 342. In some examples, components coupled to the bus 342 control DMA transfers between components coupled to the bus 342.

[0141] The microprocessor 320 may include one or more processors, each processor including an independent execution unit to execute instructions that conform to an instruction set architecture, the instructions being stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined in one or more multi-core processors (or "many-core" processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0142] Disk 346 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information, such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD) or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 320.

[0143] The main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM), for example, DDR2 / DDR3 SDRAM, or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium, which can be used to transmit or store the required program code and program data in the form of instructions or data structures and can be accessed by the computer. The main memory 344 provides a physical address space consisting of addressable storage locations.

[0144] The network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange packets using the links of the underlying physical network. The interface 332 may include a port interface card having one or more network ports. The NIC 330 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 330 and other devices coupled to the bus 342 may read / write from / to the NIC memory.

[0145] Memory 344, NIC 330, storage disk 346 and microprocessor 320 can provide an operating environment for a software stack, which includes an operating system kernel 380 executed in kernel space. Kernel 380 can represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel or Windows server operating system kernel available from Microsoft. In some examples, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can include a virtual machine manager (VMM). The operating system including kernel 380 provides an execution environment for one or more processes in user space 345.

[0146] Kernel 380 includes a physical driver 325 that uses a network interface card 330. Network interface card 330 may also implement SR-IOV to enable sharing of physical network functions (I / O) between one or more virtual execution elements, such as containers 329A and 329B or one or more virtual machines ( Figure 2 330). A shared virtual device (such as a virtual function) may provide dedicated resources so that each virtual execution element may access the dedicated resources of the NIC 330, and thus the NIC 330 appears as a dedicated NIC to each virtual execution element. A virtual function may represent a lightweight PCIe function that shares physical resources with a physical function used by the physical driver 325 and other virtual functions. For a NIC 330 that supports SR-IOV, the NIC 330 may have thousands of virtual functions available according to the SR-IOV standard, but for I / O intensive applications, the number of configured virtual functions is typically much smaller.

[0147] Computing device 300 may be coupled to a physical network switch fabric that includes an overlay network that extends the switch fabric from physical switches to software or "virtual" routers, including virtual router 306A, that are physical servers coupled to the switch fabric. A virtual router may be a physical server (e.g., Figure 1 A process or thread or component thereof executed by a server 12) that dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints, which can include VPCs. In one example, the virtual router implements each virtual network using an overlay network that provides the ability to separate the virtual address of the endpoint from the physical address (e.g., IP address) of the server on which the endpoint executes. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and between virtual networks on a physical network. The term "virtual router" as used herein may include Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other device and / or software that is located on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. In Figure 4 In the example computing device 300 of , the virtual router 306A executes as a virtual router in user space, but in various implementations, the virtual router 306A may execute within a hypervisor, a host operating system, a host application, or a virtual machine.

[0148] The virtual router 306A can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used for Kubernetes deployments of pods 302. The virtual router 306A can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 306A can perform networking services such as applying security policies, NAT, multicast, mirroring, and load balancing. In some examples, the virtual router 306A can perform traffic to / from one or more VPCs (e.g., Figure 2 Routing of packets to any VPC 110).

[0149] Virtual router 306A may be executed as a kernel module or a user space DPDK process (virtual router 306A is shown here in user space 345). Virtual router agent 314 may also be executed in user space. Figure 4 In the example computing device 300 of , the virtual router 306A is executed in user space as a DPDK-based virtual router, but in various implementations, the virtual router 306A can be executed within a hypervisor, a host operating system, a host application, or a virtual machine. In some examples, the virtual router agent 314 connects to the network controller 24 using a channel, which is used to download configuration and forwarding information. The virtual router agent 314 programs the forwarding state into the virtual router data (or "forwarding") plane represented by the virtual router 306A. The virtual router 306A and the virtual router agent 314 can be processes.

[0150] The virtual router 306A can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used for Kubernetes deployments of the pod 302. The virtual router 306A can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 306A can perform networking services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0151] Virtual router 306A can be multithreaded and executed on one or more processor cores. Virtual router 306A can include multiple queues. Virtual router 306A can implement a packet processing pipeline. Depending on the operation to be applied to the packet, virtual router agent 314 can stitch the pipeline from the simplest to the most complex. Virtual router 306A can maintain multiple instances of the forwarding basis. Virtual router 300 can use RCU (read copy update) locks to access and update tables.

[0152] To send packets to other computing nodes or switches, virtual router 306A uses one or more physical interfaces 332. Typically, virtual router 306A communicates with workloads 104 or 106 (eg, Figure 2 ) to exchange packets. In some examples, virtual router 306A has multiple virtual network interfaces (e.g., vifs). These interfaces may include a kernel interface vhost0 for exchanging packets with the host operating system; an interface with the virtual router agent 314 (pkt0) for obtaining forwarding state from the network controller and sending exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 332. In some examples, virtual router 306A shares a virtual interface with another virtual router facing one or more VPCs 110 (facing workloads 104 or 106). In this case, virtual router 306A and another virtual router can be in an active / standby configuration, such as, Figure 3 Configuration of the CNR 250A and CNR 250B.

[0153] In a kernel-based deployment of virtual router 306A (not shown), virtual router 306A is installed as a kernel module in the operating system. Virtual router 306A records itself to the TCP / IP stack to receive packets from any desired operating system interface it wants. The interface can be bonded, physical, tap (for VM), virtual interface (veth) (for container), etc. The virtual router 306A in this mode relies on the operating system to send and receive packets from different interfaces. For example, the operating system can display a tap interface supported by the vhost-net driver to communicate with the VM. Once the virtual router 306A records the packet from the tap interface, the TCP / IP stack sends all packets to it. Virtual router 306A sends packets via the operating system interface. In addition, the NIC queue (physical or virtual) is processed by the operating system. Packet processing may run in interrupt mode, which will generate interrupts and may cause frequent context switching. When the packet rate is high, the overhead of frequent interrupts and context switching may overwhelm the operating system, resulting in performance degradation.

[0154] In a DPDK-based deployment of virtual router 306A, virtual router 306A is installed as a user space 345 application linked to the DPDK library. This may result in faster performance than a kernel-based deployment, especially in the presence of high packet rates. The DPDK's polling mode driver (PMD) uses the physical interface 332 instead of the kernel's interrupt-based driver. The registers of the physical interface 332 can be exposed in user space 345 so that the PMD can access it; the physical interface 332 bound in this way is no longer managed or visible to the host operating system, and the DPDK-based virtual router 306A manages the physical interface 332. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the virtual router 306A DPDK data plane. When the packet rate is high, the nature of this "polling mode" makes the virtual router 306A DPDK data plane packet processing / forwarding more efficient than interrupt mode. Compared to kernel mode virtual router 306A, interrupts and context switches during packet I / O are relatively rare, and in some cases interrupts and context switches during packet I / O can be avoided entirely.

[0155] In general, each pod 302A-202B may be assigned one or more virtual network addresses for use in a corresponding virtual network, wherein each virtual network may be associated with a different virtual subnet provided by virtual router 306A. Pod 302B may be assigned its own virtual layer 3 (L3) IP address, e.g., for sending and receiving communications, but may not be aware of the IP address of the computing device 300 executing pod 302B. The virtual network address may therefore be different from the logical address of the underlying physical computer system (e.g., computing device 300).

[0156] The computing device 300 includes a virtual router agent 314 that controls the virtual network overlay of the computing device 300 and orchestrates the routing of packets within the computing device 300. Typically, the virtual router agent 314 communicates with a network controller 24 for a virtualized infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 300, and more specifically, the virtual router 306A and the virtual network interface 212. By configuring the virtual router 306A based on information received from the network controller 24, the virtual router agent 314 can support configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0157] As described above, the network controller 24 may provide a logical centralized controller to facilitate the operation of one or more virtual networks. The network controller 24 may, for example, maintain a routing information base, for example, one or more routing tables storing routing information for a physical network and one or more overlay networks. The virtual router 306A implements one or more virtual routing and forwarding instances (VRFs), such as VRF 322A, for the corresponding virtual network for which the virtual router 306A operates as a corresponding tunnel endpoint. Typically, each VRF 322 stores forwarding information for a corresponding virtual network or VPC, and identifies where a packet will be forwarded and whether the packet will be encapsulated in a tunnel protocol, such as using a tunnel header, which may include one or more headers of different layers of a virtual network protocol stack. Each VRF 322 may include a network forwarding table storing routing and forwarding information for a virtual network. For example, the virtual router 306A may learn routes from the cRPD 324 via a virtual router agent 314 that may include an API, and use the learned routes to build a forwarding table for each VRF 322.

[0158] The cRPD 324, acting as the control plane for the virtual router 306A, can learn BGP routes for packets flowing through the computing device 300. In this way, the cRPD 324 can program the routing table of the virtual router 306 (e.g., via the virtual router agent 314). The VPC table programmer 326 can communicate with the cRPD 324 to learn the BGP routes for packets flowing through the computing device 300. The VPC table programmer 326 can be a table programming agent configured to program a VPC routing table (such as, VPC routing table 118) of the VPC gateway 110. Although shown as part of the pod 302B, in some examples, the cRPD 324 and the VPC table programmer can be part of different pods, rather than part of the same pod. For example, the VPC table programmer 326 can be executed in a separate pod of the pod 302. In some examples, VPC table programmer 326 can be configured to utilize an API (such as an API of the VPC gateway) to interface with the VPC gateway when programming the VPC routing table.

[0159] The VPC routing table 118 can change from situation to situation. For example, if a VPC is restarted, the associated VPC routing table may change. In some examples, the VPC table programmer 326 can abstract the VPC routing table from a configuration perspective. For example, the VPC table programmer 326 can enable administrators to interact with the VPC routing table using tags (which may be more descriptive) (such as, "vpc-table-tag": "cnr-aws-vpc-internal-table") instead of names (such as, "RouteTableId": "rtb-08fdf3cc822baa22c"). The VPC table programmer 326 can perform any transformations, such as applying appropriate policies to the appropriate VPC routing table, or updating the appropriate VPC routing table.

[0160] An example mapping that VPC table programmer 326 may perform for VPC gateway 118 is as follows:

[0161]

[0162]

[0163] An example policy statement is as follows:

[0164]

[0165] An example of programming a routing table for internal network 130 is as follows:

[0166]

[0167]

[0168] For traffic flowing to workloads 104 and 106, VPC table programmer 326 may program the L2D MAC to identify the appropriate node.

[0169] NIC 330 may receive a packet with a BGP header. Virtual router 306A processes the packet to determine from the BGP header which BGP speaker the packet is to be routed to. For example, a first packet from internal network 130 may be destined for BGPS2 of workload 104B, or a second packet from BGPS1 of workload 106B may be destined for Internet 140 ( Figure 2 ). Virtual router 306A may route the first received packet to VPC 110A and the second received packet to VPC 116.

[0170] Pod 302A includes one or more application containers 329A. Pod 302B includes an instance of cRPD 324. Container platform 304 includes container runtime 308, orchestration agent 310, service agent 311, and CNI 312.

[0171] The container engine 308 includes code that can be executed by the microprocessor 310. The container runtime 308 can be one or more computer processes. The container engine 308 runs containerized applications in the form of containers 329A-329B. The container engine 308 can represent Dockert, rkt, or other container engines for managing containers. Typically, the container engine 308 receives, requests, and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 310, the container engine 308 can obtain images and instantiate them as executable containers in pods 302A-302B.

[0172] The service agent 311 includes code that can be executed by the microprocessor 320. The service agent 311 can be one or more computer processes. The service agent 311 monitors the addition and removal of services and endpoint objects, and maintains the network configuration of the computing device 300 to, for example, use services to ensure communication between pods and containers. The service agent 311 can also manage iptables to capture traffic to the virtual IP address and port of the service and redirect the traffic to the proxy port of the pod supported by the agent. The service agent 311 can represent a kube agent of a subsidiary node of a Kubernetes cluster. In some examples, the container platform 204 does not include the service agent 311, or disables the service agent 311 to facilitate the CNI 312 to configure the virtual router 306A and the pod 302.

[0173] Orchestration agent 310 includes code that can be executed by microprocessor 320. Orchestration agent 310 can be one or more computer processes. Orchestration agent 310 can represent a kubelet of a node attached to a Kubernetes cluster. Orchestration agent 310 is an orchestrator (e.g., Figure 1The orchestration agent 310 is an agent of the orchestrator 23 of the orchestration engine 23 that receives container specification data of the container and ensures that the container is executed by the computing device 300. The container specification data can be in the form of a manifest file sent from the orchestrator 23 to the orchestration agent 310, or is received indirectly via a command line interface, an HTTP endpoint, or an HTTP server. The container specification data can be a pod specification (e.g., PodSpec - a YAML (Yet Another Markup Language) or JSON object that describes the pod) of a pod 302 of the container 329. Based on the container specification data, the orchestration agent 310 instructs the container engine 308 to obtain and instantiate the container image of the container 329 for the computing device 300 to execute the container 329.

[0174] Orchestration agent 310 instantiates or calls CNI 312 to configure one or more virtual network interfaces for each pod 302. For example, orchestration agent 310 receives container specification data for pod 302A and directs container engine 308 to create pod 302A with container 329A based on the container specification data for pod 302A. Orchestration agent 310 also calls CNI 312 to configure a virtual network interface for pod 302A corresponding to a virtual network of VRF 322A. In this example, pod 302A is a virtual network endpoint for a virtual network corresponding to VRF 322A.

[0175] CNI 312 may obtain interface configuration data for configuring the virtual network interface of pod 302. Virtual router agent 314 may operate as a virtual network control plane module for enabling network controller 24 to configure virtual router 306A. Unlike an orchestration control plane (including container platform 304 for satellite nodes and master nodes, e.g., orchestrator 23) that manages provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 and virtual router agent 314 for satellite nodes) manages the configuration of a virtual network implemented in part in the data plane by virtual router 306A of the satellite node. Virtual router agent 314 transmits interface configuration data of the virtual network interface to CNI 312 to enable the orchestration control plane element (i.e., CNI 312) to configure the virtual network interface according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Additionally, this may enable CNI 312 to obtain interface configuration data for multiple virtual network interfaces of a pod and configure the multiple virtual network interfaces, which may reduce the communication and resource overhead inherent in calling a separate CNI 312 to configure each virtual network interface.

[0176] Figure 5is a flow diagram illustrating an example CNR VPC technique according to one or more aspects of the present disclosure. A VPC table programmer 326 of a containerized router (such as CNR 150) may be connected to a VPC gateway 110A (400). The VPC table programmer 326 may include a process (e.g., an application) configured to program a virtual private cloud (VPC) routing table (e.g., VPTRT 118A) in a VPC gateway (e.g., VPC 110A) based on routing information to the VPC gateway received by a containerized routing protocol process (e.g., cPRD 25A) via a routing protocol. For example, for programming purposes, CNR 150 may connect to VPC gateway 110A using API 116A. CNR 150 may program VPC routing table 118A in VPC gateway 110A (402).

[0177] In some examples, CNR 150 may program VPC routing table 118A so that VPC gateway 110A directs network traffic from VPC gateway 110A to CNR 150. For example, CNR 150 may program VPC routing table 118A to indicate that the forwarding next hop is CNR 150. For example, the forwarding next hop may be a virtual IP address associated with CNR 150, a MAC address associated with CNR 150, an interface address associated with CNR 150, etc.

[0178] In some examples, CNR 150 is configured to receive network traffic from VPC Gateway 110A and forward the network traffic to Interior 130. In some examples, the routing protocol includes BGP.

[0179] In some examples, VPC gateway 110A is coupled to one or more Border Gateway Protocol (BGP) virtual appliances (e.g., BGP speakers BGPS1-BGPSN) via one or more corresponding links. In some examples, the one or more BGP virtual appliances include one or more Kubernetes pods, each pod being associated with a corresponding Kubernetes worker (e.g., worker 104 and / or 106). In some examples, each corresponding Kubernetes worker includes at least one service pod (e.g., P1 and / or P2) for services other than the BGP protocol. In some examples, the one or more corresponding links include one or more corresponding Ethernet links.

[0180] In some examples, VPC gateway 110A cannot directly learn routes from BGP speakers BGPS1-BGPSN. In some examples, VPC gateway 110A includes a physical device or a virtual device. In some examples, VPC gateway 110A includes a first VPC gateway, VPC routing table 118A includes a first VPC routing table, and CNR 150 is further configured to program a second VPC routing table (e.g., VPC routing table 118N) in a second VPC gateway (e.g., VPC gateway 110N), wherein the second VPC routing table is different from the first VPC routing table. In some examples, as part of programming VPC routing table 118A in VPC gateway 110A, CNR 150 is configured to apply policies to routes.

[0181] Figure 6 2 is a flow diagram illustrating an example redundant CNR VPC technique according to one or more aspects of the present disclosure. CNR 250B may connect to CNR 250A via VRRP session 302 (500). For example, for redundancy purposes, CNR 250A and CNR 250B may establish VRRP session 302. CNR 250B may determine, via the VRRP session, that CNR 250B is the active CNR among CNR 250A and CNR 250B (502). For example, based on the VRRP session, CNR 250B may become the active CNR, for example, through an election or by CNR 250A becoming the deactivated active CNR. CNR 250B may connect to VPC gateway 210 via API 216A (504). For example, for programmatic purposes, CNR 250B may connect to VPC gateway 210 using API 216A. It should be noted that CNR 250B may connect to VPC gateway 210 via API 216A before, during, or after CNR 250B becomes an active CNR. In some examples, based on CNR 250B being an active CNR, CNR 250B may connect to VPC gateway 210. Based on being an active CNR, CNR 250B may program (506) VPC routing table 218 in VPC gateway 210. In some examples, VRRP session 302 includes a unicast session.

[0182] In some examples, CNR 250B includes VRRP component 214B. For example, VRRP component 214B may be implemented in a processing circuit such as microprocessor 320 ( Figure 4)). In some examples, one of CNR 250A or CNR 250B is configured as an active computing device, and the other of CNR 250A or CNR 250B is configured as a standby computing device. In some examples, CNR 250B is configured to be in a Figure 1 ) shares a virtual IP address with CNR250A on the interface of one or more worker nodes 204.

[0183] In some examples, CNR 250B is configured as a standby computing device, wherein CNR 250B is further configured to determine a failure of CNR 250A via a VRRP session, and in response to the determination of the failure, assume the role of an active computing device and update VPC routing table 218 of VPC gateway 210. In some examples, as part of updating VPC routing table 218, CNR 250B is configured to update VPC routing table 218 to direct network traffic from VPC gateway 210 to CNR 250B, wherein prior to updating VPC routing table 218, VPC routing table 218 directed network traffic from VPC gateway 210 to CNR 250A. In some examples, as part of updating VPC routing table 218, CNR 250B is configured to update VPC routing table 218 to indicate that the forwarding next hop is CNR 250B.

[0184] In some examples, CNR 250B can be configured to send a gratuitous Address Resolution Protocol (ARP) message to VPC gateway 210 as part of updating VPC routing table 218. In some examples, CNR 250B is further configured to modify at least one of a BGP Local Preference attribute or a BGP MED attribute in response to assuming the role of the active computing device to direct traffic from the internal routers to the routes advertised by CNR 250B.

[0185] The technology described herein can be implemented with hardware, software, firmware, or any combination thereof. The various components, functional units, and / or modules shown in the drawings and / or shown or described elsewhere in this disclosure can perform operations described using software, hardware, firmware, or a mixture of hardware, software, and firmware that resides and / or is executed in one or more computing devices. For example, a computing device can execute one or more such modules with multiple processors or multiple devices. A computing device can execute one or more such modules as a virtual machine executed on underlying hardware. One or more such modules can be executed as one or more services of an operating system or computing platform. One or more such modules can be executed as one or more executable programs at the application layer of a computing platform. In other examples, the functions provided by the modules can be implemented by dedicated hardware devices. Although some modules, data storage devices, components, programs, executable programs, data items, functional units, and / or other items contained in one or more storage devices can be shown separately, one or more of these items can be combined and operated as a single module, component, program, executable program, data item, or functional unit. For example, one or more modules or data storage devices can be combined or partially combined so that they operate or provide functions as a single module. In addition, one or more modules can be operated in conjunction with each other so that, for example, one module serves as a service or extension of another module. In addition, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device may include multiple components, subcomponents, modules, submodules, data storage and / or other components or modules or data storage not shown. In addition, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device can be implemented in various ways. For example, each module, data storage, component, program, executable program, data item, functional unit or other project shown in the storage device can be implemented as a part of an operating system executed on a computing device.

[0186] If implemented in hardware, the present disclosure may be directed to an apparatus, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions that are executed by a processor.

[0187] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0188] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may mean that the storage medium is not contained in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0189] The code or instructions may be software and / or firmware executed by a processing circuit that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the present disclosure may be provided in software modules or hardware modules.

Claims

1. A computing device, configured with a first containerized router, the computing device comprising: Processing circuit; a containerized virtual router configured to execute on the processing circuit, and the containerized virtual router is configured to implement a data plane for the first containerized router; as well as a containerized routing protocol process configured to execute on the processing circuit, and the containerized routing protocol process configured to implement a control plane for the first containerized router, wherein the containerized routing protocol process is configured to execute a routing protocol, wherein the first containerized router is configured to program a virtual private cloud (VPC) routing table in a VPC gateway based on routing information received by the containerized routing protocol process via the routing protocol, and wherein the first containerized router is communicatively coupled to the second containerized router via a virtual router redundancy protocol (VRRP) session.

2. The computing device of claim 1, wherein: The VRRP session includes a unicast session.

3. The computing device of claim 1, further comprising a VRRP component configured to execute on the processing circuit.

4. The computing device according to any one of claims 1 to 3, wherein: One of the first containerized router and the second containerized router is configured as an active containerized router, and the other of the first containerized router and the second containerized router is configured as a standby containerized router for multi-addressing with a VPC gateway.

5. The computing device of claim 4, wherein: The first containerized router is configured to share a virtual IP address with the second containerized router on an interface facing one or more worker nodes of a data center.

6. The computing device of claim 4, wherein: The first containerized router is configured as the backup containerized router, wherein the first containerized router is further configured to: determining, via the VRRP session, a failure of the second containerized router; and In response to the determination of the failure, assuming a role of the active containerized router and updating the VPC routing table of the VPC gateway via an application programming interface (API).

7. The computing device of claim 6, wherein: As part of updating the VPC routing table, the process configured to program the VPC routing table is configured to update the VPC routing table to direct network traffic from the VPC gateway to the first containerized router, wherein, prior to updating the VPC routing table, the VPC routing table directs the network traffic from the VPC gateway to the second containerized router.

8. The computing device of claim 7, wherein: As part of updating the VPC routing table, the process configured to program the VPC routing table is configured to update the VPC routing table to indicate that the forwarding next hop is the first containerized router.

9. The computing device of claim 6, wherein: As part of updating the VPC routing table, a process configured to program the VPC routing table is configured to send a gratuitous Address Resolution Protocol (ARP) message to the VPC gateway.

10. The computing device of claim 6, wherein: The first containerized router is further configured to, in response to assuming the role of the active containerized router, modify at least one of a Border Gateway Protocol (BGP) local preference attribute and a BGP multi-exit discriminator (MED) attribute to direct traffic from an internal router to a route advertised by the first containerized router.

11. A computer networking method, comprising: connecting, through the first containerized router, to the second containerized router via a Virtual Router Redundancy Protocol (VRRP) session; determining, via the VRRP session, that the first containerized router is an active containerized router among the first containerized router and the second containerized router; Connect to a Virtual Private Cloud (VPC) gateway; as well as The VPC routing table in the VPC gateway is programmed, by a process configured to program the VPC routing table and based on the first containerized router being the active containerized router.

12. The method according to claim 11, wherein: The VRRP session includes a unicast session.

13. The method according to claim 11, wherein: The second containerized router is configured as a backup containerized router.

14. The method of claim 11, further comprising sharing, through the first containerized router, a virtual IP address with the second containerized router on an interface facing one or more worker nodes in a data center.

15. The method according to any one of claims 11 to 14, wherein: Determining that the first containerized router is the active containerized router includes: determining, by the first containerized router via the VRRP session, a failure of the second containerized router; and In response to the determination of the failure, the first containerized router assumes a role of the active containerized router, and the first containerized router updates the VPC routing table of the VPC gateway via an application programming interface (API).

16. The method according to claim 15, wherein: Updating the VPC routing table includes updating the VPC routing table, by a process configured to program the VPC routing table, to direct network traffic from the VPC gateway to the first containerized router, wherein, prior to updating the VPC routing table, the VPC routing table directs the network traffic from the VPC gateway to the second containerized router.

17. The method according to claim 16, wherein: Updating the VPC routing table includes updating the VPC routing table to indicate that the forwarding next hop is the first containerized router.

18. The method according to claim 15, wherein: Updating the VPC routing table includes sending a gratuitous Address Resolution Protocol (ARP) message to the VPC gateway.

19. The method of claim 15, further comprising, in response to assuming the role of the active containerized router, modifying at least one of a Border Gateway Protocol (BGP) local preference attribute and a BGP multi-exit discriminator (MED) attribute to direct traffic from an internal router to a route advertised by the first containerized router.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method of any one of claims 11 to 19 or to be configured as a computing device of any one of claims 1 to 10.

Images