Network controller for software defined network (SDN) architecture system

Abstract

The present disclosure relates to a network controller for an SDN architecture system, comprising: processing circuitry of a central cluster of first one or more computing nodes; a configuration node configured to be executed by the processing circuitry; and a control node configured to be executed by the processing circuit. The configuration node includes a custom API server to process a request for operation of custom resources of the SDN architecture configuration. Each custom resource configured by the SDN architecture corresponds to the type of the configuration object in the SDN architecture system. In response to detection of an event of an instance of a first one of the custom resources, the control node obtains configuration data regarding the instance of the first custom resource and configures a corresponding instance of a configuration object in a workload cluster of the second one or more computing nodes. The first compute node may be different from the second compute node.

Description

[0001] This application is a divisional application of the Chinese patent application with the application date of June 30, 2022, application number 202210760750.3, and invention name “Cloud-native software-defined network architecture for multiple clusters”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Patent Application No. 17 / 657,603, filed on March 31, 2022, which claims the benefit of Indian Provisional Patent Application No. 202141044924, filed on October 4, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to a virtualized computing infrastructure, and more particularly, to a cloud native networking. Background Art

[0005] In a typical cloud data center environment, there is a large collection of interconnected servers that provide the computing and / or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More sophisticated data centers provide user support equipment located in various physical hosting facilities for infrastructure located throughout the world.

[0006] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have a widely used virtualization environment in which virtual hosts such as virtual machines or containers, also referred to herein as virtual runtime elements, are deployed and run on the underlying computing platforms of physical computing devices.

[0007] Virtualization within a data center, or any environment comprising one or more servers, can provide several advantages. One advantage is that virtualization can provide significantly improved efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. A second advantage is that virtualization provides effective control over the computing infrastructure. As physical computing resources, such as in cloud-based computing environments, become fungible resources, the provision and management of computing infrastructure becomes easier. As a result, enterprise IT staff often prefer virtualized computing clusters in data centers due to their management advantages, in addition to the efficiency and increased return on investment (ROI) that virtualization provides.

[0008] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight, portable runtime elements for applications that are isolated from each other and the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bundled into a container image and run as a single, lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of making software work across diverse computing environments. Containers guarantee consistent operation from one computing environment to another, virtual or physical.

[0009] Due to the inherent lightweight nature of containers, a single host can typically support more container instances than traditional virtual machines (VMs). Short-lived containers can generally be created and moved more efficiently than VMs, and containers can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the demand for container networking solutions: the network should be agile and scalable. In the same computing environment, VMs, containers, and bare metal servers may need to coexist and enable communication between multiple deployed applications. Container networking should also not be known to work with multiple types of orchestration platforms used to deploy containerized applications.

[0010] Managing the deployment and infrastructure of computing infrastructure for application execution may involve two primary roles: (1) orchestration—automating the deployment, scaling, and operation of applications across a cluster of hosts and providing computing infrastructure, which may include container-centric computing infrastructure; and (2) network management—creating virtual networks within the network infrastructure to enable packetized communication between applications running in virtual execution environments such as containers or VMs and between applications running in legacy (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0011] Overall, technology is described for a cloud-native SDN architecture deployed using multiple clusters. In some embodiments, the SDN architecture may include data plane elements implemented in compute nodes and network devices such as routers or switches, and the SDN architecture may also include a network controller for creating and managing virtual networks. The SDN architecture configuration and control plane is designed as an extended cloud-native software with a container-based microservice architecture that supports in-service upgrades. The configuration nodes for the configuration plane can be implemented to expose custom resources. These custom resources for SDN architecture configuration may include configuration elements that are typically exposed by the network controller, but the configuration elements and Kubernetes native / built-in resources may be merged together to support a unified intent model exposed by an aggregate API layer and implemented by Kubernetes controllers and custom resource controllers that work together to reconcile the actual state and the expected state of the SDN architecture.

[0012] In a multi-cluster SDN architecture deployment, configuration nodes and control nodes are deployed to a central cluster and centrally manage the configuration and control of one or more workload clusters. However, the data plane is distributed across the workload clusters. Each workload cluster and the central cluster can use similar component microservices to implement the cluster's data plane. A dedicated SDN controller manager running on the central cluster for each workload cluster creates custom resources related to the SDN architecture configuration in the central cluster, which are managed by the configuration node and configured by the control node in the corresponding workload cluster.

[0013] The present technology may provide one or more technical advantages. For example, using an SDN controller manager running on a central cluster and consolidating configuration nodes and control nodes into a single central cluster facilitates better lifecycle management (LCM) of the SDN controller manager, configuration nodes, and control nodes, and facilitates better and more manageable handling of security and permissions by consolidating these tasks into a single central cluster.

[0014] As another example, a cloud-native SDN architecture can address limitations in conventional SDN architectures related to the complexity of lifecycle management, the high mandatory nature of resource analysis components, the scalability limitations of configuration management, and the lack of command line interface (CLI)-based interfaces. For example, the network controller for the SDN architecture is a cloud-native, lightweight distributed application with a simplified installation overlay. This also facilitates easy and modular upgrades of the various component microservices of the configuration node and control node of the configuration and control plane. The technology can further support optional cloud-native monitoring (telemetry) and user interfaces, a high-performance data plane for containers connected to DPDK-enabled container pools (pods) using DPDK-based virtual routers, and cloud-native configuration management that leverages the configuration framework of existing orchestration platforms such as Kubernetes or Openstack in some cases. As a cloud-native architecture, the network controller is scalable and elastic to address and support multiple clusters. In some cases, the network controller can also support the scalability and performance requirements of key performance indicators (KPIs).

[0015] In an embodiment, a network controller for a software-defined network (SDN) architecture system is provided, the network controller comprising: processing circuitry of a first central cluster of one or more compute nodes; a configuration node configured to be executed by the processing circuitry; and a control node configured to be executed by the processing circuitry; wherein the configuration node comprises a custom application programming interface (API) server to process requests for operations on custom resources configured with respect to the SDN architecture; wherein each custom resource configured with respect to the SDN architecture corresponds to a type of configuration object in the SDN architecture system; and wherein, in response to detection of an event for an instance of a first custom resource among the custom resources, the control node is configured to obtain configuration data for the instance of the first custom resource and configure a corresponding instance of the configuration object in a workload cluster of a second one or more compute nodes, wherein the first one or more compute nodes in the central cluster are distinct from the second one or more compute nodes in the workload cluster.

[0016] In an embodiment, the method includes: a custom application programming interface (API) server implemented by a configuration node of a network controller for a software-defined network (SDN) architecture system, processing a request for an operation on a custom resource configured with respect to the SDN architecture, wherein each custom resource configured with respect to the SDN architecture corresponds to a type of configuration object in the SDN architecture system, wherein the network controller operates a first central cluster of one or more computing nodes; detecting, by a control node of the network controller, an event regarding an instance of a first custom resource among the custom resources; and in response to detecting the event regarding the instance of the first custom resource, obtaining, by the control node, configuration data regarding the instance of the first custom resource and configuring a corresponding instance of the configuration object in a workload cluster of a second one or more computing nodes, wherein the first one or more computing nodes of the central cluster are distinct from the second one or more computing nodes of the workload cluster.

[0017] In an embodiment, a non-transitory computer-readable medium includes instructions for causing a processing circuit to: process a request for an operation on a custom resource configured for the SDN architecture by a configuration node of a network controller for a software-defined network (SDN) architecture system, wherein each custom resource configured for the SDN architecture corresponds to a type of configuration object in the SDN architecture system, wherein the network controller operates a first central cluster of one or more computing nodes; detect, by a control node of the network controller, an event on an instance of a first custom resource among the custom resources; and in response to detecting the event on the instance of the first custom resource, obtain, by the control node, configuration data on the instance of the first custom resource and configure a corresponding instance of the configuration object in a workload cluster of a second one or more computing nodes, wherein the first one or more computing nodes of the central cluster are distinct from the second one or more computing nodes of the workload cluster.

[0018] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram illustrating an exemplary computing infrastructure in which embodiments of the techniques described herein may be implemented.

[0020] Figure 2 is a block diagram illustrating an embodiment of a cloud-native SDN architecture for cloud-native networks according to the techniques of this disclosure.

[0021] Figure 3is a block diagram illustrating another view of components of SDN architecture 200 in further detail, in accordance with techniques of this disclosure.

[0022] Figure 4 is a block diagram illustrating exemplary components of an SDN architecture in accordance with techniques of this disclosure.

[0023] Figure 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure.

[0024] Figure 6 is a block diagram of an exemplary computing device operating as a computing node for one or more clusters of an SDN architecture system in accordance with the techniques of this disclosure.

[0025] Figure 7A is a block diagram illustrating a control / routing plane for underlay and overlay network configuration using an SDN architecture, according to techniques of this disclosure.

[0026] Figure 7B is a block diagram illustrating a configured virtual network connecting a pool of containers using tunnels configured in an underlying network, according to the techniques of this disclosure.

[0027] Figure 8 is a block diagram illustrating an example of a custom controller for custom resources configured for an SDN architecture according to the techniques of this disclosure.

[0028] Figure 9 is a block diagram illustrating an exemplary flow of creating, monitoring, and coordinating between custom resource types that depend on different custom resource types.

[0029] Figure 10 is a block diagram illustrating a multi-cluster deployment for a cloud-native SDN architecture according to techniques of this disclosure.

[0030] Figure 11 is a flow chart illustrating an exemplary mode of operation for a multi-cluster deployment of an SDN architecture.

[0031] Like reference characters denote like elements throughout the description and drawings. DETAILED DESCRIPTION

[0032] Figure 1is a block diagram illustrating an exemplary computing infrastructure 8 in which the techniques described herein may be implemented. Current implementations of software-defined networking (SDN) architectures for virtual networks present challenges for cloud-native adoption due to, for example, the complexity of lifecycle management, the high level of mandatory resource analysis components, limited scalability of configuration modules, and the absence of command-line interface (CLI)-based interfaces (e.g., kubectl). The computing infrastructure 8 includes the cloud-native SDN architecture system described herein, which addresses these challenges and modernizes the telecom cloud-native era. Exemplary use cases for the cloud-native SDN architecture include 5G mobile networks and cloud and enterprise cloud-native use cases. The SDN architecture may include data plane elements implemented in compute nodes (e.g., servers 12) and network devices such as routers or switches, and may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. The SDN architecture configuration and control plane are designed as scale-out cloud-native software with a container-based microservices architecture that supports live upgrades.

[0033] Therefore, SDN architecture components are microservices, and unlike existing network controllers, SDN architecture assumes an underlying container orchestration platform that manages the lifecycle of SDN architecture components. The container orchestration platform is used to build SDN architecture components; the SDN architecture uses cloud-native monitoring tools that can be integrated with cloud-native options provided by customers; the SDN architecture uses an aggregated API of SDN architecture objects to provide a declarative way of declaring resources (i.e., custom resources). SDN architecture upgrades can follow a cloud-native model, and the SDN architecture can leverage Kubernetes constructs such as Multus, authentication & authorization, cluster API, KubeFederation, KubeVirt, and Kata containers. The SDN architecture can support a data plane development kit (DPDK) container pool, and the SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies.

[0034] For service providers and enterprises, the SDN architecture automates network resource configuration and orchestration to dynamically create highly scalable virtual networks and link virtualized network functions (VNFs) with physical physical functions (PNFs) to form differentiated service chains on demand. The SDN architecture can be integrated with orchestration platforms (e.g., orchestrators 23) such as Kubernetes, OpenShift, Mesos, OpenStack, and VMware vSphere, and can be integrated with service provider operations support systems / business support systems (OSS / BSS).

[0035] Typically, one or more data centers 10 provide an operating environment for applications and services at customer sites 11 (illustrated as "customer 11"), which have one or more customer networks coupled to the data centers via service provider networks 7. For example, each data center 10 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. Service provider networks 7 are coupled to public networks 15, which may represent one or more networks managed by other providers and, thus, may form part of a larger public network infrastructure (e.g., the Internet). For example, public networks 15 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating service provider networks 7, an enterprise IP network, or some combination thereof.

[0036] Although customer sites 11 and public network 15 are primarily shown and described as edge networks of service provider network 7, in some embodiments, customer sites 11 and one or more of public network 15 may be tenant networks within any data center 10. For example, data center 10 may host multiple tenants (customers), each associated with one or more virtual private networks (VPNs), where each tenant may implement one or more customer sites 11.

[0037] The service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement Multi-Protocol Label Switching (MPLS) forwarding and, in this instance, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0038] In some embodiments, each data center 10 may represent one of a plurality of geographically distributed network data centers that may be connected to each other via a service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1As shown in the example of , data center 10 may include facilities that provide network services to customers. Customers of the service provider may be collective entities such as businesses and governments or individuals. For example, a network data center may host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although shown as a discrete edge network of the service provider network 7, elements in the data center 10, such as one or more physical network functions (PNFs) or virtualized network functions (VNFs), may be included in the core of the service provider network 7.

[0039] In this example, data center 10 includes storage and / or compute servers (or "nodes") interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, and servers 12A-12X (referred to herein as "servers 12") are depicted as coupled to top-of-rack TOR switches 16A-16N. Servers 12 are computing devices and may also be referred to herein as "compute nodes," "hosts," or "host devices." Although in Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail; however, data center 10 may include multiple additional servers coupled to other TOR switches 16 in data center 10.

[0040] In the illustrated example, the switch fabric 14 includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") coupled to a distribution layer of chassis (or "spine" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0041] In this example, the TOR switches 16 and the chassis switches 18 provide redundant (multi-homed) connections to the servers 12 to the IP fabric 20 and the service provider network 7. The chassis switches 18 aggregate traffic flows and provide connections between the TOR switches 16. The TOR switches 16 can be network devices that provide Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functionality. The TOR switches 16 and the chassis switches 18 can each include one or more processors and memory and be capable of running one or more software processes. The chassis switches 18 are coupled to the IP fabric 20, which can perform Layer 3 routing to route network traffic between the data center 10 and the customer sites 11 through the service provider network 7. The switching architecture of the data center 10 is merely an example. For example, other switching architectures can have more or fewer switching layers. The IP fabric 20 can include one or more gateway routers.

[0042] The term "packet flow," "traffic flow," or simply "flow" refers to a collection of packets originating from a specific source device or endpoint and being sent to a specific destination device or endpoint. For example, a single flow of packets can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, this 5-tuple identifies the packet flow corresponding to the received packets. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to the combination of <source network address, destination network address> or <source network address, source port> for that packet.

[0043] Each server 12 may represent a computing server or a storage server. For example, each server 12 may represent a computing device configured to operate according to the techniques described herein, such as an x86 processor-based server. Servers 12 may provide a network function virtualization infrastructure (NFVI) for an NFV architecture.

[0044] Any of the servers 12 can be configured with a virtual execution element, such as a container pool or a virtual machine, by virtualizing the server's resources to provide a degree of isolation between one or more processes (applications) running on the server. "Hypervisor-based," or "hardware-level," or "platform" virtualization refers to the creation of virtual machines that each include a guest operating system for running one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is visualized from the physical hardware of the host server, the executing application is isolated from the host's hardware and other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0045] A virtual network is a logical construct implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center, such as data center 10. Each tenant or application can have one or more virtual networks. Unless explicitly allowed by security policy, each virtual network must be isolated from all other virtual networks.

[0046] Virtual networks can use the Data Center 10 Gateway Router ( Figure 1 The virtual network (not shown) is connected to and extended over physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Network (L3VPN) and Ethernet Virtual Private Network (EVPN) networks. Virtual networks can also be used to implement Network Function Virtualization (NFV) and service chaining.

[0047] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks: a physical underlay network consisting of the IP fabric 20 and the switch fabric 14, and a virtual overlay network. The role of the physical underlay network is to provide the "IP fabric" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network can provide a uniform low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0048] As further described below with respect to virtual router 21 (illustrated as and also referred to herein as "vRouter 21"), the virtual router running in server 12 creates a virtual overlay network on top of the physical underlying network using its own dynamic mesh of "tunnels." For example, these overlay tunnels can be MPLS over GRE / UDP tunnels, or VXLAN tunnels, or NVGRE tunnels. For virtual machines or other virtual execution elements, the underlying physical routers and switches may not store arbitrary state for each tenant, such as arbitrary media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may only contain the IP prefix or MAC address of the physical server 12. (The gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0049] The virtual router 21 of a server 12 typically contains per-tenant state. For example, a virtual router may contain a separate forwarding table (routing instance) for each virtual network. This forwarding table contains the IP prefixes (in the case of a Layer 3 overlay) or MAC addresses (in the case of a Layer 2 overlay) of virtual machines or other virtual execution elements (e.g., a container pool of containers). A single virtual router 21 does not need to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that are local to the server 12 (i.e., have at least one virtual execution element present on the server 12).

[0050] “Container-based” or “operating system” virtualization refers to the virtualization of an operating system running multiple isolated systems on a single machine (virtual or physical). The isolated systems represent containers such as those provided by the open source DOCKER container application or by CoreOS Rkt (“Rocket”). Like a virtual machine, each container is virtualized and can be kept isolated from the host machine and other containers. However, unlike a virtual machine, each container can omit a separate operating system and instead provide a suite of applications and application-specific libraries. Typically, containers are run by a host machine as isolated user-space instances and the containers can share an operating system and common libraries with other containers running on the host machine. As a result, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on a corresponding virtual network.

[0051] In some examples, containers are managed by their host kernel to allow for resource restriction and prioritization (CPU, memory, block I / O, network, etc.) in some cases without launching arbitrary virtual machines using namespace isolation capabilities, which allow applications (e.g., a given container) to view an operating environment completely isolated, including process trees, networking, user identifiers, and mounted file systems. In some embodiments, containers can be deployed according to Linux Containers (LXC), an operating system-level virtualization method that uses a single Linux kernel to run multiple isolated Linux systems (containers) on a control host.

[0052] Servers 12 host virtual network endpoints of one or more virtual networks operating on a physical network represented here by IP fabric 20 and switch fabric 14. Although primarily described with respect to a data center-based switched network, other physical networks such as service provider network 7 may underlie one or more virtual networks.

[0053] Each server 12 can host one or more virtual execution elements, each of which has at least one virtual network endpoint of one or more virtual networks configured in the physical network. The virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, one or more container sets (e.g., a container pool), or another virtual execution element such as a layer 3 endpoint of a virtual network. The term "virtual execution element" covers virtual machines, containers, and other virtualized computing resources that provide at least partially independent operating environments for applications. The term "virtual execution element" can also cover a container pool of one or more containers. A virtual execution element can represent an application workload. As Figure 1 As shown in FIG, server 12A hosts a virtual network endpoint in the form of a container pool 22 having one or more containers. However, in practice, given the hardware resource limitations of server 12, server 12 may execute as multiple virtual execution elements. Each virtual network endpoint may use one or more virtual network interfaces to perform packet I / O or other processing on packets. For example, a virtual network endpoint may use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0054] Each server 12 includes at least one network interface card (NIC) 13, each NIC 13 including at least one interface for exchanging packets with a TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component of I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) as described by the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to provide one or more virtual network interfaces as "virtual functions" for use by corresponding endpoints running on the server 12. In this way, the virtual network endpoints can share the same PCIe physical hardware resources and the virtual functions are examples of virtual hardware components 21. As another example, one or more servers 12 can implement Virtio, a paravirtualization framework available, for example, in the Linux operating system, which provides emulated NIC functions as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement Open vSwitch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) of hosted virtual machines, where the vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function. In some examples, any of the servers 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For Docker implementations of containers hosted by the servers, a Linux bridge or other operating system bridge running on the server that switches packets between containers may be referred to as a "Docker bridge." As used herein, the term “virtual router” may encompass a Contrail or Tungsten Fabric virtual router, Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other device and / or software that resides on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12.

[0055] Any NIC 13 may include an internal device switch that switches data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch may be a virtual Ethernet bridge (VEB) that switches between SR-IOV virtual functions and, correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. Alternatively, the internal device switch may be referred to as a NIC switch, or, for an SR-IOV implementation, an SR-IOV NIC switch. The virtual hardware components associated with the NIC 13A may be associated with a layer 2 destination address assigned by the NIC 13A or a software process responsible for configuring the NIC 13A. Physical hardware components (or "physical functions" for an SR-IOV implementation) are also associated with a layer 2 destination address.

[0056] One or more servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) of an Internet Protocol-Virtual Private Network (IP-VPN). For example, a packet received by the virtual router 21 of server 12A from the underlying physical network fabric of the data center 10 (i.e., the IP fabric 20 and the switch fabric 14) may include an outer header to allow the physical network fabric to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the server's network interface card 13A, but also a virtual network identifier such as a VxLAN tag or a Multiprotocol Label Switching (MPLS) label that identifies a virtual network and the corresponding routing instance executed by the virtual router 21. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0057] The virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the packet's tunnel encapsulation header, and forwards the packet to the appropriate destination virtual network endpoint for the packet. For example, with respect to server 12A, for each packet outbound from a virtual network endpoint (e.g., container pool 22) hosted by server 12A, the virtual router 21 appends a tunnel encapsulation header indicating the virtual network to the packet to generate an encapsulated or "tunnel" packet, and the virtual router 21 outputs the encapsulated packet to a physical destination computing device, such as another server 12, via the virtual network's overlay tunnel. As used herein, the virtual router 21 can perform the operations of a tunnel endpoint to encapsulate internal packets originating from a virtual network endpoint to generate tunnel packets, and decapsulate tunnel packets to obtain internal packets for routing to other virtual network endpoints.

[0058] In some examples, virtual router 21 may be kernel-based and may run as part of the kernel of the operating system of server 12A.

[0059] In some examples, virtual router 21 can be a virtual router that supports the Data Plane Development Kit (DPDK). In this example, virtual router 21 uses DPDK as the data plane. In this mode, virtual router 21 runs as a user space application linked to the DPDK library (not shown). This is the execution version of the virtual router and is commonly used by telecommunications companies, where VNFs are typically DPDK-based applications. The implementation of virtual router 21 as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the DPDK's poll mode driver (PMD) rather than the Linux kernel's interrupt-based driver.

[0060] A User-I / O (UIO) kernel module such as vfio or uio_pci_generic can be used to expose the registers of the physical network interface to user space so that the DPDK PMD can access the registers. When the NIC 13A is bound to the UIO driver, the NIC 13A is moved from the Linux kernel space to the user space and is therefore no longer managed or visible to the Linux OS. Therefore, the DPDK application (i.e., in this example, the virtual router 21A) fully manages the NIC 13. This includes packet polling, packet processing, and packet forwarding. The user packet processing steps can be performed by the virtual router 21DPDK data plane, and the kernel (in Figure 1The kernel (not shown) has limited or no involvement. This "polling mode" nature makes virtual router 21 DPDK data plane packet processing / forwarding more efficient than interrupt mode, particularly at higher packet rates. Limited or no interrupts occur during packet I / O, and context switches are present.

[0061] Additional details on the DPDK vRouter example can be found in “DAY ONE: CONTRAIL DPDK vROUTER,” Kiran KN et al., Juniper Networks, 2021, which is incorporated herein by reference in its entirety.

[0062] The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration system that provides a container-centric infrastructure to automate the deployment, scaling, and operation of containers, thereby providing a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services running on the virtual execution elements by the orchestration platform. Specifically, for example, container orchestration permits container coordination and refers to the deployment, management, scaling, and configuration of containers to host servers by the container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0063] The automation platform components of computing infrastructure 8 include at least servers 12, orchestrator 23, and network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework, in which a cluster master manages the deployment and operation of containers to one or more cluster slaves. The terms "master" and "slave" used herein encompass different orchestration platform terminology, distinguishing between the cluster's primary management element and a simulated device for the cluster's primary container hosting device. For example, the Kubernetes platform uses the terms "cluster master" and "slave," while the Docker Swarm platform refers to both the cluster manager and cluster nodes.

[0064] Orchestrator 23 and network controller 24 can run on separate computing devices or on the same computing device. Each of orchestrator 23 and network controller 24 can be a distributed application running on one or more computing devices. Orchestrator 23 and network controller 24 can implement respective master nodes of one or more clusters, each of which has one or more slave nodes (also referred to as "compute nodes") implemented by respective servers 12.

[0065] In general, for example, network controller 24 controls the network configuration of the fabric of data center 10 to establish one or more virtual networks for packetized communications between virtual network endpoints. Network controller 24 provides a logically, and in some cases, physically, centralized controller that facilitates the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or administrators / operators. Additional information regarding exemplary operation of network controller 24 operating in conjunction with other devices of data center 10 or other software-defined networks can be found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS,” and in U.S. Patent Application No. 14 / 226,509, filed on May 26, 2014, and entitled “Tunneled Packet Aggregation for Virtual Networks,” each of which is incorporated by reference as fully set forth herein.

[0066] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides computing infrastructure, which may include container-centric computing infrastructure. The orchestrator 23, and in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud may provide a virtualized infrastructure to the container management platform. Figure 3 Describe the example components of the Kubernetes orchestration system.

[0067] Kubernetes operates using various Kubernetes objects—entities—that represent the state of a Kubernetes cluster. Kubernetes objects can include any combination of names, namespaces, labels, annotations, field selectors, and recommended labels. For example, a Kubernetes cluster can include one or more "namespace" objects. Each namespace in a Kubernetes cluster is isolated from other namespaces in the Kubernetes cluster. Namespace objects can include at least one of the organization, security, and performance of the Kubernetes cluster. As an example, a container pool can be associated with a namespace, thereby associating the container pool with a characteristic of the namespace (e.g., a virtual network). This characteristic enables organizing multiple newly created container pools by associating the container pools with a common set of characteristics. Namespaces can be created based on namespace specification data that defines the characteristics of the namespace (including the namespace name). In one example, a namespace may be named "Namespace A," and each newly created container pool can be associated with the feature set represented by "Namespace A." In addition, Kubernetes includes a "default" namespace. If a newly created container pool does not specify a namespace, the newly created container pool can be associated with the characteristics of the "default" namespace.

[0068] Namespaces enable multiple users, user teams, or a single user with multiple applications to utilize a single Kubernetes cluster. Furthermore, each user, user team, or application can be isolated from every other user of the cluster within a namespace. Therefore, each user of a Kubernetes cluster within a namespace operates as if they were the only user of the Kubernetes cluster. Multiple virtual networks can be associated with a single namespace. Therefore, a container pool belonging to a specific namespace has access to every virtual network in the virtual network associated with the namespace, including other container pools that serve as virtual network endpoints within that group of virtual networks.

[0069] In one example, the container pool 22 is a Kubernetes container pool and is an example of a virtual network endpoint. A container pool is a group of one or more logically related containers ( Figure 1), shared memory for containers, and options for how to run containers. Alternatively, if instantiated for execution, a container pool can be referred to as a "container pool replica." Each container in a container pool 22 is an example of a virtual execution element. Containers of a container pool are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of a container pool can be a collection of Linux namespaces, cgroups, and other isolation aspects. Within the context of a container pool, individual applications may further apply sub-isolation. Typically, containers within a container pool have a common IP address and port space and can detect each other via the local host. Because they have a shared context, containers within a container pool also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different container pools have different IP addresses, and in the absence of configuration to enable this feature, containers cannot communicate via IPC. Instead, containers that are members of different container pools typically communicate with each other via the container pool IP address.

[0070] Server 12A includes a container platform 19 for running containerized applications such as those in container pool 22. Container platform 19 receives requests from orchestrator 23 to obtain containers and causes the containers to be hosted in server 12A. Container platform 19 obtains and runs the containers.

[0071] The container network interface (CNI) 17 configures virtual network interfaces for virtual network endpoints. The orchestrator 23 and the container platform 19 use CNI 17 to manage the networks of container pools (including container pool 22). For example, CNI 17 creates a virtual network interface that connects the container pool to the virtual router 21 and enables containers in the container pool to communicate with other virtual network endpoints on the virtual network via the virtual network interface. For example, CNI 17 may insert a virtual network interface for a virtual network into the network namespace of a container in container pool 22 and configure (or request configuration) a virtual network interface for the virtual network in virtual router 21, so that virtual router 21 is configured to send packets received from the virtual network via the virtual network interface to containers in container pool 22 and to send packets received from containers in container pool 22 via the virtual network interface on the virtual network. CNI 17 may allocate network addresses (e.g., virtual IP addresses for the virtual networks) and set up routes for the virtual network interfaces. In Kubernetes, by default, all container pools can communicate with all other container pools without using network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a container pool virtual network shared by all namespaces, and allocate service and container pool network addresses from the namespaces, respectively. In some cases, all container pools generated in the Kubernetes cluster across all namespaces can communicate with each other, and their network addresses can be allocated from the container pool subnet specified by the orchestrator 23. When a user creates an isolated namespace for a container pool, the orchestrator 23 and the network controller 24 can create a new container pool virtual network and a new shared service virtual network for the new isolated namespace. Container pools generated in the Kubernetes cluster in the isolated namespace obtain network addresses from the new container pool virtual network, and the corresponding services of the container pool obtain network addresses from the new service virtual network.

[0072] CNI 17 may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17 may conform, at least in part, to the Container Networking Interface (CNI) specification or the rkt networking proposal. CNI 17 may represent Contrail, OpenContrail, Multus, Calico, cRPD, or another CNI. Alternatively, CNI 17 may be referred to as a network plugin, CNI plugin, or CNI instance. For example, a separate CNI may be called by the Multus CNI to establish different virtual network interfaces for container pool 22.

[0073] The CNI 17 may be called by the orchestrator 23. For the purposes of the CNI specification, containers may be considered to be synchronized with a Linux network namespace. The unit to which this corresponds depends on the specific container runtime implementation: for example, in an implementation of the application container specification such as rkt, each pool of containers runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and capable of communicating with each other. This may be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers may be conceptually added to or removed from one or more networks. The CNI specification specifies a number of considerations for conformant plugins ("CNI plugins").

[0074] Container pool 22 includes one or more containers. In some examples, container pool 22 includes a containerized DPDK workload designed to use DPDK to accelerate packet processing, for example, by exchanging data with other components using DPDK libraries. In some examples, virtual router 21 can run as a containerized DPDK workload.

[0075] Container pool 22 is configured with a virtual network interface 26 for sending and receiving packets through virtual router 21. Virtual network interface 26 may be the default interface of container pool 22. Container pool 22 may implement virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while virtual router 21 may implement virtual network interface 26 as a tap interface, a virtio user interface, or another type of interface.

[0076] Container pool 22 and virtual router 21 exchange data packets using virtual network interface 26. Virtual network interface 26 can be a DPDK interface. Container pool 22 and virtual router 21 can use vhost to set up virtual network interface 26. Container pool 22 can operate according to the aggregation model. Container pool 22 can use virtual devices, such as virtio devices with vhost-user adapters, for inter-userspace container process communication on virtual network interface 26.

[0077] CNI 17 can be combined with Figure 1 One or more other components shown in FIG configure a virtual network interface 26 for the container pool 22. Any container in the container pool 22 can utilize (ie, share) the virtual network interface 26 of the container pool 22.

[0078] Virtual network interface 26 may represent a virtual Ethernet ("veth") pair, where each end of the pair is a separate device (e.g., a Linux / Unix device), and one end of the pair is assigned to container pool 22 and one end of the pair is assigned to virtual router 21. A veth pair or one end of a veth pair is sometimes referred to as a "port." The virtual network interface may represent a macvlan network with media access control (MAC) addresses assigned to container pool 22 and virtual router 21 for communication between containers of container pool 22 and virtual router 21. Alternatively, for example, the virtual network interface may be referred to as a virtual machine interface (VMI), a container pool interface, a container network interface, a tap interface, a veth interface, or simply a network interface (in the specific context).

[0079] exist Figure 1 In the exemplary server 12A, container pool 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment, which specifies the virtual network and specifies that container pool 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. For example, orchestrator 23 may receive the configuration data from a user, an operator / administrator, or another machine system.

[0080] As part of the process of creating a container pool 22, the orchestrator 23 requests the network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). A container pool 22 may have a different virtual network interface for each virtual network to which it belongs. For example, virtual network interface 26 may be a virtual network interface for a specific virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks. The network controller 24 processes the request to generate interface configuration data for the virtual network interfaces of the container pool 22. The interface configuration data may include a unique identifier for the container or container pool and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0081] Network controller 24 sends the interface configuration data to server 12A, and more specifically, in some cases, to virtual router 21. To configure the virtual network interface for container pool 22, orchestrator 23 may invoke CNI 17. CNI 17 obtains the interface configuration data from virtual router 21 and processes it. CNI 17 creates the virtual network interfaces specified in the interface configuration data. For example, CNI 17 may attach one end of a veth pair implementing management interface 26 to virtual router 21 and the other end of the same veth pair implementing management interface 26 using a virtio user to container pool 22.

[0082] The following is exemplary interface configuration data for virtual network interface 26 of container pool 22 .

[0083]

[0084]

[0085] A conventional CNI plugin is invoked by a container platform / runtime, receives an Add command from the container platform to add a container to a single virtual network, and can subsequently be invoked to receive a Del command from the container / runtime and remove the container from the virtual network. The term "invocation" can refer to the instantiation of a software component or module in memory as executable code for execution by processing circuitry.

[0086] According to the techniques described in this disclosure, network controller 24 is a cloud-native distributed network controller for software-defined networking (SDN) implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself may be implemented using one or more cloud-native component microservices. Each control node 32 itself may be implemented using one or more cloud-native component microservices.

[0087] In some examples, and as described in further detail below, the configuration node 30 can be implemented by extending the local orchestration platform to support custom resources for the orchestration platform for software-defined networking, and more specifically, by providing a northbound interface to the orchestration platform to support purpose-driven / declarative creation and management of virtual networks, for example, by configuring virtual network interfaces of virtual execution elements, configuring the underlying network connection server 12, and configuring overlay routing functions including overlay tunnels for virtual networks and overlay trees for multicast Layer 2 and Layer 3.

[0088] As Figure 1As part of the SDN architecture shown in

[15] , the network controller 24 can be multi-tenant aware and support multi-tenancy of the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, native identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined network constructs and advanced network features such as virtual networks, BGPaaS, network policies, service chaining, and other telecom features. The network controller 24 can support network isolation using virtual network constructs and can support Layer 3 networks.

[0089] To interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual router 21) a network policy called a virtual network policy (VPN) and alternatively referred to herein as a virtual network router or virtual network topology. VPNs define the connection policy between virtual networks. A single network controller 24 can support multiple Kubernetes clusters, and thus, VPNs allow multiple virtual networks to be connected within a namespace, within a Kubernetes cluster, and across Kubernetes clusters. VPNs can also be extended to support virtual network connections across multiple instances of the network controller 24.

[0090] The network controller 24 is capable of supporting multi-layer security using network policies. The default behavior of Kubernetes is for container pools to communicate with each other. To apply network security policies, the SDN architecture implemented by the network controller 24 and virtual router 21 can operate as the CNI of Kubernetes through CNI 17. For Layer 3, network-level isolation occurs and virtual networks operate at L3. Virtual networks are connected through policies. Kubernetes native network policies provide security at Layer 4. The SDN architecture can support Kubernetes network policies. Kubernetes network policies operate at the boundaries of the Kubernetes namespace. The SDN architecture can add custom resources for enhancing network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta-tags to apply granular security policies in an extensible manner). In some examples, for Layer 4+, the SDN architecture can support integration with containerized security devices and / or Istio and can provide encryption support.

[0091] As Figure 1As part of the SDN architecture shown in Figure 2, the network controller 24 can support multi-cluster deployments, which are important for telco cloud and high-end enterprise use cases. For example, the SDN architecture can support multiple Kubernetes clusters. The Cluster API can be used to support lifecycle management of Kubernetes clusters. Kubefed V2 can be used to configure node 30 federation across Kubernetes clusters. The Cluster API and Kubefed V2 are optional components used to support a single instance of the network controller 24, which supports multiple Kubernetes clusters.

[0092] SDN architectures can use web user interfaces and telemetry components to provide insights into infrastructure, clusters, and applications. Telemetry nodes can be cloud-native and include microservices that support insights.

[0093] Due to the above and other features described elsewhere herein, the computing infrastructure 8 implements a cloud-native SDN architecture and can provide one or more of the following technical advantages. For example, the network controller 24 is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrades of the various component microservices of the configuration node 30 and the control node 32 (as well as any other components of other examples of the network controller described in this disclosure). The present technology can further enable optional cloud-native monitoring (telemetry) and user interfaces, a high-performance data plane for containers using DPDK-based virtual routers connected to a pool of DPDK-enabled containers, and cloud-native configuration management that, in some cases, takes advantage of the configuration frameworks of existing orchestration platforms (such as Kubernetes or Openstack). As a cloud-native architecture, the network controller 24 is a scalable and resilient architecture that addresses and supports multiple clusters. In some cases, the network controller 24 can also support the scalability and performance requirements of key performance indicators (KPIs).

[0094] For example, an SDN architecture with features and technical advantages such as those described herein can be used to implement a cloud-native telco cloud to support 5G mobile networks (and subsequent generations) and edge computing, as well as an enterprise Kubernetes platform including, for example, high-performance cloud-native application hosting. Telecom cloud applications are rapidly moving towards containerized cloud-native solutions. 5G fixed and mobile networks are driving the need to deploy workloads as microservices with significant disaggregation, specifically in 5G Next Generation RAN (5GNR). The 5G Next Generation Core (5GNC) may be deployed as a set of microservice-based applications corresponding to the various components described by 3GPP. When viewed as a group of microservices delivering applications, the 5GNC may be a highly complex combination of a pool of containers with complex network, security, and policy requirements. For such use cases, the cloud-native SDN architecture described herein with well-defined constructs for network, security, and policy can be utilized. The network controller 24 may provide relevant APIs that enable the creation of these complex constructs.

[0095] Similarly, the User Plane Function (UPF) within the 5GNC is an ultra-high-performance application. It can be delivered as a highly distributed pool of high-performance containers. The SDN architecture described here is capable of providing a very high-throughput data plane (in terms of bits per segment (bps) and packets per second (pps)). Integration with the DPDK virtual router, featuring the recent performance enhancement eBPF, and with smart NICs will help achieve the required throughput. The DPDK-based virtual router is described in further detail in U.S. application Ser. No. 17 / 649,632, filed on February 1, 2022, entitled “CONTAINERIZED ROUTER WITH VIRTUAL NETWORKING,” which is incorporated herein by reference in its entirety.

[0096] High performance processing may also be relevant in GiLAN as workloads migrate from more traditional virtualized workloads to containerized microservices. In the UPF and data plane of GiLAN services, such as the GiLAN firewall, intrusion detection and prevention, virtualized IP Multimedia Subsystem (vIMS) voice / video, throughput is high and sustained in both bps and pps. For the control plane of 5GNC functions, such as the Access and Mobility Management Function (AMF), Session Management Function (SMF), and for some GiLAN services (e.g., IMS), the advantage of small packets is that pps will remain high while absolute traffic can be moderate in bps. In some examples, the SDN controller and data plane provide millions of packets per second for each virtual router 21 implemented on the server 12. In the 5G radio access network (RAN), to move away from the proprietary, vertically integrated RAN stacks provided by traditional radio vendors, Open RAN decouples RAN hardware and software across multiple components, including the non-RT radio intelligent controller (RIC), the near-real-time RIC, the centralized unit (CU) control plane and user plane (CU-CP and CU-UP), the distributed unit (DU), and the radio unit (RU). If necessary, the software components are deployed on a commodity server architecture implemented with programmable accelerators. The SDN architecture described here can support the O-RAN specification.

[0097] Edge computing may primarily target two distinct use cases. The first is as a support for containerized telecom infrastructure (e.g., 5G RAN, UPF, security functions), and the second is for containerized service workloads for telecom and third parties such as suppliers or enterprise customers. In both cases, edge computing is effectively a special case of GiLAN, where traffic is interrupted for specialized processing at highly distributed locations. In many cases, these locations will have limited resources (power, cooling, space). The SDN architecture described here may be well-suited to supporting very lightweight footprint requirements, supporting compute and storage resources in sites that are remote from the associated control functions, and being location-aware in how workloads and storage are deployed. Some sites may have as few as one or two compute nodes, which deliver a very specific set of services to a highly localized set of users or other services. A hierarchy of sites may exist, with central sites densely connected with multiple paths, regional sites multiply connected with two or four uplink paths, and remote edge sites having connections to only one or two upstream sites. This requires extreme flexibility in how the SDN architecture can be deployed and how (and where) tunneled traffic within the overlay is terminated and bound to the core transport network (SRv6, MPLS, etc.). Similarly, in sites hosting telco cloud infrastructure workloads, the SDN architecture described here can support specialized hardware (GPUs, smart NICs, etc.) required for high-performance workloads. There may also be workloads that require SR-IOV. Therefore, the SDN architecture can also support the creation of VTEPs at the ToR and have the VTEPs link back to the overlay as VXLAN.

[0098] It is expected that there will be a mix of fully distributed Kubernetes microclusters with each site running its own master cluster, and the SDN architecture can support scenarios like remote computing.

[0099] For use cases involving enterprise Kubernetes platforms, high-performance cloud-native applications, power financial services platforms, online gaming services, and hosted application service providers, the cloud platforms that deliver these applications must provide high performance, resilience to failures, high security, and visibility. Applications hosted on these platforms tend to be developed in-house. Application developers and platform owners work with infrastructure teams to deploy and operate instances of the organization's applications. These applications tend to require high throughput (>20Gbps per server) and low latency. Some applications may also use multicast for signaling or payload traffic. Additional hardware and network infrastructure can be utilized to ensure availability. Applications and microservices are partitioned using namespaces within the cluster. In highly secure environments, isolation between namespaces is critical. While a default deny policy is standard in zero-trust application deployment environments, additional network segmentation using virtual routing and forwarding instances (VRFs) adds an extra layer of security and allows the use of overlapping network ranges. Overlapping network ranges are a key requirement for managing application hosting environments, which tend to standardize the set of reachable endpoints for all managed customers.

[0100] Complex microservices-based applications tend to utilize complex network filters. The SDN architecture described herein can deliver high-performance firewall filtering at scale. This filtering can exhibit consistent forwarding performance and lower latency degradation regardless of rule set length or sequence. Some customers may also face regulatory pressures similar to those in telecommunications for application segmentation not only at the network layer but also within the kernel. Specifically, financial services, as well as other services, have requirements for data plane encryption when running on public clouds. In some examples, the SDN architecture described herein can include features that meet these requirements.

[0101] In some examples, when the SDN architecture is automated by applying a dev / test / stage / prod continuous integration / continuous deployment (CI / CD) pipeline, the SDN architecture can provide a GitOps-friendly UX for strict change management control, auditing, and reliability for products that change several times or even hundreds of times per day.

[0102] Figure 2 2 is a block diagram illustrating an example of a cloud-native SDN architecture for a cloud-native network according to the techniques of the present disclosure. The SDN architecture 200 is illustrated by abstracting the underlying connections between the various components. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A-230N ("configuration nodes" or "config nodes" and collectively referred to as "configuration nodes 230") and control nodes 232A-232K (collectively referred to as "control nodes 232"). The configuration nodes 230 and the control nodes 232 may represent Figure 1 Although shown as separate from the server 12, the configuration node 230 and the control node 232 may be executed as one or more workloads on the server 12.

[0103] Configuration node 230 provides a northbound representational state transfer (REST) interface 248 to support intent-driven configuration of SDN architecture 200. Exemplary platforms and applications for pushing intent to configuration node 230 include a virtual machine orchestrator 240 (e.g., Openstack), a container orchestrator 242 (e.g., Kubernetes), a user interface 244, or one or more other applications 246. In some examples, SDN architecture 200 uses Kubernetes as its underlying platform.

[0104] The SDN architecture 200 is divided into a configuration plane, a control plane, a data plane, and an optional telemetry (or analysis) plane. The configuration plane is implemented using horizontally scalable configuration nodes 230, the control plane is implemented using horizontally scalable control nodes 232, and the data plane is implemented using compute nodes.

[0105] At a high level, the configuration node 230 uses the configuration store 224 to manage the configuration resource state of the SDN architecture 200. Generally, a configuration resource (or more simply a "resource") is a named object schema that includes data and / or methods that describe a custom resource and defines an application programming interface (API) for creating and manipulating the data through the API server. A kind is the name of the object schema. Configuration resources can include Kubernetes native resources such as container pools, ingresses, configuration graphs, services, roles, namespaces, nodes, network policies, or load balancing. According to the techniques of this disclosure, configuration resources also include custom resources that are used to extend the Kubernetes platform by defining application programming interfaces (APIs) that are not available in a default installation of the Kubernetes platform. In the example of the SDN architecture 200, custom resources can describe the physical infrastructure, virtual infrastructure, configuration, and / or other resources of the SDN architecture 200. Various custom resources can be instantiated as part of configuring and operating the SDN architecture 200. Instantiated resources (whether local or custom) can be referred to as objects or instances of resources, i.e., persistent entities within the SDN architecture 200 that represent the intent (desired state) and status (actual state) of the SDN architecture 200. Configuration nodes 230 provide an aggregate API for performing operations (i.e., create, read, update, and delete) on configuration resources of the SDN architecture 200 in the configuration store 224. Load balancers 226 represent one or more load balancing objects that load balance configuration requests between configuration nodes 230. Configuration store 224 can represent one or more etcd databases. Configuration nodes 230 can be implemented using Nginx.

[0106] SDN architecture 200 can provide networking for OpenStack and Kubernetes. OpenStack uses a plug-in architecture to support networking. Using virtual machine orchestrator 240 (i.e., OpenStack), the OpenStack networking plug-in driver converts OpenStack configuration objects into configuration objects (resources) for SDN architecture 200. Compute nodes run OpenStack nova to build virtual machines.

[0107] Utilizing the container orchestrator 242, i.e., Kubernetes, the SDN architecture 200 serves as a Kubernetes CNI. As described above, Kubernetes native resources (container pools, services, ingress, external load balancing, etc.) can be supported, and the SDN architecture 200 can support Kubernetes' custom resources for advanced networking and security of the SDN architecture 200.

[0108] Configuration nodes 230 provide REST monitoring to control nodes 232 to monitor changes in configuration resources / objects, and control nodes 232 affect configuration resource changes within the computing infrastructure. Control nodes 232 receive configuration resource data from configuration nodes 230 by monitoring resources and build a complete configuration graph. A given control node in control nodes 232 consumes configuration resource data associated with the control node and sends it to virtual router 21 (i.e., virtual router agent- Figure 1 A control interface 254, in terms of the control plane (not shown), distributes the required configuration to the compute nodes (servers 12). Any compute node 232 may receive only a portion of the graph, as required for processing. Control interface 254 may be an Extensible Messaging and Presence Protocol (XMPP). The number of deployed configuration nodes 230 and control nodes 232 may be a function of the number of supported clusters. To support high availability, the configuration plane may include 2N+1 configuration nodes 230 and 2N control nodes 232.

[0109] Control nodes 232 distribute routes between compute nodes. Control nodes 232 use internal Border Gateway Protocol (iBGP) to exchange routes between control nodes 232, and control nodes 232 can peer with any external BGP-supported gateway or other router. Control nodes 232 can use route reflectors.

[0110] Container pool 250 and virtual machines 252 are examples of workloads that can be deployed to compute nodes by virtual machine orchestrator 240 or container orchestrator 242 and interconnected by SDN architecture 200 using one or more virtual networks.

[0111] Figure 3 2 is a block diagram illustrating another view of components of an SDN architecture 200 according to techniques of the present disclosure and further illustrating in detail. Configuration node 230, control node 232, and user interface 244 are shown with their corresponding component microservices to implement network controller 24 and SDN architecture 200 as a cloud-native SDN architecture. Each component microservice can be deployed to a compute node.

[0112] Figure 3 A single cluster is shown partitioned into network controller 24, user interface 244, compute nodes (servers 12), and telemetry 260 features. Configuration node 230 and control node 232 together comprise network controller 24.

[0113] Configuration node 230 may include component microservice API server 300 (or "Kubernetes API server 300" - corresponding controller 406, Figure 3), a custom API server 301, a custom resource controller 302, and an SDN controller manager 303 (sometimes referred to as a "kubemanager" or "SDN kubemanager" where the orchestration platform for network controller 24 is Kubernetes). "contrail-k8s-kubemanager" is an example of an SDN controller manager 303. The SDN controller manager 303 is distinct from and has different responsibilities than the kube-controller-manager, which is a daemon that embeds core control loops into Kubernetes' controllers, such as the replication controller, endpoint controller, namespace controller, and service account controller. The configuration node 230 utilizes the custom API server 301 to extend the API server 300 interface to form an aggregation layer to support the data model of the SDN architecture 200. As described above, the configuration intent of the SDN architecture 200 can be a custom resource.

[0114] The control node 232 may include a component microservice control 320. As mentioned above Figure 2 As described, control 320 performs configuration distribution and route learning and distribution.

[0115] The server 12 represents a compute node. Each compute node includes a virtual router agent 316 and a virtual router forwarding component (vRouter) 318. Either the virtual router agent 316 or the vRouter 318 can be a component microservice, or both can be component microservices. Generally, the virtual router agent 316 performs control-related functions. The virtual router agent 316 receives configuration data from the control node 232 and converts the configuration data into forwarding information for the vRouter 318. The virtual router agent 316 can also perform firewall rule processing, set up flows for the vRouter 318, and interface with orchestration plugins (CNI for Kubernetes and Nova plugin for OpenStack). When a workload (container pool or VM) is called up on a compute node, the virtual router agent 316 generates a route, and the virtual router 316 exchanges the route with the control node 232 for distribution to other compute nodes (the control node 232 uses BGP to distribute routes between control nodes 232). The virtual router agent 316 also withdraws the route when the workload is terminated. vRouter 318 may support one or more forwarding modes, such as kernel mode, DPDK, smart NIC offload, etc. In some examples of container architectures or virtual machine workloads, the compute nodes may be Kubernetes worker / slave nodes or Openstack nova-compute nodes, depending on the specific orchestrator used.

[0116] One or more optional telemetry nodes 260 provide metrics, alerts, logging, and streaming analytics. Telemetry for the SDN architecture 200 leverages cloud-native monitoring services such as Prometheus, Elastic, Fluentd, Kinaba stack (EFK), and Influx TSDB. The SDN architecture component microservices of the configuration nodes 230, control nodes 232, compute nodes, user interface 244, and analytics nodes (not shown) can generate telemetry data. Services on the telemetry nodes 260 can consume this telemetry data. The telemetry nodes 260 can expose REST endpoints for users and can support insights and event correlation.

[0117] Optionally, the user interface 244 includes a network user interface (UI) 306 and a UI backend 308 service. Generally, the user interface 244 provides configuration, monitoring, visualization, security, and troubleshooting for SDN architecture components.

[0118] Each of the telemetry 260, user interface 244, configuration node 230, control node 232, and server 12 / compute node can be considered a node of the SDN architecture 200, in that each of these nodes is an entity that implements the functionality of a configuration plane, a control plane, or a data plane, or a UI and telemetry node. Node scale is configured during the "build" process, and the SDN architecture 200 supports automatic scaling of nodes of the SDN architecture 200 using an orchestration system operator such as the Kubernetes operator.

[0119] Figure 4 4 is a block diagram illustrating exemplary components of an SDN architecture according to the techniques of this disclosure. In this example, the SDN architecture 400 extends and uses the Kubernetes API server for network configuration objects to implement user intent for that network configuration. In Kubernetes terminology, the network configuration objects are called custom resources and are simply referred to as objects when persisted in the SDN architecture. Configuration objects are primarily user intent (e.g., virtual networks, BGPaaS, network policies, service chains, etc.). Similar objects may exist for Kubernetes native resources.

[0120] The configuration node 230 of the SDN architecture 400 can use the Kubernetes API server for configuration objects, which are called custom resources in Kubernetes terminology.

[0121] Kubernetes provides two ways to add custom resources to the cluster:

[0122] Custom Resource Definitions (CRDs) are simple and can be created without any programming.

[0123] API aggregation requires programming, but allows more control over API behavior, such as controlling how data is stored and controlling transitions between API versions.

[0124] The aggregate API is a slave API server that sits behind the master API server and acts as a proxy. This arrangement is called API Aggregation (AA). To the user, it appears to be just an extension of the Kubernetes API. CRDs allow users to create new types of resources without adding another API server. Regardless of how they are installed, the new resources are called Custom Resources (CRs) to distinguish them from local Kubernetes resources (for example, container pools). CRDs were used in the initial Config prototype. The architecture can use the API Server Builder Alpha library to implement the aggregate API. API Server Builder is a general term for libraries and tools for building local Kubernetes aggregate extensions.

[0125] Typically, each resource in the Kubernetes API requires code to handle REST requests and manage persistent storage for the object. The main Kubernetes API server 300 (implemented by API server microservices 300A–300J) handles local resources and can also perform generic processing for custom resources through CRDs. Aggregate API 402 represents an aggregation layer that extends the Kubernetes API server 300 to allow specialized implementations of custom resources by writing and deploying custom API servers 301 (using custom API server microservices 301A–301M). The main API server 300 delegates requests for custom resources to the custom API server 301, thereby making the resource available to all of its clients.

[0126] In this way, the API server 300 (e.g., kube-api server) receives Kubernetes configuration objects, local objects (container pools, services), and custom resources defined according to the techniques of this disclosure. Custom resources for the SDN architecture 400 can include configuration objects that, when the desired state of the configuration object is achieved in the SDN architecture 400, implement the desired network configuration of the SDN architecture 400. Custom resources can correspond to configuration schemes that are traditionally defined for network configuration but in accordance with the techniques of this disclosure, and are extended to be manipulated through the aggregate API 402. Alternatively, the custom resources can be named and referred to herein as "custom resources for SDN architecture configuration." Each custom resource for SDN architecture configuration can correspond to a configuration object type typically exposed by an SDN controller but in accordance with the techniques described herein, with the custom resource being used to expose the configuration object and merge the configuration object with Kubernetes native / built-in resources. These configuration objects may include virtual networks, BGP as a Service (BGPaaS), subnets, virtual routers, service instances, projects, physical interfaces, logical interfaces, nodes, network ipams, floating ips, alarms, alias ips, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances, etc. Thus, the SDN architecture system 400 supports a unified intent model exposed by the aggregated API 402, i.e., through the Kubernetes controllers 406A-406N and through the custom resource controller 302 (in Figure 4 302L), Kubernetes controllers 406A-406N and custom resource controller 302 work to align the actual state of the computing infrastructure, including network elements, with the desired state. Controller 406 can represent kube-controller-managers.

[0127] The aggregation layer of API server 300 sends the API custom resource to its corresponding registered custom API server 301. There may be multiple custom API servers / custom resource controllers supporting different types of custom resources. Custom API server 301 processes the custom resources for SDN architecture configuration and writes the custom resources to configuration storage 304 (which may be etcd). Custom API server 301 can be a host and exposes the SDN controller identifier allocation service required by custom resource controller 302.

[0128] The custom resource controller 302 starts applying business logic to satisfy the user intent configured with the user intent. The business logic is implemented as a coordination loop. Figure 8is a block diagram illustrating an example of a custom controller for custom resources configured in an SDN architecture according to the techniques of this disclosure. Custom controller 814 may represent an exemplary instance of custom resource controller 301. Figure 8 In the example shown, a custom controller 814 can be associated with a custom resource 818. Custom resource 818 can be any custom resource used for SDN architecture configuration. Custom controller 814 can include a coordinator 816, which includes logic for executing a reconciliation loop in which custom controller 814 observes 834 (e.g., monitors) a current state 832 of custom resource 818. In response to determining that a desired state 836 does not match the current state 832, coordinator 816 can perform an action to adjust 838 the state of the custom resource so that the current state 832 matches the desired state 836. A request can be received by API server 300 and relayed to custom API server 301 to change the current state 832 of custom resource 818 to the desired state 836.

[0129] If API request 301 is a request to create a custom resource, coordinator 816 can act on a creation event for instance data of the custom resource. Coordinator 816 can create instance data for custom resources on which the requested custom resource depends. As an example, an edge node custom resource can depend on a virtual network custom resource, a virtual interface custom resource, and an IP address custom resource. In this example, when coordinator 816 receives a creation event for an edge node custom resource, it can also create the custom resources on which the edge node custom resource depends, such as the virtual network custom resource, the virtual interface custom resource, and the IP address custom resource.

[0130] By default, the custom resource controller 302 operates in active-passive mode and uses master election for consistency. When a controller pool starts, it attempts to create a ConfigMap resource in Kubernetes using the specified key. If the creation succeeds, the pool becomes the master and begins handling coordination requests; otherwise, the pool blocks the ConfigMap creation attempt, which results in an infinite loop.

[0131] Custom resource controller 300 can track the status of custom resources it creates. For example, a virtual network (VN) creates a routing instance (RI), which in turn creates a routing target (RT). If the creation of a routing target fails, the routing instance status degrades, and thus the virtual network status as well. Therefore, custom resource controller 300 can output custom messages indicating the status of these custom resources to facilitate troubleshooting. Figure 9An exemplary flow of creating, observing, and coordinating between custom resource types that depend on different custom resource types is shown in FIG.

[0132] The configuration plane implemented by the configuration node 230 is highly available. The configuration node 230 can be based on Kubernetes and include a kube-api server service (e.g., API server 300) and a storage backend etcd (e.g., configuration store 304). In practice, the aggregated API 402 implemented by the configuration node 230 operates as a front end to the control plane implemented by the control node 232. The primary implementation of the API server 300 is a kube-api server designed to scale horizontally by deploying more instances. As shown, several instances of the API server 300 can be run to load balance API requests and processing.

[0133] Configuration store 304 can be implemented as etcd. etcd is a consistent and highly available key-value store used as a Kubernetes backing store for cluster data.

[0134] exist Figure 4 In the example of FIG4 , servers 12 in SDN architecture 400 each include an orchestration agent 420 and a containerized (or “cloud-native”) routing protocol daemon 324. These components of SDN architecture 400 are described in further detail below.

[0135] The SDN controller manager 303 can operate as an interface between Kubernetes core resources (services, namespaces, container pools, network policies, network attachment definitions) and the extended SDN architecture resources (virtual networks, routing instances, etc.). The SDN controller manager 303 monitors the Kubernetes API for changes to custom resources configured in the Kubernetes core network and SDN architecture and can therefore perform CRUD operations on related resources. As used herein, monitoring a resource can refer to monitoring an object or object instance of a resource type of the resource.

[0136] In some examples, the SDN controller manager 303 is a collective term for one or more Kubernetes custom controllers. In some examples, in a single or multiple cluster deployment, the SDN controller manager 303 can run on the Kubernetes cluster it manages.

[0137] The SDN controller manager 303 listens to the following Kubernetes objects for create, delete, and update events:

[0138] ·Container pool

[0139] ·Serve

[0140] Node port

[0141] ·Entrance

[0142] Endpoints

[0143] Namespace

[0144] ·deploy

[0145] Network strategy.

[0146] When these events are generated, the SDN controller manager 303 creates appropriate SDN architecture objects, which in turn are defined as custom resources configured by the SDN architecture. In response to detecting an event on an instance of a custom resource, whether instantiated by the SDN controller manager 303 and / or through the custom API server 301, the control node 232 obtains configuration data about the instance of the custom resource and configures a corresponding instance of the configuration object in the SDN architecture 400.

[0147] For example, the SDN controller manager 303 monitors container pool creation events and, accordingly, may create the following SDN architecture objects: virtual machine (workload / container pool), virtual machine interface (virtual network interface), and instance IP (IP address). In this case, the control node 232 may then instantiate the SDN architecture objects in the selected compute node.

[0148] As an example, based on a watch, control node 232A may detect an event on an instance of a first custom resource exposed by custom API server 301A. The first custom resource is used to configure an aspect of SDN architecture system 400 and corresponds to a configuration object type of SDN architecture system 400. For example, the type of configuration object may be a firewall rule corresponding to the first custom resource. In response to the event, control node 232A may obtain configuration data regarding the firewall rule instance (e.g., a firewall rule specification) and provision the firewall rule in a virtual router for server 12A. Configuration node 230 and control node 232 may utilize configuration objects of corresponding types in the SDN architecture to perform similar operations on other custom resources, such as virtual networks, BGP as a Service (BGPaaS), subnets, virtual routers, service instances, projects, physical interfaces, logical interfaces, nodes, network IPAMs, floating IP addresses, alarms, alias IP addresses, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances, and the like.

[0149] Figure 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure. Figure 5The computing device 500 in FIG. 5 can represent an actual server or a virtual server and can represent an exemplary example of any server 12 and can be referred to as a computing node, a master / slave node, or a host. In this example, the computing device 500 includes a bus 542 coupled to hardware components in the hardware environment of the computing device 500. The bus 542 is coupled to a network interface card (NIC) 530, a storage disk 546, and one or more microprocessors 510 (hereinafter referred to as "microprocessors 510"). The NIC 530 can support SR-IOV. In some cases, the front-side bus can be coupled to the microprocessor 510 and the memory device 544. In some examples, the bus 542 can be coupled to the memory device 544, the microprocessor 510, and the NIC 530. The bus 542 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to the bus 542. In some examples, components coupled to the bus 542 control DMA transfers between components coupled to the bus 542.

[0150] The microprocessor 510 may include one or more processors, each of which includes an independent execution unit that executes instructions in accordance with an instruction set architecture and instructions stored in a storage medium. The execution units may be implemented as independent integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip microprocessor).

[0151] Disk 546 represents computer-readable storage media, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical memory, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by microprocessor 510.

[0152] The main memory 544 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the necessary program code and program data in the form of instructions or data structures and that can be accessed by a computer. The main memory 544 provides a physical address space consisting of addressable memory locations.

[0153] The network interface card (NIC) 530 includes one or more interfaces 532 configured to exchange packets using the links of the underlying physical network. The interfaces 532 may include a port interface card having one or more network ports. The NIC 530 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 530 and other devices coupled to the bus 542 may be read from the NIC memory or written to the NIC memory.

[0154] Memory 544, NIC 530, storage disk 546, and microprocessor 510 can provide an operating environment for a software stack including an operating system kernel 580 executed in kernel space. For example, kernel 580 can represent the kernel of Linux, Berkeley Software Distribution (BSD), another Unix variant, or the Windows server operating system kernel commercially available from Microsoft. In some examples, the operating system can run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include kernel-based virtual machines (KVM) for the Linux kernel, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can cover virtual machine managers (VMMs). The operating system including kernel 580 provides an execution environment for one or more processes in user space 545.

[0155] The kernel 580 includes a physical driver 525 that uses a network interface card 530. The network interface card 530 may also implement SR-IOV to enable communication between the network interface card 530 and a container 529A or one or more virtual machines ( Figure 5530). A shared virtual device such as a virtual function can provide dedicated resources so that each virtual function can access the dedicated resources of the NIC 530, so that the NIC 530 appears to be a dedicated NIC to each virtual function. A virtual function can represent a lightweight PCIe function that shares physical resources with the physical function used by the physical driver 525 and with other virtual functions. For a NIC 530 that supports SR-IOV, the NIC 530 can have thousands of available virtual functions that comply with the SR-IOV standard, but for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0156] The computing device 500 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from physical switches to software or "virtual" routers (including virtual router 506) on physical servers coupled to the switching fabric. A virtual router may be a physical server (e.g., Figure 1 A virtual router dynamically creates and manages one or more virtual networks that can be used for communications between virtual network endpoints by a process or thread executed by a server 12 in the virtual router. In one example, the virtual routers each implement a virtual network using an overlay network that provides the ability to decouple the virtual address of the endpoint from the physical address (e.g., IP address) on the server executing the endpoint. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and across virtual networks on top of a physical network. As used herein, the term "virtual router" can cover Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other device and / or software that is located on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. In Figure 5 In the exemplary computing device 500 of FIG. 5 , the virtual machine router 506 runs in user space as a DPDK-based virtual router. However, the virtual router 506 can run in a hypervisor, a host operating system, a host application, or a virtual machine through various implementations.

[0157] The virtual router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module typically used by the Kubernetes deployment of the container pool 502. The virtual router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 506 can perform network services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0158] The virtual router 506 can run as a kernel module or as a user space DPDK process (here, the virtual router 506 is shown as being located in user space 545). A virtual router agent 514 can also run in user space. In the exemplary computing device 500, the virtual router 506 runs as a DPDK-based virtual router in user space, but the virtual router 506 can run in a hypervisor, a host operating system, a host application, or a virtual machine in various implementations. The virtual router agent 514 connects to the network controller 24 using a channel for downloading configuration and forwarding information. The virtual router agent 514 programs the forwarding state into the virtual router data (or "forwarding") plane represented by the virtual router 506. The virtual router 506 and the virtual router agent 514 can be processes. The virtual router 506 and the virtual router agent 514 are containerized / cloud native.

[0159] Virtual router 506 can be multi-threaded and run on one or more processor cores. Virtual router 506 can include multiple queues. Virtual router 506 can implement a packet processing pipeline. Virtual router agent 514 stitches the pipeline in various ways, from the simplest to the most complex, depending on the operations applied to the packets. Virtual router 506 can maintain multiple instances of the forwarding infrastructure. Virtual router 506 can access and update tables using RCU (read-copy-update) locks.

[0160] To send packets to other compute nodes or switches, the virtual router 506 uses one or more physical interfaces 532. Typically, the virtual router 506 exchanges overlay packets with workloads (such as VMs or container pools 502). The virtual router 506 has multiple virtual network interfaces (e.g., vifs). These interfaces may include the kernel interface vhost0, which exchanges packets with the host operating system, and the interface pkt0, which exchanges packets with the virtual router agent 514 to obtain forwarding state from the network controller and send exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 532. The other virtual network interfaces of the virtual router 506 are used to exchange packets with workloads.

[0161] In a kernel-based deployment of virtual router 506 (not shown), virtual router 506 is installed as a kernel module within the operating system. Virtual router 506 registers itself with the TCP / IP stack to receive packets from any desired operating system interface it desires. Interfaces can be bound, physicalized, tapped (for VMs), vethed (for containers), and so on. In this mode, virtual router 506 relies on the operating system to send packets to and receive packets from different interfaces. For example, the operating system can expose a tapped interface supported by a vhost network driver to communicate with the VM. Once virtual router 506 registers packets from the tapped interface, the TCP / IP stack sends all packets to the virtual router. Virtual router 506 sends packets via the operating system. In addition, the operating system handles NIC queues (physical or virtual). Packet processing can operate in interrupt mode, which generates interrupts and may cause frequent context switches. When there is a high packet rate, the overhead associated with frequent interrupts and context switches may overwhelm the operating system and lead to poor performance.

[0162] In a DPDK-based deployment of virtual router 506 ( Figure 5 (not shown), virtual router 506 is installed as a userspace 545 application linked to the DPDK library. Specifically, at higher packet rates, this can result in faster performance than a kernel-based deployment. Physical interface 532 is used by the DPDK's poll-mode driver (PMD), rather than the kernel's interrupt-based driver. The registers of physical interface 532 are exposed to userspace 545 for access by the PMD. In this way, the bound physical interface 532 is no longer managed by or visible to the host operating system, and the DPDK-based virtual router 506 manages the physical interface 532. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the DPDK data plane of virtual router 506. Compared to interrupt mode, this "polling mode" nature makes the DPDK data plane packet processing / forwarding of virtual router 506 more efficient at higher packet rates. Compared to kernel mode virtual router 506, there are relatively fewer interrupts and context switches during packet I / O, and in some cases, interrupts and context switches can be avoided altogether during packet I / O.

[0163] In general, each of container pools 502A-502B may be assigned one or more virtual network addresses for use within a corresponding virtual network, where each virtual network may be associated with a different virtual subnet provided by virtual router 506. For example, container pool 502B may be assigned its own virtual layer 3 (L3) IP address for sending and receiving communications, but container pool 502B may not be aware of the IP address of computing device 500 on which container pool 502B is running. Thus, the virtual network address may differ from the logical address of the underlying physical computer system (e.g., computing device 500).

[0164] The computing device 500 includes a virtual router agent 514 that controls the overlay of the virtual network of the computing device 500 and coordinates the routing of data packets within the computing device 500. Typically, the virtual router agent 514 communicates with the network controller 24 for the virtualization infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 500, and more specifically, the virtual router 506, and virtual network interfaces. By configuring the virtual router 506 based on information received from the network controller 24, the virtual router agent 514 can support the configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0165] In one example, a network packet, such as a Layer 3 (L3) IP packet or a Layer 2 (L2) Ethernet packet generated and used by containers 529A–529B located within a virtual network domain, can be encapsulated within another packet (e.g., another IP or Ethernet packet) transmitted by a physical network. The packet transmitted within the virtual network can be referred to herein as an "inner packet," while the physical network packet can be referred to herein as an "outer packet" or "tunnel packet." The encapsulation and / or decapsulation of the virtual network packet within the physical network packet can be performed by the virtual router 506. This functionality is referred to herein as tunneling and can be used to create one or more overlay networks. In addition to IPinIP, other exemplary tunneling protocols that can be used include Multiprotocol Label Switching (MPLS) based on GRE, MPLS based on User Datagram Protocol (UDP), and the like. The virtual router 506 performs tunnel encapsulation / decapsulation on packets originating from / destined for any container of the container pool 502 , and exchanges packets with the container pool 502 via the bus 542 and / or the NIC 530 bridge.

[0166] As described above, the network controller 24 can provide a logically centralized controller that facilitates the operation of one or more virtual networks. For example, the network controller 24 can maintain a routing information base, such as one or more routing tables that store routing information for the physical network and one or more overlay networks. The virtual router 506 implements one or more virtual routing and forwarding instances (VRFs), such as VRF 222A, for the corresponding virtual network for which its virtual router 506 operates as a corresponding tunnel endpoint. Generally, each VRF stores forwarding information for the corresponding virtual network and identifies where data packets are forwarded and whether the packets are encapsulated in a tunnel protocol, such as using a tunnel header, which can include one or more headers from different layers of the virtual network protocol stack. Each VRF can include a network forwarding table that stores routing and forwarding information for the virtual network.

[0167] NIC 530 may receive the tunnel packet. Virtual router 506 processes the tunnel packet to determine the virtual networks of the source and destination endpoints of the inner packet from the tunnel encapsulation header. Virtual router 506 may strip off the Layer 2 header and the tunnel encapsulation header to forward only the inner packet internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN tag or an MPLS label, indicating the virtual network (e.g., the virtual network corresponding to VRF 222A). VRF 222A may include forwarding information about the inner packet. For example, VRF 222A may map the destination Layer 3 address of the inner packet to a virtual network interface. Accordingly, VRF 222A forwards the inner packet via the virtual network interface of container pool 502A.

[0168] Container 529A can also originate internal packets as a source virtual network endpoint. For example, container 529A can generate a Layer 3 internal packet that is destined for a destination virtual network endpoint executed by another computing device (i.e., not computing device 500) or destined for another container. Container 529A can send the Layer 3 internal packet to virtual router 506 via the virtual network interface attached to VRF 222A.

[0169] Virtual router 506 receives the inner packet and the Layer 2 header and determines the virtual network of the inner packet. Virtual router 506 may determine the virtual network using any of the virtual network interface implementation technologies described above (e.g., macvlan, veth, etc.). Virtual router 506 uses VRF 222A corresponding to the virtual network of the inner packet to generate an outer header for the inner packet, an outer header including an outer IP header for the overlay tunnel, and a tunnel encapsulation header identifying the virtual network. Virtual router 506 encapsulates the inner packet with the outer header. Virtual router 506 may encapsulate the tunnel packet with a new Layer 2 header that has a destination Layer 2 address associated with a device located outside of computing device 500 (e.g., TOR switch 16 or a server 12). If located outside of computing device 500, virtual router 506 outputs the tunnel packet with the new Layer 2 header to NIC 530 using a physical function. NIC 530 outputs the packet on an outbound interface. If the destination is another virtual network endpoint executing on computing device 500 , virtual router 506 routes the packet to an appropriate one of the virtual network interfaces.

[0170] In some examples, a controller for computing device 500 (e.g., Figure 1 The network controller 24 in the NIC 530 configures the default route in each container pool 502 to cause the virtual machine 224 to use the virtual router 506 as the initial next hop for outbound packets. In some examples, the NIC 530 is configured with one or more forwarding rules that cause all packets received from the virtual machine 224 to be switched to the virtual router 506.

[0171] Container pool 502A includes one or more application containers 529A. Container pool 502B includes an instance of containerized routing protocol daemon (cRPD) 560. Container platform 588 includes container engine 590, orchestration agent 592, service agent 593, and CNI 570.

[0172] The container engine 590 includes code executed by the microprocessor 510. The container engine 590 can be one or more computer processes. The container engine 590 runs containerized applications in the form of containers 529A-529B. The container engine 590 can represent Docker, rkt, or other container engines for managing containers. Generally, the container engine 590 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 592, the container engine 590 can obtain an image and instantiate it as an executable container in the container pool 502A-502B.

[0173] The service agent 593 includes code that can be executed by the microprocessor 510. The service agent 593 can be one or more computer processes. The service agent 593 monitors the addition and removal of services and endpoint objects, and, for example, uses services to maintain the network configuration of the computing device 500 to ensure communication between the container pool and the containers. The service agent 593 can also manage the management ip table to capture traffic to the virtual IP address and port of the service and redirect the traffic to the proxy port of the proxy backup container pool. The service agent 593 can represent the kube agent of the slave node of the Kubernetes cluster. In some examples, the container platform 588 does not include the service agent 593, or the service agent 593 is disabled by the CNI 570 to facilitate the configuration of the virtual router 506 and the container pool 502.

[0174] The orchestration agent 592 includes code that can be executed by the microprocessor 510. The orchestration agent 592 can be one or more computer processes. The orchestration agent 592 can represent the kubelet of a slave node of the Kubernetes cluster. The orchestration agent 592 is an agent that receives container-specific data of a container and ensures that the container is executed by the computing device 500. For example, Figure 1 The container-specific data may be in the form of a manifest file sent from orchestrator 23 to orchestration agent 592 or received indirectly via a command-line interface, an HTTP endpoint, or an HTTP server. The container-specific data may be a container pool specification (e.g., a PodSpec - YAML (yet another markup language) or a JSON object describing a container pool) for a container pool 502 of containers. Based on the container-specific data, orchestration agent 592 directs container engine 590 to obtain a container image for container 529 and instantiate the container image for execution of container 529 by computing device 500.

[0175] Orchestration agent 592 instantiates CNI 570 and otherwise calls CNI 570 to configure one or more virtual network interfaces for each container pool 502. For example, orchestration agent 592 receives container-specific data for container pool 502A and directs container engine 590 to create container pool 502A with container 529A based on the container-specific data for container pool 502A. Orchestration agent 592 also calls CNI 570 to configure a virtual network interface for the virtual network corresponding to VRF 222A for container pool 502A. In this example, container pool 502A is a virtual network endpoint for the virtual network corresponding to VRF 222A.

[0176] CNI 570 can obtain interface configuration data for configuring the virtual network interfaces of container pool 502. Virtual router agent 514 operates as a virtual network control plane module that enables network controller 24 to configure virtual router 506. Unlike the orchestration control plane (including container platform 588 for slave and master nodes, such as orchestrator 23), which manages the provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 and virtual router agent 514 for slave nodes) manages the configuration of virtual networks partially implemented in the data plane by virtual router 506 of the slave nodes. Virtual router agent 514 communicates interface configuration data regarding virtual network interfaces to CNI 570, enabling the orchestration control plane element (i.e., CNI 570) to configure the virtual network interfaces according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this enables the CNI 570 to obtain interface configuration data for multiple virtual network interfaces of a container pool and configure the multiple virtual network interfaces, which can reduce the communication and resource overhead inherent in invoking a separate CNI 570 to configure each virtual network interface.

[0177] A containerized routing protocol daemon is described in U.S. application Ser. No. 17 / 649,632, filed on February 1, 2022, the entire contents of which are incorporated herein by reference.

[0178] Figure 6 1300 is a block diagram of an exemplary computing device operating as a compute node for one or more clusters of an SDN architecture system according to the techniques of this disclosure. Computing device 1300 can represent one or more real or virtual servers. In some instances, computing device 1300 can implement one or more master nodes for a corresponding cluster or clusters.

[0179] Although shown and described as being executed by a single computing device 1300, the scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration storage 1328 may be distributed among multiple computing devices comprising a computing system or hardware / server cluster. In other words, each of the multiple computing devices may provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, network controller manager 1326, network controller, SDN controller manager 1325, control node 232A, or configuration storage 1328.

[0180] In this example, computing device 1300 includes a bus 1342 that couples hardware components in the hardware environment of computing device 1300. Bus 1342 couples a network interface card (NIC) 1330, a storage disk 1346, and one or more microprocessors 1310 (hereinafter referred to as "microprocessors 1310"). In some cases, a front-side bus can couple microprocessor 1310 to memory device 1344. In some examples, bus 1342 can couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to bus 1342. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

[0181] The microprocessor 1310 may include one or more processors, each of which includes an independent execution unit that executes instructions in accordance with an instruction set architecture and instructions stored in a storage medium. The execution units may be implemented as independent integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0182] Disk 1346 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical memory, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by microprocessor 1310.

[0183] The main memory 1344 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the necessary program code and program data in the form of instructions or data structures and that can be accessed by a computer. The main memory 1344 provides a physical address space consisting of addressable memory locations.

[0184] The network interface card (NIC) 1330 includes one or more interfaces 1332 configured to exchange packets using the links of the underlying physical network. The interfaces 1332 may include a port interface card having one or more network ports. The NIC 1330 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 1330 and other devices coupled to the bus 1342 may be read from the NIC memory or written to the NIC memory.

[0185] Memory 1344, NIC 1330, storage disk 1346, and microprocessor 1310 can provide an operating environment for a software stack including an operating system kernel 1314 executed in kernel space. For example, kernel 1314 can represent the kernel of Linux, Berkeley Software Distribution (BSD), another Unix variant, or the Windows server operating system kernel commercially available from Microsoft. In some instances, the operating system can run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can encompass a virtual machine manager (VMM). The operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. Kernel 1314 includes a physical driver 1327 that uses a network interface card 1330.

[0186] Computing device 1300 may be coupled to a physical network switching fabric that includes software or an overlay network of virtual routers that extend the switching fabric from physical switches to physical servers coupled to the switching fabric, such as virtual router 21. Computing device 1300 may configure slave nodes of a cluster using one or more dedicated virtual networks.

[0187] API server 300A, scheduler 1322, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, and configuration store 1328 may implement a master node for a cluster and may alternatively be referred to as a "master component." The cluster may be a Kubernetes cluster, and the master node may be a Kubernetes master node, in which case the master component is a Kubernetes master component.

[0188] Each of the API server 300A, controller 406A, custom API server 301A, and custom resource controller 302A includes code executable by microprocessor 1310. Custom API server 301A validates and configures data regarding custom resources configured for the SDN architecture. A service can be an abstraction that defines a logical set of container pools and policies for accessing the container pools. A set of container pools that implement the service is selected based on the service definition. A service can be partially implemented as a load balancer or otherwise include a load balancer. API server 300A and custom API server 301A can implement a representational state transfer (REST) interface to process REST operations and provide a front-end to the corresponding cluster shared state stored in configuration store 1328 as part of the configuration plane of the SDN architecture. API server 300A can represent a Kubernetes API server.

[0189] Configuration store 1328 is a backup store for all cluster data. Cluster data can include cluster state and configuration data. Configuration data can also provide a backend for service discovery and / or provide locking services. Configuration store 1328 can be implemented as a key-value store. Configuration store 1328 can be a central database or a distributed database. Configuration store 1328 can represent etcd storage. Configuration store 1328 can also represent the Kubernetes configuration store.

[0190] Scheduler 1322 includes code executable by microprocessor 1310. Scheduler 1322 can be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., a container pool in a container) and selects slave nodes on which to run the virtual execution elements. Scheduler 1322 can select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, locality, etc. Scheduler 1322 can represent a Kubernetes scheduler.

[0191] Typically, API server 300A may call scheduler 1322 to schedule a container pool. Scheduler 1322 may select a slave node and return an identifier for the selected slave node to API server 300A, which may write the identifier to configuration storage 1328 associated with the container pool. API server 300A may call the orchestration agent for the selected slave node, causing the container engine 208 for the selected slave node to obtain the container pool from the storage server and create a virtual execution element on the slave node. Orchestration agent 310 for the selected slave node may update the status of the container pool to API server 1320, and API server 300A persists the new status to configuration storage 1328. In this manner, computing device 1300 instantiates the new container pool in computing infrastructure 8.

[0192] Controller manager 1326 includes code that can be executed by microprocessor 1310. Controller manager 1326 can be one or more computer processes. Controller manager 1326 can be embedded in a kernel control loop to monitor the shared state of the cluster by obtaining notifications from API server 300A. Controller manager 1326 can attempt to move the state of the cluster toward a desired state. Exemplary controller 406A and custom resource controller 302A can be managed by controller manager 1326. Other controllers can include replication controllers, endpoint controllers, namespace controllers, and service account controllers. Controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminated container pool garbage collection, cascade deletion garbage collection, node garbage collection, etc. Controller manager 1326 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0193] The network controller used in the SDN architecture described herein can provide cloud networking for computing infrastructure operating on top of the network infrastructure. Cloud networking can include private clouds for enterprises or service providers, Infrastructure as a Service (IaaS), and Virtual Private Clouds (VPCs) for Cloud Service Providers (CSPs). Private cloud, VPC, and IaaS use cases may involve references such as Figure 1 The multi-tenant virtualized data center described above. In this scenario, multiple tenants within the data center share the same physical resources (physical servers, physical storage, and physical networks). Each tenant is assigned its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; and virtual networks). These logical resources are isolated from each other unless explicitly permitted by security policy. Virtual networks within the data center can also interconnect with physical IP VPNs or L2VPNs.

[0194] Network controllers (or "SDN controllers") can provide network function virtualization (NFV) for networks such as the service edge network, broadband subscriber management edge network, and mobile edge network. NFV involves the orchestration and management of network functions such as firewalls, intrusion detection and prevention systems (IDS / IPS), deep packet inspection (DPI), caching, and wide area network (WAN) optimization within virtual machines, containers, or other virtual execution elements rather than physical hardware devices.

[0195] The SDN controller manager 1325 includes code executable by the microprocessor 1310. The SDN controller manager 1325 can be one or more computer processes. The SDN controller manager 1325 operates as an interface between orchestration-oriented elements (e.g., the scheduler 1322, the API server 300A and the custom API server 301A, the controller manager 1326, and the configuration store 1328). Generally, the SDN controller manager 1325 monitors clusters of new Kubernetes native objects (e.g., container pools and services). The SDN controller manager 1325 can isolate container pools in a virtual network and connect container pools to services.

[0196] The SDN controller manager 1325 can run as a container on the master node of the cluster. As described herein, in some cases, using the SDN controller manager 1325 can enable disabling the service proxy (e.g., Kubernetes kube-proxy) of the slave node so that all container pool connections are implemented using the virtual router.

[0197] The components of network controller 24 can operate as a CNI for Kubernetes and can support multiple deployment modes. CNI 17 and CNI 750 are compute node interfaces for the entire CNI framework used to manage Kubernetes networks. Deployment modes can be divided into two categories: (1) an SDN architecture cluster that is a CNI integrated into the workload Kubernetes cluster; and (2) an SDN architecture cluster that is a CNI separate from the workload Kubernetes cluster.

[0198] Integrate with your workload Kubernetes cluster

[0199] Run the components of the network controller 24 (e.g., custom API server 301, custom resource controller 302, SDN controller manager 1325, and control node 232) in a management Kubernetes cluster on a master node close to the Kubernetes controller components. In this mode, the components of the network controller 24 are actually part of the same Kubernetes cluster as the workload.

[0200] Separate from workload Kubernetes cluster

[0201] Components of the network controller 24 are executed by a separate Kubernetes cluster from the workload Kubernetes cluster.

[0202] The SDN controller manager 1325 can use the controller framework for the orchestration platform to listen to (or otherwise monitor) changes to objects defined in the Kubernetes native API and add annotations to some of these objects. Annotations can be labels or other identifiers that indicate the characteristics of an object (e.g., "virtual network green"). The SDN controller manager 1325 is a component of the SDN architecture that listens to events related to Kubernetes kernel resources (such as container pools, network policies, services, etc.) and converts these events into custom resources configured by the SDN architecture as needed. CNI plugins (e.g., CNI 17, 570) are SDN architecture components that support the Kubernetes network plugin standard: the Container Network Interface.

[0203] The SDN controller manager 1325 can create a network solution for an application using the REST interface exposed by the aggregated API 402 to define network objects such as virtual networks, virtual network interfaces, and access control policies. For example, the network controller 24 component can implement a network solution in a computing infrastructure by configuring one or more virtual networks and virtual network interfaces in a virtual router. (This is just one example of an SDN configuration.)

[0204] The following example deployment configuration for this application consists of a container pool and its virtual network information:

[0205]

[0206]

[0207] This metadata information may be copied to each container pool replica created by the controller manager 1326. When the SDN controller manager 1325 is informed of these container pools, the SDN controller manager 1325 may create the virtual networks listed in the annotations ("red-network," "blue-network," and "default / extns-network" in the above embodiment) and create a virtual network interface for each container pool replica (e.g., container pool 202A) for each virtual network using a unique private virtual network address from the virtual network's cluster-wide address block (e.g., 10.0 / 16).

[0208] Additional techniques according to the present disclosure are described below. Contrail is an exemplary network controller architecture. Contrail CNI may be a CNI developed for Contrail. A cloud-native Contrail controller may be an example of a network controller described in this disclosure, such as network controller 24.

[0209] Figure 7A is a block diagram illustrating the control / routing planes of an underlay network and an overlay network configuration using an SDN architecture according to the techniques of this disclosure. Figure 7B is a block diagram illustrating a configured virtual network connecting a pool of containers using tunnels configured in an underlying network, according to the techniques of this disclosure.

[0210] The network controller 24 for the SDN architecture can use a distributed or centralized routing plane architecture. The SDN architecture can use containerized routing protocol daemons (processes).

[0211] In terms of network signaling, the routing plane can operate according to a distributed model, where cRPD runs on every compute node in the cluster. Essentially, this means that intelligence is built into the compute nodes and involves complex configuration at each node. In this model, route reflectors (RRs) may not make intelligent routing decisions, but instead act as relays that reflect routes between nodes. A distributed container routing protocol daemon (cRPD) is a routing protocol process that can be used, where each compute node runs its own instance of the routing daemon. At the same time, a centralized cRPD master instance can act as the RR to relay routing information between compute nodes. Routing and configuration intelligence are distributed across nodes using the RRs at a central location.

[0212] Alternatively, the routing plane can operate according to a more centralized model, where components of the network controller run centrally and absorb the intelligence required to process configuration information, build the network topology, and program the forwarding plane into virtual routers. Virtual router agents are local agents that process the information programmed by the network controller. This design results in more limited intelligence required at the compute nodes and tends to produce a simpler configuration state.

[0213] The centralized control plane provides the following:

[0214] Allows the proxy routing framework to be simpler and more lightweight. It hides the complexity and limitations of BGP from the proxy. The proxy does not need to understand concepts like route distinguishers, route targets, etc. The proxy simply exchanges prefixes and constructs the corresponding forwarding information.

[0215] There are indeed more control nodes than routes. Control nodes are built on the virtual network concept and can generate new routes using route replication and reorganization (e.g., supporting features like service chaining and intra-VN routing for other use cases).

[0216] Build a BUM tree for optimal broadcast and multicast forwarding.

[0217] It should be noted that the control plane has certain aspects of distributed nature. As a control plane that supports distributed functionality, it allows each local virtual router agent to publish its local routes and subscribe to configurations based on demand.

[0218] The following functions may be provided by the cRPD or control node of the network controller 24 .

[0219] Routing daemon / process

[0220] The control node and cRPD can act as routing daemons that implement different protocols and have the ability to program routing information in the forwarding plane.

[0221] cRPD implements routing protocols with a rich routing stack, including Interior Gateway Protocol (IGP) (e.g., Intermediate System to Intermediate System (IS-IS)), BGP-LU, BGP-CT, SR-MPLS / SRv6, Bidirectional Forwarding Detection (BFD), Path Computation Element Protocol (PCEP), etc. It can also be deployed to provide services such as route reflectors for the control plane only and is popular in Internet routing use cases due to these capabilities.

[0222] Control node 232 also implements routing protocols, but primarily based on BGP. Control node 232 understands overlay networks. Control node 232 offers a rich feature set when overlaying virtualization and caters to SDN use cases. Overlay features such as virtualization (abstraction using virtual networks) and service chaining are very popular among telecom and cloud providers. In some cases, cRPD may not include support for this overlay functionality. However, cRPD's rich feature set provides strong support for the underlying network.

[0223] Network Orchestration / Automation

[0224] Routing functionality is only part of the control node 232. An integral part of the overlay network is orchestration. In addition to providing overlay routing, the control node 232 helps model orchestration functionality and provides network automation. The core of the control node 232's orchestration capabilities is the ability to model network virtualization using abstractions based on virtual networks (and related objects). The control node 232 interfaces with the configuration node 230 to relay configuration information to the control plane and data plane. The control node 232 also helps build overlay trees for multicast Layer 2 and Layer 3. For example, the control node can build a virtual topology of its cluster for this purpose. Typically, cRPD does not include this orchestration capability.

[0225] High availability and horizontal scalability

[0226] The control node design is more centralized, while cRPD is more distributed. There are cRPD worker nodes running on each compute node. On the other hand, the control node 232 does not run on the compute and can even run on a remote cluster (i.e., independent and in some cases geographically far away from the workload cluster). The control node 232 also provides horizontal scalability for HA and operates in active-active mode. The computing load is shared between the control nodes 232. On the other hand, cRPD does not generally provide horizontal scalability. The control node 232 and cRPD can provide a smooth restart for HA and can allow data plane operation in headless mode - where the virtual router can operate even if the control plane restarts.

[0227] The control plane should be more than just a routing daemon. The control plane should support overlay routing and network orchestration / automation, and while cRPD is just as good as a routing protocol at managing underlay routing, cRPD generally lacks network orchestration capabilities and does not provide strong support for overlay routing.

[0228] Accordingly, in some examples, the SDN architecture can be 7A to 7B The compute nodes shown in Figure 4 have cRPDs on them. Figure 7A An SDN architecture 700 is shown that can represent an exemplary implementation of the SDN architecture 200 or 400. In the SDN architecture 700, the cRPD 324 runs on the compute nodes and provides underlying routing to the forwarding plane while running a centralized (and horizontally scalable) set of control nodes 232 that provide orchestration and overlay services. In some examples, instead of running the cRPD 324 on the compute nodes, a default gateway can be used.

[0229] The cRPD 324 on the compute node provides rich underlying routing to the forwarding plane by interacting with the virtual router agent 514 using an interface 540 (which can be a gRPC interface). The virtual router agent interface can allow programming of routes, configuration of virtual network interfaces for overlays, and other configuration of the virtual router 506. This is described in further detail in U.S. application Ser. No. 17 / 649,632. Simultaneously, one or more control nodes 232 operate as a separate container pool providing overlay services. Thus, the SDN architecture 700 can benefit from the rich overlay and orchestration provided by the control nodes 232 and the modern underlying routing provided by the cRPD 324 on the compute node, complementing the control nodes 232. A separate cRPD controller 720 can be used to configure the cRPD 324. The cRPD controller 720 can be a device / element management system, a network management system, an orchestrator, a user interface / CLI, or other controller. The cRPD 324 runs routing protocols and exchanges routing protocol messages with routers, including other cRPDs 324. Each cRPD 324 may be a centralized routing protocol process and, in effect, operates as a software-only version of the router control plane.

[0230] The enhanced underlay routing provided by cRPD 324 can replace the default gateway of the forwarding plane and provide a rich routing stack for supported use cases. In some examples where cRPD 324 is not used, virtual router 506 relies on the default gateway of the underlay routing. In some examples, cRPD 324, as an underlay routing process, is limited to programming only the default inet(6).0 structure with control plane routing information. In this example, non-default overlay VRFs can be programmed by control node 232.

[0231] 7A to 7B The dual routing / control plane solution described above is shown in Figure 7A In some aspects, cRPD 324 provides underlying routing / forwarding information to virtual router agent 514, similar to how a router control plane programs a router forwarding / data plane.

[0232] like Figure 7BAs shown in FIG, cRPD 324 exchanges routing information that can be used to create a tunnel through the underlying network 702 of the VRF. Tunnel 710 is an example and connects server 12A to the virtual router 506 of server 12X. Tunnel 710 can represent a segment routing (SR) or SRv6 tunnel, a generic routing encapsulation (GRE) tunnel, an IP-in-IP tunnel, an LSP, or other tunnels. Control node 232 uses tunnel 710 to create a virtual network 712 that connects server 12A, which is attached to the VRF of the virtual network, with the container pool 22 of server 12X.

[0233] As described above, cRPD 324 and virtual router agent 514 can exchange routing information using a gRPC interface, and virtual router agent 514 can use the gRPC interface to program virtual router 506 through configuration. It should also be noted that control node 232 can be used for overlay and orchestration, while cRPD 324 can be used to manage the underlying routing protocol. While using XMPP to communicate with the control node and domain name service (DNS), virtual router agent 514 can use a gRPC interface with cRPD 324.

[0234] Because there may be workers running on various compute nodes, the gRPC model performs well for cRPD 324. And the virtual router proxy 514 acts as a gRPC server that exposes the services of the client (cRPD 324) for programming routing and configuration information (underlying layer). Thus, when compared with XMPP, gRPC is an attractive solution. Specifically, gRPC transmits data as a binary stream and does not add overhead when encoding / decoding data sent through gRPC.

[0235] In some examples, the control node 232 can use XMPP to interface with the virtual router proxy 514. Since the virtual router proxy 514 acts as a gRPC server, the cRPD 324 acts as a gRPC client. This means that the client (cRPD) needs to initiate a connection to the server (vRouter proxy). In the SDN architecture 700, the virtual router proxy 514 selects the set of control nodes 232 to which it subscribes (because there are multiple control nodes). In this aspect, the control node 232 acts as a server and the virtual router proxy 514 connects as a client and subscribes for updates.

[0236] The control node 232 needs to select the virtual router agent 514 to which it needs to connect through gRPC and then subscribe as a client. Because the control node 232 does not run on each computing node, this requires implementing an algorithm for selecting the virtual router agent 514 to which it can subscribe. Furthermore, the control nodes 232 need to synchronize this information with each other. When a restart occurs, this also complicates the situation, and synchronization is required between the control nodes 232 to select the agent it serves. Features such as smooth restart (GP) and fast convergence have been implemented on top of XMPP. XMPP is already lightweight and efficient. Therefore, for communication from the control node 232 to the virtual router agent 514, XMPP may be better than gRPC.

[0237] The control nodes 232 and their use are further enhanced as follows. High availability (HA) and horizontal scalability require three control nodes. As with any routing platform, having only two control nodes 232 is sufficient to meet HA requirements. In many cases, this is advantageous. (However, one or more control nodes 232 can be used.) For example, it provides a more deterministic infrastructure and aligns with standard routing best practices. Each virtual router agent 514 is attached to a unique pair of control nodes 232 to avoid randomness. Using two control nodes 232 can simplify debugging. Furthermore, using only two control nodes 232 simplifies edge replication for building multicast / broadcast trees. Currently, because the vRouter agent 314 only connects to two of the three control nodes, all control nodes may not have a complete picture of the tree at any given time and rely on BGP to synchronize state between control nodes. Having three control nodes 232 exacerbates this situation because the virtual router agent 314 can randomly select two. If there were only two control nodes 232, each virtual router agent 314 would connect to the same control node. This in turn means that the control node 232 does not need to rely on BGP to synchronize state and has the same picture of the multicast tree.

[0238] The SDN architecture 200 can provide ingress replication as an alternative to edge replication and provide an option to the user. Ingress replication can be considered as a special degenerate case of a general overlay multicast tree. However, in practice, the signaling of the ingress replication tree is simpler than that of the general overlay multicast tree. With ingress replication, each virtual router 21 ends up with a tree with itself as the root and every other vrouter as a leaf. A failed virtual router 21 should theoretically not result in a rebuild of the tree. It should be noted that the performance of ingress replication deteriorates as the cluster becomes larger. However, ingress replication performs well for smaller clusters. Further, multicast is not a popular and common requirement for many customers. It is mainly limited to transmitting only the initial broadcast BUM traffic.

[0239] Configuration processing module enhancements

[0240] In a conventional SDN architecture, the network controller handles the orchestration of all use cases. Configuration nodes convert intents into configuration objects based on the data model and write them to a database (e.g., Cassandra). In some cases, notifications are simultaneously sent via RabbitMQ to all clients awaiting configuration.

[0241] The control node not only acts as a BGP speaker but also includes a configuration processing module that reads configuration objects from the database in the following manner. First, when the control node starts (or restarts), it connects to the database and reads the entire configuration directly from the database. Second, the control node can also be a message client. When there is an update to a configuration object, the control node receives a message notification listing the updated objects. This again causes the configuration processing module to read the object from the database.

[0242] The configuration processing module reads configuration objects for the control plane (BGP-related configuration) and the vRouter forwarding plane. The configuration can be stored as a graph with objects as nodes and relationships as links. This graph can then be downloaded to the client (BGP / cRPD and / or vRouter agent).

[0243] According to the techniques of this disclosure, in some examples, the conventionally configured API server and messaging service are replaced by the kubeapi-server (API server 300 and custom API server 301), and the previous Cassandra database is replaced by etcd in Kubernetes. Due to this change, clients interested in configuration objects can directly monitor the etcd database for updates, rather than relying on RabbitMQ notifications.

[0244] Controller Orchestration of CRPD

[0245] BGP configuration may be provided to cRPD 324. In some examples, cRPD controller 720 may be a Kubernetes controller that caters to a Kubernetes controller that develops its own controller that caters to the Kubernetes space and implements the CRDs needed to orchestrate and provision cRPD 324.

[0246] Distributed configuration processing

[0247] As mentioned earlier in this section, the configuration processing module can be part of the control node 232. The configuration processing module reads the configuration directly from the database, converts the data into JSON format, and stores it in its local IFMAP database as a graph with objects as nodes and relationships between them as links. This graph is then downloaded via XMPP to the interested virtual router agent 514 on the compute node. The virtual router agent 514 also locally builds an IFMAP-based dependency graph to store these objects.

[0248] By having the virtual router agent 514 monitor directly on the etcd server in the API server 300, the need for IFMAP as an intermediate module and the need to store the dependency graph can be avoided. The cRPD 324 running on the compute nodes can use the same model. This will avoid the need for an IFMAP-XMPP configuration channel. The Kubernetes configuration client (for control nodes 232) can be used as part of this configuration. The virtual router agent can also use this client.

[0249] However, this can increase the number of clients reading configuration from the etcd server, especially in clusters with hundreds of compute nodes. Adding more monitors eventually causes write rates to drop and event rates to fall below ideal values. etcd's gRPC proxy replays events from a single server monitor to multiple client monitors. The gRPC proxy consolidates multiple client monitors (c-monitors) for the same key or range into a single monitor (s-monitor) connected to the etcd server. The proxy broadcasts all events from the s-monitor to its c-monitor. Assuming N clients monitor the same key, a single gRPC proxy can reduce the monitor load on the etcd server from N to just one. Users can deploy multiple gRPC proxies to further distribute the server load. These clients share a single server monitor; the proxy effectively offloads resources from the core cluster. By adding proxies, etcd can serve over a million events per second.

[0250] DNS / Naming in SDN Architecture

[0251] In the previous architecture, contrail-dns and contrail-name daemons worked together to provide DNS services to VMs in the network. Name served as a DNS server that implemented the BIND protocol. contrail-dns received updates from vrouter-agent and pushed these records to Name.

[0252] The system supports four DNS modes, and IPAM configuration can select the required DNS mode.

[0253] 1. None - No DNS support for the VM.

[0254] 2. Default DNS Server - DNS resolution for the VM is performed based on the name server configuration in the server infrastructure. When the VM receives a DHCP response, the subnet's default gateway is configured as the VM's DNS server. DNS requests sent by the VM to this default gateway are resolved by the (fabric) name server configured on the corresponding compute node, and the response is sent back to the VM.

[0255] 3. Tenant DNS Server - This mode allows tenants to use their own DNS servers. A list of servers can be configured in IPAM, and then sent to VMs as DNS servers in DHCP responses. DNS requests sent by VMs are routed like any other data packets based on available routing information.

[0256] 4. Virtual DNS Server - In this mode, the system supports a virtual DNS server, providing a DNS server to resolve DNS requests from VMs. Those skilled in the art can define multiple virtual DNS servers for each domain in the system. Each virtual DNS server is the authoritative server for the configured DNS domain.

[0257] The SDN architecture described here is effective in the DNS services it provides. Customers in the cloud-native world benefit from a variety of DNS services. However, as we move to a next-generation Kubernetes-based architecture, the SDN architecture can replace core DNS with any DNS service.

[0258] Data plane

[0259] The data plane consists of two components: the virtual router agent 514 (also known as the proxy) and the virtual router forwarding plane 506 (also known as the DPDK vRouter / kernel vRouter). The agent 514 in the SDN architecture solution is responsible for managing the data plane components. The agent 514 establishes XMPP neighbor relationships with the two control nodes 232 and then exchanges routing information with the two control nodes 232. The vRouter agent 514 also dynamically generates flow entries and injects them into the virtual router 506. This provides instructions to the virtual router 506 on how to forward packets.

[0260] The responsibilities of the agent 514 may include: interfacing with the control node 232 to obtain configuration, converting the received configuration into a form that the data path can understand (e.g., converting a data model from IFMAP into a data model used by the data path), interfacing with the control node 232 to manage routing, and collecting statistics from the data path and exporting the statistics to a monitoring solution.

[0261] Virtual router 506 implements data plane functionality that allows virtual network interfaces to be associated with VRFs. Each VRF has its own forwarding and flow tables, while the MPLS and VXLAN tables are global within virtual router 506. The forwarding table can contain routes for the IP and MAC addresses of the destination, and IP-to-MAC associations are used to provide proxy ARP capabilities. Label values in the MPLS table are selected by virtual router 506 when a VM / container interface is started and are only locally important to that vRouter. The VXLAN network identifier is global across all VRFs of the same virtual network in different virtual routers 506 within a domain.

[0262] In some examples, each virtual network has a default gateway address assigned to it, and each VM or container interface receives this address in the DHCP response it receives when it is initialized. When a workload sends a packet to an address outside its subnet, the ARP for MAC will match the gateway's IP address, and virtual router 506 will respond with its own MAC address. In this way, virtual router 506 can support fully distributed default gateway functionality for all virtual networks.

[0263] The following is an example of packet flow forwarding implemented by virtual router 506.

[0264] Packet flow between VM / container interfaces in the same subnet.

[0265] The worker node can be a VM or a container interface. In some examples, packet processing is performed as follows:

[0266] The VMI / Container Interface needs to send a packet to VM2, so the virtual router 506 first queries its own DNS cache for the IP address, but since this is the first packet, there is no entry.

[0267] When its interface comes up, VM1 sends a DNS request to the DNS server address supplied in the DHCP response.

[0268] • The virtual router 506 intercepts the DNS request and forwards it to the DNS server running in the SDN architecture controller.

[0269] The DNS server in the controller responds with the IP address of VM2.

[0270] • Virtual router 506 sends the DNS response to VM1.

[0271] VM1 needs to form an Ethernet frame and therefore needs VM2's MAC address. VM1 checks its own ARP cache, but because this is the first packet, no entry exists.

[0272] VM1 sends an ARP request.

[0273] • Virtual router 506 intercepts the ARP request and looks up the MAC address of IP-VM2 in its own forwarding table and finds the association in the L2 / L3 routing that the controller sent for the MAC address of VM2.

[0274] Virtual router 506 sends an ARP reply to VM1 using VM2's MAC address.

[0275] A TCP timeout occurred in VM1's network stack.

[0276] VM1's network stack retries to send the packet, and this time, finds VM2's MAC address in the ARP cache and is able to form an Ethernet frame and send it out.

[0277] Virtual router 506 queries VM2's MAC address and finds the encapsulation route. Virtual router 506 constructs the outer header and sends the resulting packet to server S2.

[0278] The virtual router 506 on server S2 decapsulates the packet and looks up the MPLS label to identify the virtual interface to which to send the original Ethernet frame. The Ethernet frame is sent to the interface and is received by VM2.

[0279] Packet flow between VMs in different subnets

[0280] In some examples, the sequence for sending packets to destinations in different subnets is the same, except that virtual router 506 responds as the default gateway. VM1 sends a packet in an Ethernet frame with the MAC address of the default gateway, which was provided in the DHCP response from virtual router 506 when VM1 booted. When VM1 makes an ARP request for the gateway IP address, virtual router 506 responds with its own MAC address. When VM1 sends an Ethernet frame using the gateway MAC address, virtual router 506 uses the destination IP address of the packet in the frame to look up the forwarding table in the VRF to find a route to the host running the destination through the encapsulation tunnel.

[0281] Figure 10 is a block diagram illustrating a multi-cluster deployment for a cloud-native SDN architecture according to the techniques of the present disclosure. As used herein, the term "cluster" may refer to a Kubernetes cluster or other terms for similar structures in an orchestration platform for implementing the SDN architecture described in this disclosure.

[0282] In a multi-cluster deployment of the SDN architecture 1000, configuration nodes and control nodes are deployed to a central cluster 902 and centrally manage the configuration and control of multiple distributed workload clusters 930-1 to 930-N (collectively referred to as "workload clusters 930"). However, the data plane is distributed between the workload clusters 930. As described with reference to other SDN architectures in this disclosure, workload clusters 930 and each cluster in the central cluster 902 use similar component microservices to implement the data plane of the cluster. For example, workload cluster 930-1 includes virtual routers and virtual router agents deployed to the compute nodes that make up the cluster. The compute nodes of workload cluster 930-1 may also include CNI, orchestration agents, and / or other components described with reference to the server 12 and other compute nodes described elsewhere in this disclosure. Workload cluster 930-1 also includes an API server 300-1 for creating and managing local resources of the orchestration platform. The API server 300-1 may be connected to Figure 3 Kubernetes is an orchestration platform. In addition to the API server 300-1 (kube-api server), other orchestration platform components may include kube-scheduler, kube-controller-manager, and kubelet. Configuration storage 920-1 may be similar to Figure 3 Similar to one or more configuration stores 304 in .

[0283] Workload clusters 930 can be geographically distributed to / toward the edge of a network, such as an edge or micro data center, while central cluster 902 can be geographically consolidated / centralized, such as in a regional data center or cloud provider. The compute nodes of each workload cluster 930 can be distinct from the compute nodes of central cluster 902, and the compute nodes between the clusters do not overlap.

[0284] In a multi-cluster mode for a multi-cluster deployment of the SDN architecture 1000, the central cluster 902 runs a complete orchestration platform installation including the API server 300-C and other orchestration platform components (which may be similar to those described above with reference to the workload cluster 930-1). The central cluster 902 also runs the configuration plane and control plane component microservices for the network controller of the SDN architecture. Specifically, the configuration node 230 includes one or more instances of each of the API server 300-C, the custom API server 301, the custom resource controller 302, and the SDN controller manager 303-C. These may be Figure 3 and Figure 4 and similar like-named components.

[0285] API server 300-C and custom API server 301 form an aggregate API server for SDN architecture resources (virtual networks, VM interfaces, routing instances, etc.). Custom API server 301 can be registered as an APIService with API server 300-C. As described above, API server 300-C receives requests for SDN architecture resources and forwards them to custom API server 301, which then performs operations on the custom resources configured in the SDN architecture.

[0286] The custom resource controller 302 of the central cluster 902 implements the business logic of the custom resources configured by the SDN architecture. The custom resource controller 302 converts user intent into low-level resources consumed by the control node 232.

[0287] One or more instances of control node 232 operate as SDN architecture control nodes for managing the data planes of central cluster 902, workload cluster 930-1, and workload cluster 930-N. Central cluster 902 includes virtual router 910-C deployed to the compute nodes of central cluster 902, workload cluster 930-1 includes virtual router 910-1 deployed to the compute nodes of workload cluster 930-1, and workload cluster 930-N includes virtual router 910-N deployed to the compute nodes of workload cluster 930-N. For ease of illustration, virtual routers located on independent compute nodes are not shown. As described in detail elsewhere in this disclosure, different instances of virtual routers run on the compute nodes of corresponding clusters. For example, each of the compute nodes can be a server 12.

[0288] Virtual network 1012 represents one or more virtual networks that can be configured by configuration node 230 and control node 232. Virtual network 1012 can connect workloads running in different workload clusters 930 as well as in the same workload cluster. Workloads in different virtual networks or in the same virtual network can communicate across workload clusters 930 as well as within the same workload cluster.

[0289] Central cluster 902 includes corresponding SDN controller managers 303-1–303-N (collectively, “SDN controller managers 303”) for workload cluster 930. SDN controller manager 303-1 is the interface between the native resources of the orchestration platform (e.g., services, namespaces, container pools, network policies, network attachment definitions) and the custom resources configured by the SDN architecture, and more specifically, between the native resources of the orchestration platform and the custom resources configured by workload cluster 930-1.

[0290] The SDN controller manager 303-1 monitors the API server 300-1 of the workload cluster 930-1 for changes in the local resources of the orchestration platform of the workload cluster 930-1. The SDN controller manager 303-1 also monitors the custom API server 301 of the central cluster 902. This is called "dual monitoring". For example, to implement dual monitoring, the SDN controller manager 303 can use the theadmiralty multicluster-controller go library or a multi-cluster manager implementation provided by the Kubernetes community, each of which supports the function of monitoring resources in multiple clusters. Due to dual monitoring, the SDN controller manager 303-1 performs operations on custom resources whether initiated at the API server 300-1 of the workload cluster 930-1 or at the 300-C / custom API server 301. In other words, custom resources for the central cluster 902 may be created: (1) directly or interactively through user or agent interaction with the configuration node 230; or (2) indirectly through events caused by an orchestration platform local resource operation of a workload cluster 930 and detected by a responsible SDN controller manager 303, which may responsively create custom resources in the configuration store 920-C using the custom API server 301 to implement local resources for the workload cluster.

[0291] The SDN controller manager 303-N operates similarly with respect to monitoring and configuring workload clusters 930-N. The SDN controller manager 303-C running on the central cluster 902 operates similarly, but does not monitor the individual API servers in the workload clusters because it is only responsible for interfacing the local resources of the API server 300 with the custom resources configured in the central cluster 902 (e.g., virtual router 910-C).

[0292] In the multi-cluster mode of a multi-cluster deployment of the SDN architecture 1000, each distributed workload cluster 930 is associated with a central cluster 902 via a dedicated SDN controller manager 303. The SDN controller manager 303 runs on the central cluster 902 to facilitate better life cycle management (LCM) of the SDN controller manager 303, configuration nodes 230, and control nodes 232, and to facilitate better and more manageable handling of security and permissions by consolidating these tasks into a single central processor 902.

[0293] As described elsewhere in this disclosure, the virtual router agent of virtual router 910 communicates with control node 232 to obtain routing and configuration information. For example, when an orchestration platform-local resource such as a container pool or service is created in workload cluster 930-1, SDN controller manager 303-1 running in central cluster 902 receives an indication of the creation event and its coordinator can create / update / delete custom resources configured in the SDN architecture, such as virtual machines, virtual machine interfaces, and instance IPs. In addition, SDN controller manager 303-1 can associate these new custom resources with a virtual network for the container pool or service. This virtual network can be the default virtual network or a virtual network indicated in the manifest of the container pool or service (user annotation).

[0294] Custom resources can have namespace scopes across different clusters. Custom resources associated with one workload cluster 930 will have a corresponding namespace and cluster identifier (e.g., cluster name or unique identifier) created in the central cluster 902, and the custom resource will be created based on that namespace. Cluster-scoped custom resources can be stored with a naming convention such as clustername-resourcename-unique identifier. The unique identifier can be a hash of the cluster name, the resource's namespace, and the resource name.

[0295] Using resources appropriately associated with the corresponding workload cluster 930, for example, using the cluster identifier described above, the SDN controller manager 303 authenticates the user or agent and allows the user or agent to use only custom resources for the SDN architecture configuration that belong to the namespace associated with the workload cluster. For example, if a user attempts to create a container pool in workload cluster 930-1 by sending a request to API server 300-1, and the container pool manifest annotation specifies a specific virtual network "VN1," the SDN controller manager 303-1 for workload cluster 930-1 will verify the request by determining whether "VN1" belongs to the namespace associated with workload cluster 930-1. If so, the SDN controller manager 303-1 creates the custom resources for the SDN architecture configuration using the custom API server 301. The control node 232 configures the configuration object for the new custom resources in workload cluster 930-1. If not, the SDN controller manager 303-1 can delete the resource using API server 300-1.

[0296] A user or agent can create a custom resource configured for the SDN architecture in a namespace using a NetworkAttachmentDefinition. In this case, the custom resource controller 302 for the NetworkAttachmentDefinition custom resource can create a virtual network in the namespace with a name prepended with the associated cluster identifier. For example, a virtual network can be created in the namespace clustername-namespace. As described above, this allows the SDN controller manager 303 to perform validation when the container pool is annotated with the listed virtual network.

[0297] In some embodiments, predicates can be implemented to filter events generated due to changes in custom resources configured by the SDN architecture of other workload clusters 930. The coordinator of the SDN controller manager 303 of the workload cluster only needs to process events related to resources owned by the specific workload cluster (i.e., namespace and cluster scope).

[0298] In some examples, each workload cluster 930 can have its own default Kubernetes container pool and service subnet, which should not overlap with the container pools and service subnets of other workload clusters 930. In some cases, although it does not support network address translation (NAT), Kubernetes network policies with routing can be supported. In some examples, there are no container pools or service networks across workload clusters. Each workload cluster can have its own pool. The Kubernetes administrator should set up the cluster accordingly.

[0299] Virtual networks are bound to workload clusters 930 using namespaces and stitched together with network policies / network routers, as shown by virtual network 1012. However, there is no common virtual network that extends across clusters. Container pools or container pool pings across workload clusters 930, located on the default container pool network interface and on secondary virtual network interfaces, should work together, as long as the virtual network subnets are distinct and routing targets are shared between virtual networks.

[0300] Below is an example workflow for an SDN controller manager 303-1 to start a container pool for workload cluster 930-1 (named "Cluster 1" in these examples). Other SDN controller managers 303 can operate similarly with respect to the central cluster 902 and their dedicated workload clusters 930. NetworkAttachmentDefinition is a CRD scheme specified by the Kubernetes Network Plumbing Working Group to express the intention to attach a container pool to one or more logical or physical networks. The NetworkAttachmentDefinition specification typically defines the desired state of a network attachment for a secondary interface of a container pool. The container pool can then be started using a text annotation and the Kubernetes CNI attaches the network's interface to the container pool. As described elsewhere in this disclosure, the SDN architecture can operate as a CNI and attach virtual network interfaces of a virtual network to a container pool based on the NAD and container pool manifest annotations.

[0301] 1. A user creates a NetworkAttachmentDefinition (NAD) in the distributed cluster “Cluster 1” (e.g., workload cluster 930-1).

[0302]

[0303] The SDN controller manager 303-1 listens to the creation event of the NAD and then:

[0304] a. Check if the namespace "cluster1-ns1" exists. If not, the SDN controller manager 303-1 creates the namespace in the central cluster using the following:

[0305]

[0306] b. The SDN controller manager 303-1 creates a VirtualNetwork in the namespace:

[0307]

[0308]

[0309] 2. The user creates a container pool in the distributed cluster "cluster1" with the default container pool network

[0310]

[0311] The SDN controller manager 303 - 1 monitors container pool creation events and in turn:

[0312] a. Check if the cluster1_default namespace exists, and if not, create the cluster1_default namespace.

[0313] b. Create the following resources in the central cluster:

[0314]

[0315]

[0316] 3. User creates a container pool in distributed cluster "cluster1" with VN ns1 / vn1

[0317]

[0318] The SDN controller manager 303 - 1 monitors container pool creation events and in turn:

[0319] a. Check if a virtual network exists in namespace cluster1-ns1 with name vn1. If so, continue.

[0320] b. Create the following resources in the central cluster:

[0321]

[0322] Virtual Machine Interface

[0323] kind:VirtualMachineInterface

[0324] metadata:

[0325] name:vmiName

[0326] namespace:cluster1-ns1

[0327] spec:

[0328] ·InstanceIP

[0329] kind:InstanceIP

[0330] metadata:

[0331] name:cluster1-<Pod name> -hash

[0332] spec:

[0333] Life Cycle Management

[0334] For the deployer in the central cluster 902, users can manually create a secret in the central cluster 902 using the workload cluster's kubeconfig file. The name of the created secret is passed as the value to the kubeconfigSecretName string defined in the CRD of the SDN controller manager 303-C. The deployer will package the data available in the kubeconfigSecretName and pass it to the workload cluster's SDN controller manager 303. If the kubeconfigSecretName value is provided to the deployer, it is assumed that the SDN controller manager 303 is installed in multi-cluster mode. In multi-cluster mode, the deployer will obtain the PodSubnet and ServiceSubnet from the workload cluster's kubeadmConfigMap. Each workload cluster 930 should have a unique name. To achieve this, the LCM will use the custom resource name of the corresponding SDN controller manager 303 as the cluster name of the workload cluster 930. Because there are multiple deployed instances of the SDN controller manager 303 in the central cluster 902, the deployment name of each SDN controller manager 303 must be unique. For example, to achieve this, a deployment name of the SDN controller manager 303 is created based on the custom resource name of the SDN controller manager 303 as a [Prefix]-k8s-Kubemanager-kubemanager CR name.

[0335] For example, for a deployer in workload cluster 930-1, a user can manually create a Secret in workload cluster 930-1 using the kubeconfig file of central cluster 902. Then, during the deployment process of workload cluster 930-1, the Secret created above is packaged and passed to the controller of the virtual router 910 component via the Deployment yaml. The controller uses the context of central cluster 902 to access or modify SDN architecture resources and uses the context within the cluster to access Kubernetes resources. An assertion can be added to the controller of the virtual router 910 component to find the SDN controller manager 303 related only to that specific workload cluster, for example, the SDN controller manager 303-1 of workload cluster 930-1. The user can configure the IP and XMPP port of one or more control nodes 232 for custom resources of the virtual router 910. When the custom resource is applied, the deployer will connect the workload cluster's virtual router 910 (eg, an instance of virtual router 910 - 1 ) with the appropriate control node 232 of the central cluster 902 .

[0336] In other words, the core components of the SDN architecture 1000 are divided into three planes: the configuration plane, the control plane, and the data plane. The configuration plane includes components of the configuration node 230, including the SDN controller manager 303-C, the custom API server 301, and the custom resource controller 302 (these are respectively referred to as contrail-k8s-kubemanager, contrail-k8s-apiserver, and contrail-k8s-controller, where the SDN architecture 1000 is based on Contrail). The control plane includes the control node 232 component (referred to as contrail-control, where the SDN architecture 1000 is based on Contrail). The data plane includes the virtual router 910 component (referred to as contrail-vrouter, where the SDN architecture 1000 is based on Contrail).

[0337] The data plane components are installed in the workload cluster 930, and the control plane and configuration components are run in the central cluster 902. Configuration data of the SDN controller manager 303, control nodes 232 (such as SMPP ports and control node IP, Pod CIDR), and virtual router 910 components may need to be synchronized between different clusters:

[0338] 1. When the configuration data is updated;

[0339] 2. When adding a workload cluster to an existing Multicluster setup;

[0340] 3. When a change occurs in the control node 232 of an existing Multicluster configuration.

[0341] To resolve this synchronization issue:

[0342] 1. Shared configuration data of the SDN controller manager 303, control node 232, and virtual router 910 components between clusters via installed ConfigMaps.

[0343] 2. The SDN controller manager 303 instance will collect data about the control node 232 and create a map of its associated workload cluster 930 and install its container pool of virtual routers 910 for use.

[0344] 3. If there is a new workload cluster added, the related SDN controller manager 303 will synchronize the data of the control node 232 with the resources of its virtual router 910 via ConfigMap.

[0345] 4. If the control node 232 changes, the affected SDN controller manager 303 will update the resources of its virtual router 910 , and the affected virtual router container pool will be restarted to connect to the new control node 232 .

[0346] Role-based access control

[0347] In central cluster 902, controllers running in central cluster 902 can use a single SDN architecture ServiceAccount defined by the deployer. This ServiceAccount grants full access permissions to resources in central cluster 902. Because there are many cluster-wide SDN architecture custom resources and controllers running in central cluster 902 can access most of these custom resources, this access cannot be restricted.

[0348] In workload cluster 930, users can create a cluster role with restricted read / write access to resources in the workload cluster. Users can create a ServiceAccount and a Cluster-role binding to connect the cluster role and the ServiceAccount. Users can manually generate a kubeconfig file using the token generated for the ServiceAccount above. As shown above, the central cluster 902 uses this kubeconfig file to communicate with the workload cluster.

[0349] The following are examples of Cluster-Role, Service Account, and Cluster-Role Binding:

[0350]

[0351] Test scenario

[0352] Verify that the deployer is able to configure the SDN controller manager 303-C in single cluster mode or multi-cluster mode.

[0353] Verify that the user can create a container pool in the workload cluster 930 and the corresponding SDN infrastructure resources in the central cluster 902 .

[0354] Verify that the user can create a Service (ClusterIP and NodePort) in the workload cluster 930 and the corresponding SDN infrastructure resources in the central cluster 902.

[0355] Verify that the user can delete a Pod or Service in the workload cluster 930 and can delete the reference to the corresponding SDN architecture resource in the central cluster 902 .

[0356] Verify the SDN controller manager 303 coordinator functionality to automatically correct for lost resources in the SDN architecture.

[0357] Verify that container pool network status annotation updates occur in the container pool, as each VirtualNetwork is used.

[0358] Verify connectivity across container pools within a distributed cluster using different Virtual Networks with a common RouteTarget.

[0359] Verify container pool connectivity between clusters using different Virtual Networks with a common RouteTarget.

[0360] Verify that the user can create a Kubernetes network policy in the workload cluster 930 and a corresponding SDN architecture configuration firewall policy object in the central cluster 902.

[0361] Each cluster in a multi-cluster deployment should have non-overlapping Service CIDRs. Due to identical Service CIDRs, clusters will not have connectivity from the vRouter (data path) to an appropriate API server 300 and may not be able to provide services properly.

[0362] Each cluster in a multi-cluster deployment should have non-overlapping Pod CIDRs. Clusters with overlapping Pod CIDRs can cause problems. For example, consider a scenario where a container pool in each workload cluster 930 has the same container pool IP. The two container pools cannot communicate with each other. When a service is created in one workload cluster 930 using the container pool as an endpoint, two routes exist in BGP: one for each container pool in workload cluster 930, and traffic across the two container pools will be ECMP traffic, which is undesirable.

[0363] Figure 11 1100 is a flowchart illustrating an exemplary mode of operation for a multi-cluster deployment of an SDN architecture. In the mode of operation 1100, a network controller for a software-defined network (SDN) architecture system includes: processing circuitry of a central cluster of a first one or more compute nodes; a configuration node configured to be executed by the processing circuitry; and a control node configured to be executed by the processing circuitry. The configuration node of the network controller includes a custom API server. The custom API server processes requests for operations on custom resources configured with respect to the SDN architecture (1102). Each custom resource configured with respect to the SDN architecture corresponds to a type of configuration object in the SDN architecture system. In response to detection of an event for an instance of a first custom resource in the custom resources, the control node of the network controller obtains configuration data for the instance of the first custom resource and configures the corresponding instance of the configuration object in the workload cluster of the second one or more compute nodes (1104). The first one or more compute nodes in the central cluster are distinct from the second one or more compute nodes in the workload cluster.

[0364] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete, but interoperable, logic devices or other hardware devices. In some cases, the various features of the electronic circuitry may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0365] If implemented in hardware, the present disclosure may relate to an apparatus such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively, or in addition, if implemented in software or firmware, the techniques may be implemented at least in part via a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store the instructions to be executed by the processor.

[0366] Computer-readable media may constitute part of a computer program product including packaging materials. Computer-readable media may include computer data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0367] In some examples, computer-readable storage media may include non-volatile media. The term "non-volatile" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In a specific example, a non-volatile storage medium may store data that changes over time (e.g., in RAM or cache).

[0368] The code or instructions may be software and / or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, as used herein, the term "processor" may refer to any of the aforementioned structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure may be provided within software modules or hardware modules.

Claims

1. A network controller for a software-defined network (SDN) architecture system, the network controller comprising: processing circuitry of a central cluster of one or more computing nodes; a configuration node configured to be executed by the processing circuit; as well as a control node configured to be executed by the processing circuit; Wherein, the configuration node includes a custom application programming interface API server to process requests for operations on custom resources configured in the SDN architecture; wherein each of the custom resources configured in the SDN architecture corresponds to a type of configuration object in the SDN architecture system; and Wherein, in response to detecting an event of an instance of a first custom resource among the custom resources, the control node is configured to obtain configuration data of the instance of the first custom resource and configure a corresponding instance of the configuration object in a workload cluster of a second one or more computing nodes, wherein the first one or more computing nodes in the central cluster are different from the second one or more computing nodes in the workload cluster.

2. The network controller according to claim 1, in, The instance of the first custom resource is a first instance and the workload cluster is a first workload cluster; Wherein, in response to detecting an event of a second instance of the first custom resource, the control node is configured to obtain configuration data of the second instance of the first custom resource and configure a corresponding instance of a configuration object in a second workload cluster of one or more third computing nodes, wherein the first one or more computing nodes, the second one or more computing nodes, and the third one or more computing nodes are different.

3. The network controller according to claim 1 , further comprising: a first SDN controller manager configured to be executed by the processing circuit; as well as a second SDN controller manager configured to be executed by the processing circuit; wherein the first SDN controller manager is configured to monitor events of instances of a first resource in the central cluster and to monitor events of instances of resources in a first workload cluster; and The second SDN controller manager is configured to monitor events of instances of a second resource in the central cluster and to monitor events of instances of resources in a second workload cluster.

4. The network controller according to claim 1, further comprising: an SDN controller manager configured to be executed by the processing circuit; The SDN controller manager is configured to monitor events of instances of resources in the central cluster and to monitor events of instances of resources in the workload cluster.

5. The network controller according to claim 1, an SDN controller manager configured to be executed by the processing circuit; in, In response to detecting a creation event of an instance of a local resource of the container orchestration system of the workload cluster, the SDN controller manager is configured to create a custom resource for the SDN architecture configuration in the central cluster using a custom API server. The network controller according to claim 5 , wherein: The event of the instance of the first custom resource includes a creation event of the instance of the first custom resource.

7. A software-defined networking method, comprising: a custom application programming interface (API) server implemented by a configuration node of a network controller for a software-defined network (SDN) architecture system, processing a request for an operation on a custom resource configured in the SDN architecture, wherein each of the custom resources configured in the SDN architecture corresponds to a type of configuration object in the SDN architecture system, wherein the network controller operates on a first central cluster of one or more computing nodes; The control node of the network controller detects an event of an instance of a first custom resource among the custom resources; and In response to detecting the event of the instance of the first custom resource, the control node obtains configuration data of the instance of the first custom resource and configures corresponding instances of the configuration object in a workload cluster of one or more second computing nodes, wherein the first one or more computing nodes of the central cluster are different from the second one or more computing nodes of the workload cluster.

8. The method according to claim 7, in, The instance of the first custom resource is a first instance and the workload cluster is a first workload cluster, the method further comprising: In response to detecting an event of a second instance of the first custom resource, the control node obtains configuration data of the second instance of the first custom resource and configures a corresponding instance of the configuration object in a second workload cluster of one or more third computing nodes, wherein the first one or more computing nodes, the second one or more computing nodes, and the third one or more computing nodes are different.

9. The method according to any one of claims 7 to 8, further comprising: Monitoring, by a first SDN controller manager, events of instances of a first resource in the central cluster and events of instances of resources in a first workload cluster; and The second SDN controller manager monitors events of the instance of the second resource in the central cluster and monitors events of the instance of the resource in the second workload cluster.

10. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 7 to 9.

Images