Virtual network router for cloud-native software-defined network architectures

Abstract

The present disclosure relates to a virtual network router for a cloud-native software-defined network architecture. In general, techniques for creating a virtual network router within a software-defined network (SDN) architecture are described. A network controller for an SDN architecture system may include a processing circuit configured to execute a configuration node and a control node. The configuration node may process a request to create a virtual network router (VNR), wherein the virtual network router may cause the network controller to interconnect a first virtual network (VN) with a second VN. The VNR may represent a logical abstraction of one or more policies that cause the import and / or export of routing information between a first VN and a second VN. The control node configures the first VN and the second VN according to the one or more policies so that routing information is imported and / or exported between the first VN and the second VN via the VNR.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Patent Application No. 17 / 809,659 filed on June 29, 2022, U.S. Provisional Patent Application No. 63 / 364,283 filed on May 6, 2022, and Indian Provisional Patent Application No. 202141044924 filed on October 4, 2021, the entire contents of each application are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a virtualized computing infrastructure, and more particularly, to a cloud-native network. Background Art

[0004] In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and / or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More sophisticated data centers provide user support equipment located in various physical hosting facilities for infrastructures located throughout the world.

[0005] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have a widely used virtualized environment in which virtual hosts such as virtual machines or containers, also referred to herein as virtual runtime elements, are deployed on and run on the underlying computing platforms of physical computing devices.

[0006] Virtualization within a data center or any environment including one or more servers can provide several advantages. One advantage is that virtualization can provide significantly improved efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. A second advantage is that virtualization provides effective control over the computing infrastructure. As physical computing resources in computing environments such as cloud-based computing become replaceable resources, the provision and management of computing infrastructure becomes easier. As a result, enterprise IT staff often prefer virtualized computing clusters in data centers due to their management advantages in addition to the efficiency and increased return on investment (ROI) provided by virtualization.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable runtime elements for applications that are isolated from each other and from the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and run as a single lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers provide a guarantee of continuous operation from one computing environment to another virtual or physical environment.

[0008] Due to the inherent lightweight nature of containers, a single host can typically support more container instances than traditional virtual machines (VMs). Typically, containers that are short-lived (compared to most VMs) can be created and moved more efficiently than VMs, and containers can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the demand for container networking solutions: the network should be agile and scalable. In the same computing environment, VMs, containers, and bare metal servers may need to coexist and enable communication between multiple deployed applications. Container networking should also not be aware of working with multiple types of orchestration platforms for deploying containerized applications.

[0009] Managing the deployment and infrastructure of computing infrastructure for application execution may involve two main roles: (1) Orchestration - used to automate the deployment, scaling, and operation of applications across a cluster of hosts and to provide computing infrastructure, which may include container-centric computing infrastructure; and (2) Network management - used to create virtual networks in the network infrastructure to enable packetized communications between applications running in virtual execution environments such as containers or VMs and between applications running in legacy (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the invention

[0010] In summary, technology for cloud-native software-defined networking (SDN) architecture is described. In some embodiments, the SDN architecture may include data plane elements implemented in computing nodes, and network devices such as routers or switches, and the SDN architecture may also include a network controller for creating and managing virtual networks. The SDN architecture configuration and control plane are designed to be cloud-native software with extensions of a container-based microservice architecture that supports upgrades in services. Custom resources can be exposed by implementing a configuration node for the configuration plane. These custom resources for SDN architecture configuration may include configuration elements that are typically exposed by a network controller, but configuration elements may be merged with Kubernetes local / built-in resources to support a unified expected model exposed by an aggregated API layer and implemented by a Kubernetes controller and a custom resource controller, and the custom resource controller is dedicated to coordinating the actual state of the SDN architecture with the expected state.

[0011] The custom resources configured for the SDN architecture include a virtual network router, which represents a logical abstraction for interconnecting and communicating between virtual networks implemented by the SDN architecture. Instances of the virtual network router custom resource facilitate the exchange (referring to asymmetric or symmetric import and export) of routing information using import and export routing targets configured within the routing instance for the corresponding virtual network. Therefore, in practice, the virtual network router enables routing leakage between virtual network resources defined for the SDN architecture. Because the virtual network router resource hides the complex details of the virtual routing and forwarding instance (VRF or "routing instance") configuration between VRFs used to implement various virtual network resources, the technology provides an intuitive, expectation-based model for defining the interconnection between virtual network resources configured in the SDN architecture and the corresponding virtual networks and even improves the usability of people who are not familiar with the border gateway protocol and VRF configuration.

[0012] The technology may provide one or more additional technical advantages. For example, a cloud-native SDN architecture may address limitations in conventional SDN architectures related to complexity of lifecycle management, highly mandatory resource analysis components, scale limitations in configuration management, and lack of command line interface (CLI)-based interfaces. For example, a network controller for an SDN architecture is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrades of the various component microservices of the configuration node and control node of the configuration and control plane.

[0013] The technology can further enable optional cloud-native monitoring (telemetry) and user interfaces, high-performance data planes for containers using a data plane development kit (based on DPDK) virtual router connected to a DPDK-enabled container pool, and cloud-native configuration management that leverages the configuration framework of existing orchestration platforms such as Kubernetes or Openstack in some cases. As a cloud-native architecture, the network controller is scalable and elastic to address and support multiple clusters. In some cases, the network controller can also support the scalability and performance requirements of key performance indicators (KPIs).

[0014] In one embodiment, various aspects of the technology relate to a network controller for a software defined network (SDN) architecture system, the network controller comprising: a processing circuit; a configuration node configured to be executed by the processing circuit; and a control node configured to be executed by the processing circuit; wherein the configuration node is executable to process a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within the SDN architecture system, the virtual network router representing a logical abstraction of one or more policies, the one or more policies causing one or more of the importing and exporting of routing information between the first virtual network and the second virtual network to occur; and wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of the importing and exporting of routing information between the first virtual network and the second virtual network via the virtual network router.

[0015] In another embodiment, various aspects of the technology relate to a method, including: processing, by a network controller, a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within an SDN architecture system, the virtual network router representing a logical abstraction of one or more policies, the one or more policies causing one or more of the importing and exporting of routing information between the first virtual network and the second virtual network to occur; and configuring, by the network controller, the first virtual network and the second virtual network according to the one or more policies to enable one or more of the importing and exporting of routing information between the first virtual network and the second virtual network via the virtual network router.

[0016] In another embodiment, various aspects of the technology relate to a non-transitory computer-readable medium including instructions, the instructions being used to cause a processing circuit of a network controller to: process a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within an SDN architecture system, the virtual network router representing a logical abstraction of one or more policies that cause one or more of importing and exporting routing information between the first virtual network and the second virtual network to occur; and configure the first virtual network and the second virtual network according to the one or more policies to enable one or more of importing and exporting routing information between the first virtual network and the second virtual network via the virtual network router.

[0017] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram illustrating an exemplary computing infrastructure in which embodiments of the techniques described herein may be implemented.

[0019] Figure 2 is a block diagram illustrating an embodiment of a cloud-native SDN architecture for a cloud-native network in accordance with the techniques of this disclosure.

[0020] Figure 3 is a block diagram illustrating another view of components of an SDN architecture in further detail according to techniques of this disclosure.

[0021] Figure 4 is a block diagram illustrating example components of an SDN architecture in accordance with techniques of this disclosure.

[0022] Figure 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure.

[0023] Figure 6 is a block diagram of an exemplary computing device operating as a computing node for one or more clusters of an SDN architecture system in accordance with the techniques of this disclosure.

[0024] Fig. 7A is a block diagram illustrating a control / routing plane for underlay and overlay network configuration using an SDN architecture in accordance with the techniques of this disclosure.

[0025] Figure 7B is a block diagram illustrating a configured virtual network connecting a pool of containers using tunnels configured in an underlying network in accordance with the techniques of this disclosure.

[0026] Figure 8 is a block diagram illustrating an example of a custom controller for custom resources configured for an SDN architecture according to the techniques of this disclosure.

[0027] Fig. 9 is a block diagram illustrating an exemplary flow of creating, monitoring, and coordinating between custom resource types that depend on different custom resource types.

[0028] Fig.10 , Fig.11 , Fig. 12A , Fig. 12B , Fig.13A , Fig. 13B , Fig.14 , Fig.15 and Fig.16 is a diagram illustrating various instances in which a virtual network router may be configured to implement interconnections between virtual networks in accordance with various aspects of the techniques described in this disclosure.

[0029] Fig.17A , Fig. 17B , Fig. 17C and Fig.17D is a flow diagram illustrating operations performed when establishing one or more virtual network routers within an SDN architecture in accordance with various aspects of the techniques described in this disclosure.

[0030] Fig.18 is a diagram showing various aspects of implementing the techniques described herein. Figure 1 Another flow chart of the operation of the computer architecture shown in an embodiment.

[0031] Like reference numerals identify like elements throughout the description and drawings. DETAILED DESCRIPTION

[0032] Figure 1is a block diagram of an exemplary computing infrastructure 8 that illustrates an example of the techniques described herein. Current implementations of software defined network (SDN) architectures for virtual networks present challenges to cloud-native adoption due to, for example, complexity of lifecycle management, high mandatory nature of resource analysis components, limited scalability of configuration modules, and no command line interface (CLI)-based (e.g., kubectl) interfaces. The computing infrastructure 8 includes a cloud-native SDN architecture system described herein that addresses these challenges and modernizes the telecom cloud-native era. Exemplary use cases for cloud-native SDN architectures include 5G mobile networks and cloud and enterprise cloud-native use cases. The SDN architecture may include data plane elements implemented in computing nodes (e.g., servers 12) and network devices such as routers or switches, and the SDN architecture may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. The SDN architecture configuration and control plane are designed as a scale-out cloud-native software with a container-based microservices architecture that supports hot upgrades.

[0033] Therefore, SDN architecture components are microservices, and in contrast to existing network controllers, the SDN architecture assumes an underlying container orchestration platform that manages the lifecycle of SDN architecture components. The container orchestration platform is used to build SDN architecture components; the SDN architecture uses cloud-native monitoring tools that can be integrated with cloud-native options provided by customers; the SDN architecture uses an aggregated API of SDN architecture objects to provide a declarative way of resources (i.e., custom resources). SDN architecture upgrades can follow a cloud-native model, and the SDN architecture can leverage Kubernetes constructs such as Multus, Authentication & Authorization, Cluster API, KubeFederation, KubeVirt, and Kata containers. The SDN architecture can support a pool of Data Plane Development Kit (DPDK) containers, and the SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies.

[0034] For service providers and enterprises, the SDN architecture automates network resource configuration and orchestration to dynamically create highly scalable virtual networks and link virtualized network functions (VNFs) with physical physical functions (PNFs) to form differentiated service chains on demand. The SDN architecture can be integrated with an orchestration platform (e.g., orchestrator 23), such as Kubernetes, OpenShift, Mesos, OpenStack, VMware vSphere, and can be integrated with a service provider operations support system / business support system (OSS / BSS).

[0035] Typically, one or more data centers 10 provide an operating environment for applications and services at customer sites 11 (shown as "customer 11"), which have one or more customer networks coupled to the data centers through service provider networks 7. For example, each data center 10 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. Service provider networks 7 are coupled to public networks 15, which may represent one or more networks managed by other providers, and thus, may form part of a large-scale public network infrastructure (e.g., the Internet). For example, public network 15 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating service provider network 7, an enterprise IP network, or some combination thereof.

[0036] Although the customer sites 11 and the public network 15 are primarily shown and described as edge networks of the service provider network 7, in some embodiments, the customer sites 11 and one or more of the public network 15 may be tenant networks within any data center 10. For example, the data center 10 may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), wherein each tenant may implement one or more customer sites 11.

[0037] The service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement multi-protocol label switching (MPLS) forwarding, and in this instance, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0038] In some embodiments, each data center 10 may represent one of a plurality of geographically distributed network data centers that may be connected to each other via a service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1As shown in the example of , data center 10 may include facilities that provide network services to customers. Customers of the service provider may be collective entities such as businesses and governments or individuals. For example, a network data center may host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, science or supercomputing, etc. Although shown as a discrete edge network of the service provider network 7, elements in the data center 10 such as one or more physical network functions (PNFs) or virtualized network functions (VNFs) may be included in the core of the service provider network 7.

[0039] In this example, data center 10 includes storage and / or computing servers (or "nodes") interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, and servers 12A-12X (herein, referred to as "servers 12") are depicted as coupled to top-of-rack TOR switches 16A-16N. Servers 12 are computing devices and may also be referred to herein as "computing nodes," "hosts," or "host devices." Although in Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, however, data center 10 may include multiple additional servers coupled to other TOR switches 16 in data center 10.

[0040] In the illustrated example, the switch fabric 14 includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") coupled to a distribution layer of chassis (or "spine" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, for example, the data center 10 may also include one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0041] In this example, the TOR switch 16 and the chassis switch 18 provide redundant (multi-homed) connections to the IP structure 20 and the service provider network 7 to the server 12. The chassis switch 18 aggregates traffic flows and provides connections between the TOR switches 16. The TOR switch 16 can be a network device that provides Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functions. The TOR switch 16 and the chassis switch 18 can each include one or more processors and memories and can run one or more software processes. The chassis switch 18 is coupled to the IP structure 20, which can perform Layer 3 routing to route network traffic between the data center 10 and the customer site 11 through the service provider network 7. The switching architecture of the data center 10 is only an example. For example, other switching architectures can have more or fewer switching layers. The IP structure 20 may include one or more gateway routers.

[0042] The term "packet flow", "traffic flow", or simply "flow" refers to a collection of packets originating from a specific source device or endpoint and being sent to a specific destination device or endpoint. For example, a single flow of packets can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, the 5-tuple identifies the packet flow corresponding to the received packets. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for that packet.

[0043] Each server 12 may represent a computing server or a storage server. For example, each server 12 may represent a computing device configured to operate according to the techniques described herein, such as a server based on an x86 processor. The server 12 may provide a network function virtualization infrastructure (NFVI) for the NFV architecture.

[0044] Any of the servers 12 may be configured with a virtual execution element, such as a container pool or a virtual machine, by virtualizing the server's resources to provide a degree of isolation between one or more processes (applications) running on the server. "Hypervisor-based," or "hardware-level," or "platform" virtualization refers to the creation of virtual machines that each include a guest operating system for running one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are visualized from the physical hardware of the host server, the executing applications are isolated from the host's hardware and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0045] A virtual network refers to a logical structure implemented on top of a physical network. A virtual network can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center, such as data center 10. Each tenant or application can have one or more virtual networks. Unless explicitly allowed by security policy, each virtual network must be isolated from all other virtual networks.

[0046] Virtual networks can use the data center 10 gateway router ( Figure 1 The virtual network is connected to the physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and Ethernet virtual private network (EVPN) network and extended on the physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and Ethernet virtual private network (EVPN) network. The virtual network can also be used to implement network function virtualization (NFV) and service chaining.

[0047] A variety of mechanisms can be used to implement virtual networks. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. A virtual network can also be implemented using two networks - a physical underlying network consisting of an IP structure 20 and a switching structure 14 and a virtual overlay network. The role of the physical underlying network is to provide an "IP structure" that provides a unicast IP connection from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0048] As further described below with respect to virtual router 21 (illustrated as and also referred to herein as "vRouter 21"), the virtual router running in server 12 creates a virtual overlay network on top of the physical underlying network using its own dynamic "tunnel" mesh. For example, these overlay tunnels may be MPLS over GRE / UDP tunnels, or VXLAN tunnels, or NVGRE tunnels. For virtual machines or other virtual execution elements, the underlying physical routers and switches may not store arbitrary state for each tenant, such as arbitrary media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may contain only the IP prefix or MAC address of the physical server 12. (The gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0049] The virtual router 21 of the server 12 typically contains per-tenant state. For example, the virtual router may contain a separate forwarding table (routing instance) for each virtual network. The forwarding table contains the IP prefixes (in the case of a layer 3 overlay) or MAC addresses (in the case of a layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., a container pool of containers). A single virtual router 21 does not need to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that are locally present on the server 12 (i.e., have at least one virtual execution element present on the server 12.)

[0050] "Container-based" or "operating system" virtualization refers to the virtualization of an operating system running multiple isolated systems on a single machine (virtual or physical). The isolated systems represent containers such as those provided by the open source DOCKER container application or by CoreOS Rkt ("Rocket"). Like a virtual machine, each container is virtualized and can be kept isolated from the host and other containers. However, unlike a virtual machine, each container can omit an independent operating system and instead provide an application suite and application-specific libraries. Typically, containers are run by a host as isolated user-space instances and the containers can share an operating system and common libraries with other containers running on the host. As a result, containers may require less processing power, storage, and network resources than virtual machines ("VMs"). A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on a corresponding virtual network.

[0051] In some examples, containers are managed by their host kernel to allow for limiting and prioritizing resources (CPU, memory, block I / O, network, etc.) in some cases without launching arbitrary virtual machines using namespace isolation capabilities, which allow applications (e.g., a given container) to view the operating environment completely isolated, including process trees, networking, user identifiers, and mounted file systems. In some embodiments, containers can be deployed according to Linux Containers (LXC), an operating system-level virtualization method that uses a single Linux kernel to run multiple isolated Linux systems (containers) on a control host.

[0052] Servers 12 host virtual network endpoints of one or more virtual networks operating on a physical network represented here by IP fabric 20 and switch fabric 14. Although primarily described with respect to data center based switched networks, other physical networks such as service provider network 7 may underlie one or more virtual networks.

[0053] Each server 12 can host one or more virtual execution elements, each of which has at least one virtual network endpoint of one or more virtual networks configured in the physical network. The virtual network endpoint of a virtual network can represent one or more virtual execution elements that share the virtual network interface of the virtual network. For example, the virtual network endpoint can be a virtual machine, one or more container sets (e.g., a container pool), or another virtual execution element such as a layer 3 endpoint of a virtual network. The term "virtual execution element" covers virtual machines, containers, and other virtualized computing resources that provide at least partially independent operating environments for applications. The term "virtual execution element" can also cover container pools of one or more containers. A virtual execution element can represent an application workload. As Figure 1 As shown in , server 12A hosts a virtual network endpoint in the form of a container pool 22 having one or more containers. However, in practice, given the hardware resource limitations of server 12, server 12 can be executed as multiple virtual execution elements. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or other processing on packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0054] The servers 12 each include at least one network interface card (NIC) 13, each of which includes at least one interface for exchanging packets with the TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component of I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) described by the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to provide one or more virtual network interfaces as "virtual functions" for use by corresponding endpoints running on the server 12. In this way, the virtual network endpoints can share the same PCIe physical hardware resources and the virtual functions are examples of virtual hardware components 21. As another example, one or more servers 12 can implement Virtio, a paravirtualization framework available for example in the Linux operating system, Virtio provides emulated NIC functions as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement Open vSwitch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) of hosted virtual machines, where the vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function. In some examples, any of the servers 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For Docker implementations of containers hosted by servers, a Linux bridge or other operating system bridge running on the server that switches packets between containers may be referred to as a "Docker bridge." As used herein, the term “virtual router” may encompass a Contrail or Tungsten Fabric virtual router, Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other device and / or software that resides on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, in which the virtual network endpoints are hosted by one or more servers 12.

[0055] Any NIC 13 may include an internal device switch that switches data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch may be a virtual Ethernet bridge (VEB) that switches between SR-IOV virtual functions and, correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. Alternatively, the internal device switch may be referred to as a NIC switch, or, for an SR-IOV implementation, may be referred to as an SR-IOV NIC switch. The virtual hardware components associated with the NIC 13A may be associated with a layer 2 destination address assigned by the NIC 13A or a software process responsible for configuring the NIC 13A. Physical hardware components (or "physical functions" for SR-IOV implementations) are also associated with a layer 2 destination address.

[0056] One or more servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) of an Internet Protocol-Virtual Private Network (IP-VPN). For example, a packet received by the virtual router 21 of the server 12A from the underlying physical network structure of the data center 10 (i.e., the IP structure 20 and the switching structure 14) may include an outer header to allow the physical network structure to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the server's network interface card 13A, but also a virtual network identifier such as a VxLAN tag or a multi-protocol label switching (MPLS) label that identifies a virtual network and the corresponding routing instance executed by the virtual router 21. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0057] The virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the tunnel encapsulation header of the packet, and forwards the packet to the appropriate destination virtual network endpoint of the packet. For example, for server 12A, for each packet outbound from a virtual network endpoint (e.g., container pool 22) hosted by server 12A, the virtual router 21 appends a tunnel encapsulation header indicating the virtual network to the packet to generate an encapsulated or "tunnel" packet, and the virtual router 21 outputs the encapsulated packet to a physical destination computing device such as another server 12 via the overlay tunnel of the virtual network. As used herein, the virtual router 21 can perform the operation of a tunnel endpoint to encapsulate an internal packet originating from a virtual network endpoint to generate a tunnel packet, and decapsulate the tunnel packet to obtain an internal packet for routing to other virtual network endpoints.

[0058] In some examples, virtual router 21 may be kernel-based and may run as part of the kernel of an operating system of server 12A.

[0059] In some examples, the virtual router 21 can be a virtual router that supports the Data Plane Development Kit (DPDK). In this example, the virtual router 21 uses DPDK as a data plane. In this mode, the virtual router 21 runs as a user space application linked to the DPDK library (not shown). This is the execution version of the virtual router and is typically used by telecommunications companies, where VNFs are typically DPDK-based applications. The execution of the virtual router 21 as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the polling mode driver (PMD) of the DPDK, rather than the interrupt-based driver of the Linux kernel.

[0060] A User-I / O (UIO) kernel module such as vfio or uio_pci_generic can be used to expose the registers of the physical network interface to user space so that the DPDK PMD can access the registers. When NIC 13A is bound to the UIO driver, NIC 13A is moved from Linux kernel space to user space and is therefore no longer managed or visible to the Linux OS. Therefore, the DPDK application (i.e., in this example, virtual router 21A) fully manages NIC 13. This includes packet polling, packet processing, and packet forwarding. The user packet processing steps can be performed by the virtual router 21DPDK data plane, the kernel (wherein Figure 1The kernel (not shown) has limited or no involvement. The nature of this "polling mode" makes virtual router 21DPDK data plane packet processing / forwarding more efficient compared to interrupt mode, particularly when the packet rate is high. There are limited or no interrupts during packet I / O, and there are context switches. Additional details of the DPDK vRouter example are found in "DAY ONE: CONTRAIL DPDK vROUTER", Kiran KN et al., Juniper Networks, 2021, which is incorporated herein by reference in its entirety.

[0061] The computing infrastructure 8 implements an automation platform for automating the deployment, expansion, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration system that provides a container-centric infrastructure to automate the deployment, expansion, and operation of containers, thereby providing a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services running on the virtual execution elements available to the orchestration platform. For example, container orchestration can facilitate container coordination and refers to the deployment, management, expansion, and configuration of containers to host servers by a container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0062] The elements of the automation platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework in which a cluster master node of the cluster manages the deployment and operation of containers to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" used herein encompass different orchestration platform terms that distinguish between the master management element of the cluster and the analog device of the cluster's master container hosting device. For example, the Kubernetes platform uses the terms "cluster master node" and "slave node", while the Docker Swarm platform refers to the cluster manager and cluster nodes.

[0063] The orchestrator 23 and the network controller 24 may run on separate computing devices or on the same computing device. Each of the orchestrator 23 and the network controller 24 may be a distributed application running on one or more computing devices. The orchestrator 23 and the network controller 24 may implement the respective master nodes of one or more clusters, each of which has one or more slave nodes (also referred to as "computing nodes") implemented by the respective servers 12.

[0064] In general, network controller 24 controls network configuration of the fabric of data center 10 to establish one or more virtual networks for packetized communications between virtual network endpoints, for example. Network controller 24 provides a logically and, in some cases, physically centralized controller that facilitates operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator. Additional information regarding exemplary operation of the network controller 24 in conjunction with other devices in the data center 10 or other software defined networks may be found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUALNETWORK PACKET FLOWS,” and in U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, and entitled “TUNNELED PACKET AGGREGATION FOR VIRTUAL NETWORKS,” each of which is incorporated by reference as fully described herein.

[0065] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides computing infrastructure, which may include container-centric computing infrastructure. The orchestrator 23, and in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud can provide a virtualized infrastructure to the container management platform. Figure 3 Describe example components of the Kubernetes orchestration system.

[0066] In one example, container pool 22 is a Kubernetes container pool and is an example of a virtual network endpoint. A container pool is a group of one or more logically related containers ( Figure 1), shared memory for containers, and options on how to run containers. Alternatively, if instantiated for execution, the container pool may be referred to as a "container pool replica". Each container of the container pool 22 is an example of a virtual execution element. The containers of the container pool are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of the container pool may be a collection of Linux namespaces, cgroups, and other isolation aspects.

[0067] In the context of a container pool, individual applications may further apply sub-isolation. Typically, containers within a container pool have a common IP address and port space and can detect each other via the local host. Because they have a shared context, containers within a container pool can also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different container pools have different IP addresses, and in the absence of a configuration to enable this feature, containers cannot communicate via IPC. Instead, containers that are members of different container pools typically communicate with each other via the container pool IP address.

[0068] Server 12A includes a container platform 19 for running containerized applications such as those in container pool 22. Container platform 19 receives requests from orchestrator 23 to obtain containers and causes the containers to be hosted in server 12A. Container platform 19 obtains and runs containers.

[0069] The container network interface (CNI) 17 configures a virtual network interface for a virtual network endpoint. The orchestrator 23 and the container platform 19 use the CNI 17 to manage the network of the container pool (including the container pool 22). For example, the CNI 17 creates a virtual network interface that connects the container pool to the virtual router 21 and enables the containers of the container pool to communicate to other virtual network endpoints on the virtual network via the virtual network interface. For example, the CNI 17 can insert a virtual network interface for a virtual network into the network namespace of the container of the container pool 22 and configure (or request configuration) a virtual network interface for a virtual network in the virtual router 21 so that the virtual router 21 is configured to send packets received from the virtual network via the virtual network interface to the container of the container pool 22 and send packets received via the virtual network interface on the virtual network from the container of the container pool 22. The CNI 17 can allocate a network address (e.g., a virtual IP address for the virtual network) and can set up routes for the virtual network interface.

[0070] In Kubernetes, by default, all container pools can communicate with all other container pools without using network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a container pool virtual network shared by all namespaces, and allocate service and container pool network addresses from the namespaces respectively. In some cases, all container pools generated in the Kubernetes cluster in all namespaces can communicate with each other, and the network addresses of all container pools can be allocated from the container pool subnet specified by the orchestrator 23. When a user creates an isolated namespace for a container pool, the orchestrator 23 and the network controller 24 can create a new container pool virtual network and a new shared service virtual network for the new isolated namespace. The container pool generated in the Kubernetes cluster in the isolated namespace obtains a network address from the new container pool virtual network, and the corresponding service of the container pool obtains a network address from the new service virtual network.

[0071] CNI 17 may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17 may be at least partially compliant with the Container Network Interface (CNI) specification or the rkt network proposal. CNI 17 may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNI. Alternatively, CNI 17 may be referred to as a network plugin or CNI plugin or CNI instance. For example, a separate CNI may be called by the Multus CNI to establish different virtual network interfaces for the container pool 22.

[0072] The CNI 17 may be called by the orchestrator 23. For the purposes of the CNI specification, containers may be considered to be synchronized with the Linux network namespace. The unit corresponding to this depends on the specific container runtime implementation: for example, in implementations of application container specifications such as rkt, each container pool runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and capable of communicating with each other. This may be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers may be conceptually added to or removed from one or more networks. The CNI specification specifies many considerations for conforming plugins ("CNI plugins").

[0073] Container pool 22 includes one or more containers. In some examples, container pool 22 includes a containerized DPDK workload that is designed to use DPDK to speed up packet processing, for example, by using a DPDK library to exchange data with other components to speed up packet processing. In some examples, virtual router 21 can run as a containerized DPDK workload.

[0074] The container pool 22 is configured with a virtual network interface 26 for sending and receiving packets through the virtual router 21. The virtual network interface 26 may be a default interface for the container pool 22. The container pool 22 may implement the virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while the virtual router 21 may implement the virtual network interface 26 as a tap interface, a virtio user interface, or other types of interfaces.

[0075] The container pool 22 and the virtual router 21 use a virtual network interface 26 to exchange data packets. The virtual network interface 26 may be a DPDK interface. The container pool 22 and the virtual router 21 may use vhost to set up the virtual network interface 26. The container pool 22 may operate according to an aggregation model. The container pool 22 may use a virtual device such as a virtio device with a vhost-user adapter for user space container process communication of the virtual network interface 26.

[0076] CNI 17 can be combined with Figure 1 One or more other components shown in FIG. 2 configure a virtual network interface 26 for the container pool 22. Any container in the container pool 22 can utilize (ie, share) the virtual network interface 26 of the container pool 22.

[0077] The virtual network interface 26 may represent a virtual Ethernet ("veth") pair, where each end of the pair is an independent device (e.g., a Linux / Unix device), and one end of the pair is assigned to the container pool 22 and one end of the pair is assigned to the virtual router 21. A veth pair or one end of a veth pair is sometimes referred to as a "port". The virtual network interface may represent a macvlan network with media access control (MAC) addresses assigned to the container pool 22 and the virtual router 21 to communicate between the containers of the container pool 22 and the virtual router 21. Alternatively, for example, the virtual network interface may be referred to as a virtual machine interface (VMI), a container pool interface, a container network interface, a tap interface, a veth interface, or simply a network interface (in a specific context).

[0078] exist Figure 1In the exemplary server 12A of FIG. 1 , container pool 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment that specifies a virtual network and specifies that container pool 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. For example, orchestrator 23 may receive configuration data from a user, an operator / administrator, or other computing system.

[0079] As part of the process of creating container pool 22, orchestrator 23 requests network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). Container pool 22 may have a different virtual network interface for each virtual network to which it belongs. For example, virtual network interface 26 may be a virtual network interface for a specific virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks.

[0080] The network controller 24 processes the request to generate interface configuration data for the virtual network interface of the container pool 22. The interface configuration data may include a container or container pool unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0081] Network controller 24 sends the interface configuration data to server 12A, and more specifically, in some cases, to virtual router 21. To configure the virtual network interface for container pool 22, orchestrator 23 may call CNI 17. CNI 17 obtains the interface configuration data from virtual router 21 and processes it. CNI 17 creates each virtual network interface specified in the interface configuration data. For example, CNI 17 may attach one end of a veth pair that implements management interface 26 to virtual router 21 and may attach the other end of the same veth pair that implements management interface 26 using virtio user to container pool 22.

[0082] The following is exemplary interface configuration data for a virtual network interface 26 of a container pool 22 .

[0083]

[0084]

[0085] A conventional CNI plugin is invoked by a container platform / runtime, receives an Add command from the container platform to add a container to a single virtual network, and the plugin may subsequently be invoked to receive a Del(ete) command from the container / runtime and remove the container from the virtual network. The term "invocation" may refer to the instantiation of a software component or module in memory as executable code to be executed by a processing circuit.

[0086] According to the techniques described in the present disclosure, the network controller 24 is a cloud-native distributed network controller for software-defined networking (SDN) implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself may be implemented using one or more cloud-native component microservices. Each control node 32 itself may be implemented using one or more cloud-native component microservices.

[0087] In some examples, configuration node 30 can be implemented by extending a local orchestration platform to support custom resources of an orchestration platform for software-defined networks, and more specifically, for example, by configuring virtual network interfaces of virtual execution elements, configuring underlying network connection servers 12, and configuring overlay routing functions including overlay tunnels for virtual networks and overlay trees for multicast Layer 2 and Layer 3, thereby providing a northbound interface to the orchestration platform to support purpose-driven / declarative creation and management of virtual networks.

[0088] As Figure 1 As part of the SDN architecture shown in , the network controller 24 can be multi-tenant aware and support multi-tenancy of the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined network constructs and advanced network features such as virtual networks, BGPaaS, network policies, service chaining, and other telecommunications features. The network controller 24 can support network isolation using virtual network constructs and can support layer 3 networks.

[0089] In order to interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual router 21) import and export policies defined using virtual network router (VNR) resources. Virtual network router resources can be used to define connections between virtual networks by configuring the import and export of routing information between corresponding routing instances for implementing virtual networks in the SDN architecture. A single network controller 24 can support multiple Kubernetes clusters, and thus, VNR allows connecting multiple virtual networks in one namespace, connecting virtual networks in different namespaces, connecting virtual networks in Kubernetes clusters, and connecting virtual networks across Kubernetes clusters. VNR can also be extended to support virtual network connections across multiple instances of the network controller 24. Alternatively, VNR can be referred to as a virtual network policy (VNP) or a virtual network topology here.

[0090] like Figure 1 As shown in the embodiment of FIG. 1 , the network controller 24 may maintain configuration data (e.g., configuration data 30) representing virtual networks (VNs) 50A-50N (“VNs 50”) that represent policies and other configuration data for establishing VNs 50 within the data center 10 through physical underlying networks and / or virtual routers such as virtual router 21 (“vRouter 21”). The network controller 24 may also maintain configuration data (e.g., configuration data 30) representing virtual network routers (VNRs) 52A-52N (“VNRs 52”) that may be implemented at least in part using policies and other configuration data for establishing interconnections between VNs 50.

[0091] A user, such as an administrator, can interact with the UI 60 of the network controller 24 to define the VN 50 and the VNR 52. In some instances, the UI 60 represents a graphical user interface (GUI) that facilitates entering configuration data that defines the VN 50 and the VNR 52. In other instances, the UI 60 can represent a command line interface (CLI) or other types of interfaces. Assuming that the UI 60 represents a graphical user interface, the administrator can define the VN 50 by arranging graphical elements representing different container pools (such as, container pool 22) to associate a container pool with the VN 50, wherein any VN 50 can enable communication between one or more container pools assigned to the VN.

[0092] In this regard, an administrator may understand Kubernetes or other orchestration platforms, but not fully understand the underlying infrastructure supporting VN 50. Some controller architectures, such as Contrail, can configure VN 50 based on networking protocols that are similar (if not substantially similar) to routing protocols in traditional physical networks. For example, Contrail can leverage concepts from the Border Gateway Protocol (BGP), a routing protocol used to communicate routing information within so-called autonomous systems (ASes) and sometimes between ASes.

[0093] There are different versions of BGP, such as internal BGP (iBGP) for communicating routing information within ASes, and external BGP (eBGP) for communicating routing information between ASes. ASes may be related to the concept of projects within Contrail, i.e., similar to namespaces in Kubernetes. In various instances of AS, projects, and namespaces, AS, similar projects, and namespaces may represent a collective term for one or more networks (e.g., one or more VNs 50) that share routing information and thereby facilitate interconnection between networks (or in this instance, VNs 50).

[0094] In its simplest form, VNR 52 represents a logical abstraction of a router set in the context of Kubernetes, where VNR 52 can be defined as a custom resource that facilitates interconnection between VNs 50. Assuming that Kubernetes administrators cannot fully understand the complex propagation of routing information based on complex routing protocols such as BGP, various aspects of cloud-native networking technology may facilitate the abstraction of the underlying routing protocol (or complementary processes of Contrail or other controller architectures) into VNR 52.

[0095] That is, rather than resorting to defining how to route between two or more VNs 50, an administrator can define one or more VNRs 52 that interconnect the VNs 50 without having to manually develop and deploy a large number of policies and / or routing instance configurations to enable the exchange of routing information between the VNs 50. Instead, an administrator (who may have a slight understanding of routing protocols) can define custom resources (e.g., one or more VNRs 52) using familiar Kubernetes syntax / semantics (or even simply by dragging a graphical element and specifying the interconnections between that graphical element, e.g., representing VNR 52A, and graphical elements, again, e.g., representing VNs 50A and 50N).

[0096] In this regard, administrators may be more likely to use Figure 1The logical abstraction shown in the embodiment in the diagram interconnects the VNs 50 as VNR 50, so that the network controller 24 can convert the VNR 50 into an underlying routing target to automatically (i.e., with little or possibly no human intervention) cause the exchange of routing information of VN 50A and 50N and enable communication (i.e., exchange of packets or other data) between VN 50A and 50N.

[0097] Assuming that administrators can configure VNR 50 using familiar Kubernetes syntax / semantics, rather than configuring complex configuration data that conforms to routing protocol syntax / semantics, the network controller 24 can facilitate a better user experience while also facilitating more efficient operation of the data center 8 itself. That is, an administrator typing in configuration data that the administrator is not familiar with may result in misconfiguration, thereby wasting the underlying resources of the data center 8 (in terms of processing cycles, memory, bus bandwidth, etc., and associated power), while also delaying the proper implementation of the network topology (potentially preventing packets and other data from being successfully routed between VN 50). This delay may frustrate not only the administrator, but also the customers associated with the VN 50 who require prompt operation of the VN 50 to achieve business goals. By making it easy for administrators to facilitate communication between VN 50 using the logical abstraction shown as VNR 52, the data center 8 itself may experience more efficient operation (in terms of the above-mentioned computing resources, including processor cycles, memory, bus bandwidth, and associated power) while providing a better user experience for both administrators and customers.

[0098] In operation, the network controller 24 (representing the SDN architecture system of the data center 10) includes processing circuits that implement configuration nodes and control nodes (refer to Figure 3 ). The network controller 24 may be configured to interconnect a first virtual network (e.g., VN 50A) and a second virtual network (e.g., VN 50N) operating within the SDN architecture system represented by the data center 10. The network controller 24 may be configured to define a logical abstraction of one or more policies to perform the interconnection via one or more VNRs 52 (e.g., VNR 52A).

[0099] Policies may include import and export policies related to routing information maintained by a virtual network (in this embodiment, may refer to VN 50A and 50N). That is, Kubernetes may be extended via a custom resource representing VNR 52A to convert VNR 52A into one or more import and export policies deployed relative to VN 50A and VN 50N to configure mutual communication via the distribution of routing information between VN 50A and VN 50N. Once configured, VN 50A may then export routing information (e.g., routing representing VN 50A) to VN 50N and import routing information (e.g., routing representing VN 50N) to VN 50A. Similarly, VN 50N may export routing information (e.g., routing representing VN 50N) to VN 50A and import routing information (e.g., routing representing VN 50A) to VN 50N.

[0100] Abstraction can hide the underlying routing configuration to enable this routing leakage, such as defining routing targets for importing and exporting routing information to routing instances used to implement VN 50A and VN 50N. Alternatively, the network controller 24 can convert VNR 52A to a common routing target and configure the communication of routing information via the common routing target for routing instances used to implement VN 50A and VN 50N (in this example).

[0101] To achieve mesh connectivity, the network controller 24 may configure the import and export of routing instances of VN 50A, VN 50N, and VNR 52A with routing targets associated with VN 50A, VN 50N, and VNR 52A. To achieve hub-and-spoke connectivity, the network controller 24 may configure the export of routing instances associated with VN 50A and VN 50N to export routing information to routing instances associated with VNR 52A (serving as a hub) and routing instances of VNR 52A to import routing information to routing instances associated with VN 50A and VN 50N. In this hub-and-spoke connectivity, VN 50A and VN 50N may not communicate directly with each other.

[0102] In addition, the network controller 24 can support multi-layer security using network policies. The default behavior of Kubernetes is that container pools communicate with each other. In order to apply network security policies, the SDN architecture implemented by the network controller 24 and the virtual router 21 can be operated as the CNI of Kubernetes through CNI 17. For Layer 3, network-level isolation occurs and virtual networks operate at L3. Virtual networks are connected through policies. Kubernetes local network policies provide security at Layer 4. The SDN architecture can support Kubernetes network policies. Kubernetes network policies operate at the boundaries of Kubernetes namespaces. The SDN architecture can add custom resources for enhancing network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta tags to apply granular security policies in an extensible manner). In some examples, for Layer 4+, the SDN architecture can support integration with containerized security devices and / or Istio and can provide encryption support.

[0103] As Figure 1 As part of the SDN architecture shown in FIG. 1 , the network controller 24 can support multi-cluster deployments that are important for telco cloud and high-end enterprise use cases. For example, the SDN architecture can support multiple Kubernetes clusters. The cluster API can be used to support life cycle management of Kubernetes clusters. KubefedV2 can be used to configure node 32 federation across Kubernetes clusters. The cluster API and KubefedV2 are optional components for supporting a single instance of the network controller 24, which supports multiple Kubernetes clusters.

[0104] SDN architectures can provide insights into infrastructure, clusters, and applications using network user interfaces and telemetry components. Telemetry nodes can be cloud-native and include microservices that support insights.

[0105] Due to the above and other features described elsewhere herein, the computing infrastructure 8 implements a cloud-native SDN architecture and can provide one or more of the following technical advantages. For example, the network controller 24 is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrading of the various component microservices of the configuration node 30 and the control node 32 (as well as any other components of other examples of the network controller described in this disclosure). The present technology can further implement optional cloud-native monitoring (telemetry) and user interfaces, high-performance data planes for containers using DPDK-based virtual routers connected to a DPDK-enabled container pool, and cloud-native configuration management that in some cases takes advantage of the configuration framework of existing orchestration platforms (such as Kubernetes or Openstack). As a cloud-native architecture, the network controller 24 is a scalable and elastic architecture that addresses and supports multiple clusters. In some cases, the network controller 24 can also support the scalability and performance requirements of key performance indicators (KPIs).

[0106] For example, an SDN architecture with features and technical advantages such as those described herein can be used to implement a cloud-native telecom cloud to support 5G mobile networks (and subsequent generations) and edge computing, as well as an enterprise Kubernetes platform including, for example, high-performance cloud-native application hosting. Telecom cloud applications are rapidly moving towards containerized cloud-native solutions. 5G fixed and mobile networks are driving the need to deploy workloads as microservices with significant disaggregation, specifically, in 5G next-generation RAN (5GNR). The 5G next-generation core network (5GNC) may be deployed as a set of microservice-based applications corresponding to the various components described by 3GPP. When viewed as a microservice group for delivering applications, the 5GNC may be a highly complex combination of a pool of containers with complex network, security, and policy requirements. For such use cases, a cloud-native SDN architecture with well-defined structures for network, security, and policy as described herein can be utilized. The network controller 24 may provide relevant APIs that can create these complex structures.

[0107] Similarly, the user plane function (UPF) within the 5GNC is an ultra-high performance application. It can be delivered as a highly distributed set of high-performance container pools. The SDN architecture described here is capable of providing a very high throughput data plane (in terms of bits per segment (bps) and packets per second (pps)). Integration with a DPDK virtual router with recent performance enhancements eBPF and with smart NICs will help achieve the required throughput. The DPDK-based virtual router is further described in detail in U.S. application No. 17 / 649,632, entitled “CONTAINERIZED ROUTER WITH VIRTUAL NETWORKING,” filed on February 1, 2022, the entire contents of which are incorporated herein by reference.

[0108] High performance processing may also be relevant in GiLAN as workloads migrate from more traditional virtualized workloads to containerized microservices. In the data plane of the UPF and GiLAN services, such as the GiLAN firewall, intrusion detection and prevention, virtualized IP multimedia subsystem (vIMS) voice / video, etc., throughput is high and sustained in terms of bps and pps. For the control plane of 5GNC functions, such as the access and mobility management function (AMF), session management function (SMF), etc., and for some GiLAN services (e.g., IMS), the advantage of small packets is that pps will remain high while the absolute traffic can be moderate in terms of bps. In some examples, the SDN controller and data plane provide millions of packets per second for each virtual router 21 implemented on the server 12. In 5G radio access networks (RAN), in order to move away from proprietary vertically integrated RAN stacks provided by traditional radio vendors, open RAN decouples RAN hardware and software in multiple components, including non-RT radio intelligent controllers (RIC), near real-time RIC, centralized unit (CU) control plane and user plane (CU-CP and CU-UP), distributed units (DU), and radio units (RU). If necessary, the software components are deployed on commercial server architectures implemented with programmable accelerators. The SDN architecture described here can support O-RAN specifications.

[0109] Edge computing may be targeted at two different use cases. The first case is as a support for containerized telecom infrastructure (e.g., 5G RAN, UPF, security functions), and the second case is for containerized service workloads for telecom and third parties such as suppliers or enterprise customers. In both cases, edge computing is actually a special case of GiLAN, where traffic is interrupted for special processing at highly distributed locations. In many cases, these locations will have limited resources (power, cooling, space).

[0110] The SDN architecture described herein may be well suited to support very lightweight footprint requirements, may support compute and storage resources in sites that are remote from the associated control functions, and may be location-aware by the way workloads and storage are deployed. Some sites may have as few as one or two compute nodes that deliver very specific sets of services to highly localized sets of users or other services. There may be a hierarchy of sites where central sites are densely connected with multiple paths, regional sites are multiply connected with two or four uplink paths, and remote edge sites may have connections to only one or two upstream sites.

[0111] This requires extreme flexibility in how the SDN architecture can be deployed and how (and where) tunneled traffic within the overlay is terminated and bound to the core transport network (SRv6, MPLS, etc.). Similarly, in sites hosting telco cloud infrastructure workloads, the SDN architecture described here can support specialized hardware (GPUs, smart NICs, etc.) required for high-performance workloads. There may also be workloads that require SR-IOV. Because, the SDN architecture can also support the creation of VTEPs at the ToR and have the VTEPs link back to the overlay as VXLAN.

[0112] It is expected that there will be a mix of fully distributed Kubernetes micro-clusters with each site running its own master cluster, and the SDN architecture can support scenarios like remote computing.

[0113] For use cases involving enterprise Kubernetes platforms, high-performance cloud-native application power financial services platforms, online gaming services, and hosted application service providers, the cloud platforms that deliver these applications must provide high performance, resilience to failures, high security, and visibility. The applications hosted on these platforms tend to be developed internally. Application developers and platform owners work with infrastructure teams to deploy and operate instances of organizational applications. These applications tend to require high throughput (>20Gbps per server) and low latency. Some applications can also use multicast for signaling or payload traffic. Additional hardware and network infrastructure can be used to ensure availability. Applications and microservices are partitioned using command spaces within the cluster. In a highly secure environment, isolation between namespaces is critical. When the default deny policy is a standard posture in a zero-trust application deployment environment, additional network segmentation using virtual routing and forwarding instances (VRF) adds an additional security layer and allows the use of overlapping network ranges. Overlapping network ranges are a key requirement for managing application hosting environments, which tend to standardize the reachable endpoint sets of all managed customers.

[0114] Complex microservices-based applications tend to utilize complex network filters. The SDN architecture described herein can deliver high-performance firewall filtering at scale. The filtering can exhibit consistent forwarding performance, as well as lower latency degradation, regardless of rule set length or sequence. Some customers may also have some of the same regulatory pressures as telecommunications for application segmentation not only at the network layer, but also in the kernel. Specifically, when running on public clouds, financial, but also other services have requirements for data plane encryption. In some examples, the SDN architecture described herein can include features that meet these requirements.

[0115] In some examples, when the SDN architecture is automated by applying a dev / test / stage / prod continuous integration / continuous deployment (CI / CD) pipeline, the SDN architecture can provide a GitOps-friendly UX for strict change management control, auditing, and reliability for products that change several times or even hundreds of times a day.

[0116] Figure 2 2 is a block diagram illustrating an example of a cloud-native SDN architecture for a cloud-native network according to the techniques of the present disclosure. The SDN architecture 200 is illustrated by abstracting the underlying connections between the various components. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A-230N ("configuration nodes" or "config nodes" and collectively referred to as "configuration nodes 230") and control nodes 232A-232K (collectively referred to as "control nodes 232"). The configuration nodes 230 and the control nodes 232 may represent Figure 1 2. Although shown as being separate from the server 12, the configuration node 230 and the control node 232 may be executed as one or more workloads on the server 12.

[0117] Configuration node 230 provides a northbound representational state transfer (REST) ​​interface to support intent-driven configuration of SDN architecture 200. Exemplary platforms and applications for pushing intent to configuration node 230 include virtual machine orchestrator 240 (e.g., Openstack), container orchestrator 242 (e.g., Kubernetes), user interface 242, or other one or more applications 246. In some examples, SDN architecture 200 uses Kubernetes as its underlying platform.

[0118] The SDN architecture 200 is divided into a configuration plane, a control plane, a data plane, and an optional telemetry (or analysis) plane. The configuration plane is implemented using a horizontally scalable configuration node 230, the control plane is implemented using a horizontally scalable control node 232, and the data plane is implemented using a computing node.

[0119] At a high level, the configuration node 230 uses the configuration store 224 to manage the configuration resource state of the SDN architecture 200. In general, a configuration resource (or more simply a "resource") is a named object schema that includes data and / or methods that describe a custom resource, and defines an application programming interface (API) for creating and manipulating data through an API server. A kind is the name of an object schema. Configuration resources can include Kubernetes native resources such as a container pool, an ingress, a configuration map, a service, a role, a namespace, a node, a network policy, or a load balancer.

[0120] According to the technology of the present disclosure, configuration resources also include custom resources, which are used to extend the Kubernetes platform by defining application programming interfaces (APIs) that are not available in the default installation of the Kubernetes platform. In the example of the SDN architecture 200, the custom resources can describe the physical infrastructure, virtual infrastructure (e.g., VN 50 and / or VNR52), configuration, and / or other resources of the SDN architecture 200. As part of configuring and operating the SDN architecture 200, various custom resources can be instantiated (e.g., VNR 52 within vRouter 21). The instantiated resources (whether local or custom) can be referred to as objects or instances of resources, that is, permanent entities in the SDN architecture 200 that represent the intent (required state) and status (actual state) of the SDN architecture 200.

[0121] Configuration node 230 provides an aggregate API for performing operations (i.e., create, read, update, and delete) on configuration resources of SDN architecture 200 in configuration storage 224. Load balancing 226 represents one or more load balancing objects that load balance configuration requests between configuration nodes 230. Configuration storage 224 can represent one or more etcd databases. Configuration node 230 can be implemented using Nginx.

[0122] The SDN architecture 200 can provide networking for Openstack and Kubernetes. Openstack uses a plug-in architecture to support networking. Using the virtual machine orchestrator 240, i.e., Openstack, the Openstack networking plug-in driver converts the Openstack configuration objects into configuration objects (resources) of the SDN architecture 200. The computing nodes run Openstack nova to build virtual machines.

[0123] Utilizing the container orchestrator 242, i.e., Kubernetes, the SDN architecture 200 is used as a Kubernetes CNI. As described above, Kubernetes native resources (container pools, services, ingress, external load balancing, etc.) can be supported, and the SDN architecture 200 can support custom resources for Kubernetes for advanced networking and security of the SDN architecture 200.

[0124] Configuration nodes 230 provide REST monitoring to control nodes 232 to monitor changes in configuration resources, and control nodes 232 affect configuration resource changes within the computing infrastructure. Control nodes 232 receive configuration resource data from configuration nodes 230 by monitoring resources and build a complete configuration graph. A given control node in control nodes 232 consumes configuration resource data associated with the control node and communicates it to virtual router 21 (i.e., virtual router proxy- Figure 1 The control interface 254 of the control plane (not shown in the figure) distributes the required configuration to the computing nodes (servers 12). Any control node 232 may receive only a partial graph as required for processing. The control interface 254 may be XMPP. The number of deployed configuration nodes 230 and control nodes 232 may be a function of the number of supported clusters. To support high availability, the configuration plane may include 2N+1 configuration nodes 230 and 2N control nodes 232.

[0125] The control node 232 distributes routes between the computing nodes. The control node 232 uses iBGP to exchange routes between the control nodes 232, and the control node 232 can peer with any external BGP-supported gateway or other router. The control node 232 can use a route reflector.

[0126] Container pool 250 and virtual machines 252 are examples of workloads that can be deployed to compute nodes by virtual machine orchestrator 240 or container orchestrator 242 and interconnected by SDN architecture 200 using one or more virtual networks.

[0127] Figure 3 2 is a block diagram illustrating another view of components of an SDN architecture 200 according to the techniques of the present disclosure and further illustrating in detail. Configuration node 230, control node 232, and user interface 244 are shown with their corresponding component microservices to implement network controller 24 and SDN architecture 200 as a cloud-native SDN architecture. Each component microservice can be deployed to a compute node.

[0128] Figure 3A single cluster is shown partitioned into network controller 24, user interface 244, compute nodes (servers 12), and telemetry 260 features. Configuration node 230 together with control node 232 make up network controller 24.

[0129] Configuration node 230 may include component microservice API server 300 (or "Kubernetes API server 300" - corresponding controller 406, Figure 3 ), custom API server 301, custom resource controller 302, and SDN controller manager 303 (sometimes referred to as "kube-manager" or "SDN kube-manager", where the orchestration platform for network controller 24 is Kubernetes). Contrail-kube-manager is an example of SDN controller manager 303. Configuration node 230 extends the interface of API server 300 with custom API server 301 to form an aggregation layer to support the data model of SDN architecture 200. As described above, the configuration intent of SDN architecture 200 can be a custom resource.

[0130] The control node 232 may include components microservice control 320 and core DNS 322. Figure 2 As described, control 320 performs configuration distribution and route learning and distribution.

[0131] The computing node is represented by server 12. Each computing node includes a virtual router agent 316 and a virtual router forwarding component (vRouter) 318. Either of the virtual router agent 316 and the vRouter 318 can be a component microservice or both can be component microservices. Typically, the virtual router agent 316 performs functions related to control. The virtual router agent 316 receives configuration data from the control node 232 and converts the configuration data into forwarding information for the vRouter 318. The virtual router agent 316 can also perform firewall rule processing, set the flow of the vRouter 318, and dock with the orchestration plug-in (CNI for Kubernetes and Nova plug-in for Openstack). When a workload (container pool or VM) is called out on the computing node, the virtual router agent 316 generates a route, and the virtual router agent 316 exchanges the route with the control node 232 to distribute to other computing nodes (the control node 232 uses BGP to distribute routes between the control nodes 232). When the workload is terminated, the virtual router agent 316 also withdraws the route. vRouter 318 may support one or more forwarding modes, such as kernel mode, DPDK, smart NIC offload, etc. In some examples of container architectures or virtual machine workloads, the compute nodes may be Kubernetes worker / slave nodes or Openstack nova-compute nodes, depending on the specific orchestrator used.

[0132] One or more optional telemetry nodes 260 provide metrics, alerts, logging, and streaming analytics. Telemetry for the SDN architecture 200 leverages cloud native monitoring services such as Prometheus, Elastic, Fluentd, Kinaba stack (EFK), and Influx TSDB. The SDN architecture component microservices of the configuration node 230, control node 232, compute node, user interface 244, and analysis node (not shown) can generate telemetry data. The services of the telemetry node 260 can use the telemetry data. The telemetry node 260 can expose REST endpoints for users and can support insights and event correlation.

[0133] Optionally, the user interface 244 includes a network user interface (UI) 306 and a UI backend 308 service. In general, the user interface 244 provides configuration, monitoring, visualization, security, and troubleshooting for SDN architecture components.

[0134] Each of telemetry 260, user interface 244, configuration node 230, control node 232, and server 12 / compute node can be considered a node of SDN architecture 200, in that each of these nodes is an entity that implements the functions of configuration plane, control plane, or data plane, or UI and telemetry node. The node scale is configured during the "build" process, and SDN architecture 200 supports automatic expansion of nodes of SDN architecture 200 using an orchestration system operator such as a Kubernetes operator.

[0135] As described above, the SDN architecture 200 configuration intent may be a custom resource. The custom resource may include VNR 52 ( Figure 1 As shown in the embodiment of the present invention, communication is established between two or more VN 50s through VNR 52 in the manner described above. As described above, VNR 52 can represent a logical abstraction of a policy for configuring the import and export of routing information between VN 50, so that VNR 52 can facilitate the exchange of routing information (referring to asymmetric or symmetric import and export) using a common routing target established for each VNR 52. A common routing target can be defined and associated with a routing instance used to implement VN 50.

[0136] An administrator, such as a Kubernetes administrator, can interface with the user interface 244 (e.g., the network UI 306) to define the VNR 52 via a graphical user interface that may have graphical elements representing container pools, VNs 50, etc. To define the VNR 52, the administrator can associate the VNR 52 with one or more labels assigned to the VN 50. Using these labels, the VNR 52 can establish an import policy to and an export policy from a common routing target created when the VNR 52 is instantiated. The network UI 306 can interface with the configuration controller 230 to create a common routing target, thereby installing the common routing target into one or more virtual routers 318 via the control node 232. The network UI 306 can also define routing instances of the common routing target (to distinguish the common routing target from other common routing targets) via the configuration controller 230 and the control node 232, thereby utilizing the routing domain to facilitate the interconnection between the VNs 50 described in more detail below for a number of different networking scenarios.

[0137] Figure 4is a block diagram illustrating exemplary components of an SDN architecture according to the techniques of the present disclosure. In this example, the SDN architecture 400 extends and uses the Kubernetes API server for network configuration objects to implement user intent for the network configuration. In Kubernetes terminology, the configuration objects are referred to as custom resources, and when persisted in the SDN architecture, are simply referred to as objects. Configuration objects are primarily user intent (e.g., virtual networks such as VN 50, VNR 52, BGPaaS, network policies, service chains, etc.).

[0138] The configuration node 230 of the SDN architecture 400 can use the Kubernetes API server for configuration objects. In kubernetes terminology, it is called a custom resource.

[0139] Kubernetes provides two ways to add custom resources to the cluster:

[0140] Custom Resource Definitions (CRDs) are simple and can be created without any programming.

[0141] API aggregation requires programming, but allows more control over API behavior, such as control over how data is stored and control over transitions between API versions.

[0142] The aggregate API is a slave API server that sits behind the main API server and acts as a proxy. This arrangement is called API Aggregation (AA). To the user, it appears to be just an extension of the Kubernetes API. CRD allows users to create new types of resources without adding another API server. Regardless of how it is installed, the new resources are called Custom Resources (CR) to distinguish them from local Kubernetes resources (e.g., container pools). CRDs are used in the initial Config prototype. The architecture can use the API Server Builder Alpha library to implement the aggregate API. API Server Builder is a general term for libraries and tools for building local Kubernetes aggregate extensions.

[0143] Typically, each resource in the Kubernetes API requires code to handle REST requests and manage persistent storage of objects. The main Kubernetes API server 300 (implemented by API server microservices 300A-300J) handles local resources and is also able to perform general processing of custom resources through CRDs. Aggregate API 402 represents an aggregation layer that extends the Kubernetes API server 300 to allow dedicated implementations of custom resources to be provided by writing and deploying custom API servers 301 (using custom API server microservices 301A-301M). The main API server 300 delegates requests for custom resources to the custom API server 301, thereby making the resource available to all its clients.

[0144] In this way, the API server 300 (e.g., kube-apiserver) receives Kubernetes configuration objects, local objects (container pools, services), and custom resources. The custom resources of the SDN architecture 400 may include configuration objects, which, when the expected state of the configuration objects in the SDN architecture 400 is achieved, implement the expected network configuration of the SDN architecture 400, including implementing each VNR 52 as one or more import policies and / or one or more export policies and a common routing target (and routing instance). As described above, implementing a VNR 52 within the SDN architecture 400 may generate import and / or export policies that interconnect more than two VNs 50 described in more detail below.

[0145] In this regard, custom resources may correspond to configuration schemes typically defined for network configuration, but extended to be manipulable through the aggregate API 402 according to the techniques of the present disclosure. The custom resources may be defined alternatively and referred to herein as "custom resources for SDN architecture configuration". These may include VNs, VNRs, bgp-as-a-service (BGPaaS), subnets, virtual routers, service instances, projects, physical interfaces, logical interfaces, nodes, physical ipams, floating ips, alarms, alias ips, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances. Custom resources for SDN architecture configuration may correspond to configuration objects typically protected by SDN controllers, but, according to the techniques described herein, configuration objects are exposed as custom resources and merged with Kubernetes native / built-in resources to support exposure by the aggregate API 402, through Kubernetes controllers 406A–406N, and through custom resource controllers 302 ( Figure 4, with component microservices 302A- 302L implementing the same intent model shown in , the custom resource controller 302 is dedicated to reconciling the actual state of the computing infrastructure including network elements into the desired state.

[0146] Given the unified nature in terms of exposed custom resources merged with Kubernetes native / built-in resources, a Kubernetes administrator (or other Kubernetes user) can define VNRs such as VNR 52 using common Kubernetes semantics that are then converted into complex policies detailing the import and export of routing information to facilitate interconnection of VNs 50 without requiring much, if any, understanding of BGP and other routing protocols that are typically used to interconnect VNs 50. Thus, various aspects of the technology can facilitate a more unified user experience with potentially less misconfiguration and trial and error, which can improve the performance of the SDN architecture 400 itself (in terms of utilizing less processing cycles, memory, bandwidth, etc., and associated power).

[0147] The aggregation layer of API server 300 sends the API custom resources to its corresponding registered custom API server 301. There may be multiple custom API servers / custom resource controllers supporting different kinds of custom resources. Custom API server 301 processes custom resources for SDN architecture configuration and writes custom resources to configuration storage 304 (which may be etcd). Custom API server 301 can be a host and exposes the SDN controller identifier allocation service required by custom resource controller 302.

[0148] The custom resource controller 302 starts applying business logic to satisfy the user intent configured with the user intent. The business logic is implemented as a coordination loop. Figure 8 814 is a block diagram illustrating an example of a custom controller for custom resources configured for an SDN architecture according to the techniques of the present disclosure. Custom controller 814 may represent an exemplary instance of custom resource controller 302. Figure 8In the example shown, the custom controller 814 can be associated with a custom resource 818. The custom resource 818 can be any custom resource for SDN architecture configuration. The custom controller 814 can include a coordinator 816, which includes logic to perform a coordination loop, wherein the custom controller 814 monitors 834 (e.g., monitors) a current state 832 of the custom resource 818. In response to determining that the desired state 826 does not match the current state 832, the coordinator 816 can perform an action to adjust 838 the state of the custom resource so that the current state 832 matches the desired state 836. The request can be received by the API server 300 and relayed to the custom API server 301 to change the current state 832 of the custom resource 818 to the desired state 836.

[0149] In the case where the API request is a creation request for a custom resource, the coordinator 816 can act on a creation event for instance data of the custom resource. The coordinator 816 can create instance data of the custom resource on which the requested custom resource depends. As an example, an edge node custom resource can depend on a virtual network custom resource, a virtual interface custom resource, and an IP address custom resource. In this example, when the coordinator 816 receives a creation event for an edge node custom resource, the coordinator 816 can also create custom resources on which the edge node custom resource depends, such as a virtual network custom resource, a virtual interface custom resource, and an IP address custom resource.

[0150] By default, the custom resource controller 302 runs in active-passive mode and uses master election for consistency. When the controller container pool starts, it attempts to create a ConfigMap resource in Kubernetes using the specified key. If the creation is successful, the container pool becomes the master and starts handling coordination requests; otherwise, the container pool blocks the attempt to create an infinite loop of ConfigMaps.

[0151] The custom resource controller 302 can track the status of the custom resources it creates. For example, a virtual network (VN) creates a routing instance (RI), which creates a routing target (RT). If the creation of a routing target fails, the routing instance status degrades, and thus, the virtual network status also degrades. Therefore, the custom resource controller 302 can output custom messages indicating the status of these custom resources to troubleshoot. Similarly, a VNR creates an RI, which creates an RT in a similar manner as discussed above with reference to a VN, and also with reference to a VNR. 17A to 17D The embodiments in describe VN in more detail. Fig. 9An exemplary flow for creating, monitoring, and coordinating between custom resource types that depend on different custom resource types is shown in FIG.

[0152] The configuration plane implemented by the configuration node 230 is highly available. The configuration node 230 can be based on Kubernetes, including a kube-api server service (e.g., API server 300) and a storage backend etcd (e.g., configuration storage 304). In fact, the aggregated API 402 implemented by the configuration node 230 operates as a front end to the control plane implemented by the control node 232. The main implementation of the API server 300 is a kube-api server designed to scale horizontally by deploying more instances. As shown, several instances of the API server 300 can be run to load balance API requests and processing.

[0153] Configuration storage 304 may be implemented as etcd. etcd is a consistent and highly available key-value store used as a Kubernetes backing store for cluster data.

[0154] exist Figure 4 In the example of , the servers 12 in the SDN architecture 400 each include an orchestration agent 420 and a containerized (or "cloud native") routing protocol daemon 324. These components of the SDN architecture 400 are described in further detail below.

[0155] The SDN controller manager 303 can operate as an interface between Kubernetes core resources (services, namespaces, container pools, network policies, network attachment definitions) and the extended SDN architecture resources (virtual networks, routing instances, etc.). The SDN controller manager 303 monitors the Kubernetes API for changes on the custom resources configured by the Kubernetes core network and SDN architecture and is therefore able to perform CRUD operations on the relevant resources.

[0156] In some examples, the SDN controller manager 303 is a collective term for one or more Kubernetes custom controllers. In some examples, in a single or multiple cluster deployments, the SDN controller manager 303 can run on the Kubernetes cluster it manages.

[0157] The SDN controller manager 303 listens to the following Kubernetes objects for create, delete, and update events:

[0158] ·Container pool

[0159] ·Serve

[0160] Node port

[0161] ·Entrance

[0162] Endpoints

[0163] Namespace

[0164] ·deploy

[0165] Network strategy.

[0166] When these events are generated, the SDN controller manager 303 creates appropriate SDN architecture objects, which in turn are defined as custom resources of the SDN architecture configuration. In response to detecting an event on an instance of a custom resource, whether instantiated by the SDN controller manager 303 and / or through the custom API server 301, the control node 232 obtains configuration data about the instance of the custom resource and configures a corresponding instance of the configuration object in the SDN architecture 400.

[0167] For example, the SDN controller manager 303 monitors the container pool creation event, and accordingly, the following SDN architecture objects may be created: virtual machine (workload / container pool), virtual machine interface (virtual network interface), and instance IP (IP address). Then, in this case, the control node 232 may instantiate the SDN architecture objects in the selected computing node.

[0168] As an example, based on monitoring (watch), the control node 232A can detect an event on an instance of a first custom resource exposed by the custom API server 301A, wherein the first custom resource is used to configure a certain aspect of the SDN architecture system 400 and corresponds to a configuration object type of the SDN architecture system 400. For example, the type of the configuration object can be a firewall rule corresponding to the first custom resource. In response to the event, the control node 232A can obtain configuration data about the firewall rule instance (e.g., a firewall rule specification) and provide the firewall rule in a virtual router for the server 12A. The configuration node 230 and the control node 232 can use the configuration object of the corresponding type of the SDN architecture to perform similar operations on other custom resources, such as virtual networks, virtual network routers, BGP as a Service (BGPaaS), subnets, virtual routers, service instances, projects, physical interfaces, logical interfaces, nodes, network ipams, floating ips, alarms, alias ips, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances, etc.

[0169] Figure 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure. Figure 5The computing device 500 in the example may represent an actual server or a virtual server and may represent an exemplary example of any server 12 and may be referred to as a computing node, a master / slave node, or a host. In this example, the computing device 500 includes a bus 542 coupled to hardware components in the hardware environment of the computing device 500. The bus 542 is coupled to a network interface card (NIC) 530, a storage disk 546, and one or more microprocessors 510 (hereinafter referred to as "microprocessors 510"). The NIC 530 may support SR-IOV. In some cases, the front side bus may be coupled to the microprocessor 510 and the memory device 544. In some examples, the bus 542 may be coupled to the memory device 544, the microprocessor 510, and the NIC 530. The bus 542 may represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller may control DMA transfers between components coupled to the bus 542. In some examples, components coupled to the bus 542 control DMA transfers between components coupled to the bus 542.

[0170] The microprocessor 510 may include one or more processors, each of which includes an independent execution unit that executes instructions consistent with an instruction set architecture, instructions stored to a storage medium. The execution units may be implemented as independent integrated circuits (ICs) or may be combined in one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip microprocessor).

[0171] Disk 546 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 510.

[0172] The main memory 544 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the required program code and program data in the form of instructions or data structures and that can be accessed by the computer. The main memory 544 provides a physical address space consisting of addressable memory locations.

[0173] The network interface card (NIC) 530 includes one or more interfaces 532 configured to exchange packets using the links of the underlying physical network. The interface 532 may include a port interface card having one or more network ports. The NIC 530 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 530 and other devices coupled to the bus 542 may be read from the NIC memory, or direct memory access transmissions between the NIC 530 and other devices coupled to the bus 542 may be written to the NIC memory.

[0174] Memory 544, NIC 530, storage disk 546, and microprocessor 510 can provide an operating environment for a software stack including an operating system kernel 580 executed in kernel space. For example, kernel 580 can represent Linux, Berkeley Software Distribution (BSD), a kernel of another Unix variant, or a Windows server operating system kernel purchased from Microsoft. In some examples, the operating system can run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include kernel-based virtual machines (KVM) for Linux kernels, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can cover virtual machine managers (VMMs). The operating system including kernel 580 provides an execution environment for one or more processes in user space 545.

[0175] Kernel 580 includes physical drivers 525 using network interface card 530. Network interface card 530 may also implement SR-IOV to enable communication between containers such as container 529A or one or more virtual machines ( Figure 5530). A shared virtual device such as a virtual function may provide dedicated resources so that each virtual execution element can access the dedicated resources of the NIC 530, so that the NIC 530 appears to be a dedicated NIC to each virtual execution element. A virtual function may represent a lightweight PCIe function that shares physical resources with the physical function used by the physical driver 525 and with other virtual functions. For a NIC 530 that supports SR-IOV, the NIC 530 may have thousands of available virtual functions that comply with the SR-IOV standard, however, for I / O intensive applications, the number of configured virtual functions is typically much smaller.

[0176] Computing device 500 may be coupled to a physical network switch fabric that includes an overlay network that extends the switch fabric from physical switches to software or "virtual" routers (including virtual router 506) of physical servers coupled to the switch fabric. A virtual router may be a physical server (e.g., Figure 1 The virtual router dynamically creates and manages one or more virtual networks that can be used for communication between virtual network endpoints. In one example, the virtual routers each implement a virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from a physical address (e.g., IP address) on the server executing the endpoint.

[0177] Each virtual network may use its own addressing and security scheme and may be considered orthogonal to the physical network and its addressing scheme. Various techniques may be used to transmit packets within and across virtual networks on top of a physical network. As used herein, the term "virtual router" may encompass Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other device and / or software that resides on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. Figure 5 In the exemplary computing device 500 of FIG. 5 , the virtual machine router 506 runs as a DPDK-based virtual router in user space, but the virtual router 506 can run in a hypervisor, a host operating system, a host application, or a virtual machine through various implementations.

[0178] The virtual router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module that is typically used by the Kubernetes deployment of the container pool 502. The virtual router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 506 can perform network services such as application security policies, NAT, multicast, mirroring, and load balancing.

[0179] The virtual router 506 can run as a kernel module or as a user space DPDK process (here, the virtual router 506 located in the user space 545 is shown). A virtual router agent 514 can also be run in the user space. In the exemplary computing device 500, the virtual router 506 runs as a DPDK-based virtual router in the user space, but the virtual router 506 can run in a hypervisor, a host operating system, a host application, or a virtual machine in various implementations. The virtual router agent 514 connects to the network controller 24 using a channel, which is used to download configuration and forwarding information. The virtual router agent 514 programs the forwarding state into the virtual router data (or "forwarding") plane represented by the virtual router 506. The virtual router 506 and the virtual router agent 514 can be processes. The virtual router 506 and the virtual router agent 514 are containerized / cloud-native.

[0180] The virtual router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module that is typically used by the Kubernetes deployment of the container pool 502. The virtual router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 506 can perform network services such as application security policies, NAT, multicast, mirroring, and load balancing.

[0181] Virtual router 506 can be multithreaded and run on one or more processor cores. Virtual router 506 can include multiple queues. Virtual router 506 can implement a packet processing pipeline. Depending on the operation applied to the packet, virtual router agent 514 stitches the pipeline in the simplest way to the most complex way. Virtual router 506 can maintain multiple instances of the forwarding base. Virtual router 506 can access and update the table using RCU (read copy update) locks.

[0182] To send packets to other computing nodes or switches, virtual router 506 uses workloads, injected into VM or container pool 502. Virtual router 506 has multiple virtual network interfaces (e.g., vifs). These interfaces may include a kernel interface vhost0 that exchanges packets with the host operating system, an interface pkt0 that exchanges packets with a virtual router agent 514 to obtain forwarding state from a network controller and send exception packets. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 532. Other virtual network interfaces of virtual router 506 are used to exchange packets with workloads.

[0183] In a kernel-based deployment of virtual router 506 (not shown), virtual router 506 is installed as a kernel module in the operating system. Virtual router 506 registers itself through the TCP / IP stack to receive packets from any desired operating system interface it desires. Interfaces can be bound, physicalized, tapped (for VM), veth (for containers), etc. In this mode, virtual router 506 relies on the operating system to send packets to different interfaces and receive packets from different interfaces. For example, the operating system can expose a tapped interface supported by a vhost network driver to communicate with the VM. Once virtual router 506 registers packets from the tapped interface, the TCP / IP stack sends all packets to the virtual router. Virtual router 506 sends packets via the operating system. In addition, NIC queues (physical or virtual) are processed by the operating system. Packet processing can operate in interrupt mode, which generates interrupts and may cause frequent context switching. When there is a higher packet rate, the total overhead associated with frequent interrupts and context switching may overwhelm the operating system and cause poor performance.

[0184] In a DPDK-based deployment of virtual router 506 ( Figure 5), the virtual router 506 is installed as a user space 545 application linked to the DPDK library. Specifically, this may result in faster performance than a kernel-based deployment when there are higher packet rates. The physical interface 532 is used by the DPDK's polling mode driver (PMD) rather than the kernel's interrupt-based driver. The registers of the physical interface 532 can be exposed to the user space 545 so that the PMD can access them; the physical interface 532 bound in this way is no longer managed by the host operating system or is invisible to the host operating system, and the DPDK-based virtual router 506 manages the physical interface 532. This includes packet polling, packet processing, and packet forwarding. In other words, the user packet processing steps are performed by the DPDK data plane of the virtual router 506. Compared with the interrupt mode, the nature of this "polling mode" makes the DPDK data plane packet processing / forwarding of the virtual router 506 more efficient when the packet rate is high. Compared to kernel mode virtual router 506, there are relatively fewer interrupts and context switches during packet I / O, and in some cases, interrupts and context switches during packet I / O can be avoided altogether.

[0185] In general, each of the container pools 502A-502B may be assigned one or more virtual network addresses for use within a corresponding virtual network, wherein each virtual network may be associated with a different virtual subnet provided by the virtual router 506. For example, the container pool 502B may be assigned its own virtual layer 3 (L3) IP address to send and receive communications, but the container pool 502B may not be aware of the IP address of the computing device 500 on which the container pool 502B is running. Thus, the virtual network address may be different from the logical address of the underlying physical computer system (e.g., the computing device 500).

[0186] The computing device 500 includes a virtual router agent 514 that controls the overlay of the virtual network of the computing device 500 and coordinates the routing of data packets within the computing device 500. In general, the virtual router agent 514 communicates with the network controller 24 for the virtualization infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 500, and more specifically, the virtual router 506, and virtual network interfaces. By configuring the virtual router 506 based on information received from the network controller 24, the virtual router agent 514 can support the configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0187] In one example, a network packet, such as a Layer 3 (L3) IP packet or a Layer 2 (L2) Ethernet packet generated and used by containers 529A-529B located within a virtual network domain, can be encapsulated in another packet (e.g., another IP or Ethernet packet) transmitted by a physical network. Packets transmitted in the virtual network may be referred to herein as "inner packets", while physical network packets may be referred to herein as "outer packets" or "tunnel packets". Encapsulation and / or decapsulation of virtual network packets within physical network packets may be performed by a virtual router 506. This function is referred to herein as tunneling and may be used to create one or more overlay networks. In addition to IPinIP, other exemplary tunneling protocols that may be used include Multi-Protocol Label Switching (MPLS) based on GRE, MPLS based on User Datagram Protocol (UDP), and the like. The virtual router 506 performs tunnel encapsulation / decapsulation on packets originating from / destined for any container of the container pool 502 , and the virtual router 506 exchanges packets with the container pool 502 via the bus 542 and / or the bridge of the NIC 530 .

[0188] As described above, the network controller 24 may provide a logically centralized controller that facilitates the operation of one or more virtual networks. For example, the network controller 24 may maintain a routing information base, such as one or more routing tables that store routing information about a physical network and one or more overlay networks. The virtual router 506 implements one or more virtual routing and forwarding instances (VRFs), such as VRF 222A, for the corresponding virtual networks for which its virtual router 506 operates as a corresponding tunnel endpoint. Generally, each VRF stores forwarding information about the corresponding virtual network and identifies where a data packet is forwarded to and whether the packet is encapsulated in a tunnel protocol, such as with a tunnel header that may include one or more headers of different layers of the virtual network protocol stack. Each VRF may include a network forwarding table that stores routing and forwarding information about the virtual network.

[0189] NIC 530 may receive a tunnel packet. Virtual router 506 processes the tunnel packet to determine the virtual networks of the source endpoint and the destination endpoint of the inner packet from the tunnel encapsulation header. Virtual router 506 may strip off the Layer 2 header and the tunnel encapsulation header to forward only the inner packet internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN tag or an MPLS label, indicating a virtual network (e.g., a virtual network corresponding to VRF 222A). VRF 222A may include forwarding information about the inner packet. For example, VRF 222A may map the destination Layer 3 address of the inner packet to a virtual network interface. Accordingly, VRF 222A forwards the inner packet via the virtual network interface of container pool 502A.

[0190] Container 529A may also originate internal packets as a source virtual network endpoint. For example, container 529A may generate a layer 3 internal packet that is dispatched to a destination virtual network endpoint executed by another computing device (i.e., non-computing device 500) or to another container. Container 529A may send the layer 3 internal packet to virtual router 506 via a virtual network interface attached to VRF 222A.

[0191] The virtual router 506 receives the inner packet and the layer 2 header and determines the virtual network of the inner packet. The virtual router 506 may determine the virtual network using any of the virtual network interface implementation technologies described above (e.g., macvlan, veth, etc.). The virtual router 506 uses the VRF 222A corresponding to the virtual network of the inner packet to generate an outer header for the inner packet, an outer header including an outer IP header for covering the tunnel, and a tunnel encapsulation header identifying the virtual network. The virtual router 506 encapsulates the inner packet with the outer header. The virtual router 506 may encapsulate the tunnel packet with a new layer 2 header having a destination layer 2 address associated with a device located outside the computing device 500 (e.g., a TOR switch 16 or a server 12). If located outside the computing device 500, the virtual router 506 outputs the tunnel packet with the new layer 2 header to the NIC 530 using a physical function. The NIC 530 outputs the packet on the outbound interface. If the destination is another virtual network endpoint executing on the computing device 500 , the virtual router 506 routes the packet to an appropriate one of the virtual network interfaces 212 , 213 .

[0192] In some examples, a controller for computing device 500 (e.g., Figure 1 The network controller 24 in the NIC 530 configures the default route in each container pool 502 to cause the virtual machine 224 to use the virtual router 506 as the initial next hop for outbound packets. In some examples, the NIC 530 is configured with one or more forwarding rules that cause all packets received from the virtual machine 224 to be switched to the virtual router 506.

[0193] Container pool 502A includes one or more application containers 529A. Container pool 502B includes an instance of containerized routing protocol daemon (cRPD) 560. Container platform 588 includes container engine 590, orchestration agent 592, service agent 593, and CNI 570.

[0194] The container engine 590 includes code executed by the microprocessor 510. The container engine 590 can be one or more computer processes. The container engine 590 runs containerized applications in the form of containers 529A-529B. The container engine 590 can represent Dockert, rkt, or other container engines for managing containers. Typically, the container engine 590 receives requests and manages objects such as images, containers, networks, and capacities. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 592, the container engine 590 can obtain an image and instantiate it as an executable container in the container pool 502A-502B.

[0195] The service agent 593 includes code that can be executed by the microprocessor 510. The service agent 593 can be one or more computer processes. The service agent 593 monitors the addition and removal of services and endpoint objects, and, for example, uses services to maintain the network configuration of the computing device 500 to ensure communication between the container pool and the container. The service agent 593 can also manage the management ip table to capture the traffic of the virtual IP address and port of the service and redirect the traffic to the proxy port of the proxy backup container pool. The service agent 593 can represent the kube agent of the slave node of the Kubernetes cluster. In some examples, the container platform 588 does not include the service agent 593, or the service agent 593 is disabled by the CNI 570 to facilitate the configuration of the virtual router 506 and the container pool 502.

[0196] Orchestration agent 592 includes code that can be executed by microprocessor 510. Orchestration agent 592 can be one or more computer processes. Orchestration agent 592 can represent a kubelet of a slave node of a Kubernetes cluster. Orchestration agent 592 is an agent that receives container-specific data of a container and ensures that the container is executed by computing device 500. For example, Figure 1 The container-specifying data may be in the form of a manifest file sent from the orchestrator 23 to the orchestration agent 592 or received indirectly via a command line interface, an HTTP endpoint, or an HTTP server. The container-specifying data may be a container pool specification (e.g., PodSpec - YAML (yet another markup language) or a JSON object describing a container pool) of a container pool 502 of containers. Based on the container-specifying data, the orchestration agent 592 directs the container engine 590 to obtain a container image of the container 529 and instantiate the container image for the computing device 500 to execute the container 529.

[0197] Orchestration agent 592 instantiates CNI 570 and otherwise calls CNI 570 to configure one or more virtual network interfaces of each container pool 502. For example, orchestration agent 592 receives container-specific data for container pool 502A and directs container engine 590 to create container pool 502A with container 529A based on the container-specific data for container pool 502A. Orchestration agent 592 also calls CNI 570 to configure a virtual network interface of a virtual network corresponding to VRF 222A for container pool 502A. In this example, container pool 502A is a virtual network endpoint of a virtual network corresponding to VRF 222A.

[0198] CNI 570 may obtain interface configuration data for configuring the virtual network interface of container pool 502. Virtual router agent 514 operates as a virtual network control plane module that enables network controller 24 to configure virtual router 506. Unlike the orchestration control plane (including container platform 588 for slave nodes and master nodes, e.g., orchestrator 23) that manages provisioning, scheduling, and managing virtual execution elements, the virtual network control plane (including network controller 24 and virtual router agent 514 for slave nodes) manages the configuration of the virtual network partially implemented in the data plane by the virtual router 506 of the slave node. Virtual router agent 514 communicates interface configuration data about the virtual network interface to CNI 570 to enable the orchestration control plane element (i.e., CNI 570) to configure the virtual network interface according to the configuration state determined by network controller 24, thereby bridging the gap between the orchestration control plane and the virtual network control plane. Furthermore, this enables the CNI 570 to obtain interface configuration data for multiple virtual network interfaces of a container pool and configure the multiple virtual network interfaces, which can reduce the communication and resource overhead inherent in invoking a separate CNI 570 for configuring each virtual network interface.

[0199] A containerized routing protocol daemon is described in U.S. application Ser. No. 17 / 649,632, filed on Feb. 1, 2022, the entire contents of which are incorporated herein by reference.

[0200] In addition, CNI 570 may configure virtual router 506 in conjunction with virtual router agent 514 to implement VNR 52. VNR 52 may generate a routing plane configured via one or more import policies and / or one or more export policies to exchange routing information with a common routing target of VNR 52. These policies cause routing information maintained for VN 50 to be exchanged (or in other words, leaked) with a common routing target of VNR 52, which in turn is parsed into forwarding information. CNI 570 may obtain configuration data for installing the forwarding information from network controller 24, thereby interfacing with virtual router agent 514 to install the forwarding information for forwarding packets from VN 50.

[0201] In other words, creating the common routing target enables importing and exporting routing information from one VN 50 (e.g., VN 50A) to the common routing target provided by one or more VNRs 52 and importing and exporting routing information from the common routing target to another VN 50 (e.g., VN 50N). The network controller 24 can parse the routing information into the forwarding information described above and install it in the virtual router 506 to enable forwarding of packets between VN 50A and VN 50N (in some configurations, such as the mesh configuration described above). In this way, the virtual router 506 can establish a method for importing and exporting routing information between VNs 50, which can then be used by the VNs 50 to transmit packets between each other.

[0202] Figure 6 1300 is a block diagram of an exemplary computing device that operates as a computing node of one or more clusters of an SDN architecture system according to the techniques of the present disclosure. The computing device 1300 can represent one or more actual or virtual servers. In some instances, the computing device 1300 can implement one or more master nodes of a corresponding cluster or multiple clusters.

[0203] Although shown and described as being executed by a single computing device 1300, the scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration storage 1328 may be distributed among multiple computing devices that make up a computing system or hardware / server cluster. In other words, each of the multiple computing devices may provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, network controller manager 1326, network controller, SDN controller manager 1325, control node 232A, or configuration storage 1328.

[0204] In this example, computing device 1300 includes a bus 1342 coupled to hardware components in a hardware environment of computing device 1300. Bus 1342 is coupled to a network interface card (NIC) 1330, a storage disk 1346, and one or more microprocessors 1310 (hereinafter referred to as "microprocessors 1310"). In some cases, a front-side bus can couple microprocessor 1310 to a memory device 1344. In some examples, bus 1342 can couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to bus 1342. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

[0205] The microprocessor 1310 may include one or more processors, each of which includes an independent execution unit that executes instructions consistent with an instruction set architecture, instructions stored to a storage medium. The execution units may be implemented as independent integrated circuits (ICs) or may be combined in one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0206] Disk 1346 represents a computer-readable storage medium, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 1310.

[0207] The main memory 1344 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the required program code and program data in the form of instructions or data structures and that can be accessed by the computer. The main memory 1344 provides a physical address space consisting of addressable memory locations.

[0208] The network interface card (NIC) 1330 includes one or more interfaces 1332 configured to exchange packets using the links of the underlying physical network. The interface 1332 may include a port interface card having one or more network ports. The NIC 1330 may also include, for example, on-card memory for storing packet data. Direct memory access transmissions between the NIC 1330 and other devices coupled to the bus 1342 may be read from the NIC memory, or direct memory access transmissions between the NIC 1330 and other devices coupled to the bus 1342 may be written to the NIC memory.

[0209] Memory 1344, NIC 1330, storage disk 1346, and microprocessor 1310 can provide an operating environment for a software stack including an operating system kernel 1314 executed in kernel space. For example, kernel 1314 can represent Linux, Berkeley Software Distribution (BSD), a kernel of another Unix variant, or a Windows server operating system kernel commercially available from Microsoft. In some instances, the operating system can run a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include a kernel-based virtual machine (KVM) for the Linux kernel, Xen, ESXi commercially available from VMware, Windows Hyper-V commercially available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can cover a virtual machine manager (VMM). An operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. Kernel 1314 includes a physical driver 1327 using a network interface card 1330.

[0210] Computing device 1300 may be coupled to a physical network switching fabric that includes software or an overlay network of virtual routers that extend the switching fabric from physical switches to physical servers coupled to the switching fabric, such as virtual router 21. Computing device 1300 may configure slave nodes of a cluster using one or more dedicated virtual networks.

[0211] API server 300A, scheduler 1322, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, and configuration store 1328 may implement a master node of a cluster and may be referred to alternatively as a “master component.” The cluster may be a Kubernetes cluster, and the master node may be a Kubernetes master node, in which case the master component is a Kubernetes master component.

[0212] Each of the API server 300A, the controller 406A, the custom API server 301A, and the custom resource controller 302A includes code that can be executed by the microprocessor 1310. The custom API server 301A verifies and configures data about custom resources of the SDN architecture configuration (such as, VN 50 and VNR 52). A service can be an abstraction that defines a logical collection of container pools and policies for accessing the container pools. A set of container pools that implement the service is selected based on the service definition. A service can be partially implemented as a load balancer or include a load balancer by other means. The API server 300A and the custom API server 301A can implement a representational state transfer (REST) ​​interface to process REST operations and provide a front end as part of the configuration plane of the SDN architecture to the corresponding cluster shared state stored in the configuration storage 1328. The API server 300A can represent a Kubernetes API server.

[0213] Configuration storage 1328 is a backup storage for all cluster data. Cluster data may include cluster state and configuration data. Configuration data may also provide a backend for service discovery and / or provide locking services. Configuration storage 1328 may be implemented as a key value store. Configuration storage 1328 may be a central database or a distributed database. Configuration storage 1328 may represent etcd storage. Configuration storage 1328 may represent Kubernetes configuration storage.

[0214] Scheduler 1322 includes code that can be executed by microprocessor 1310. Scheduler 1322 can be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., container pools in containers) and selects slave nodes for virtual execution elements to run. Scheduler 1322 can select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, locality, etc. Scheduler 1322 can represent a Kubernetes scheduler.

[0215] Typically, API server 1320 may call scheduler 1322 to schedule a pool of containers. Scheduler 1322 may select a slave node and return an identifier for the selected slave node to API server 300A, which may write the identifier to configuration storage 1328 associated with the pool of containers. API server 300 may call an orchestration agent for the selected slave node, so that a container engine for the selected slave node obtains the pool of containers from the storage server and creates a virtual execution element on the slave node. The orchestration agent for the selected slave node may update the status of the container pool to API server 300A, which persists the new status to configuration storage 1328. In this way, computing device 1300 instantiates a new pool of containers in computing infrastructure 8.

[0216] Controller manager 1326 includes code that can be executed by microprocessor 1310. Controller manager 1326 can be one or more computer processes. Controller manager 1326 can be embedded in the kernel control loop to monitor the shared state of the cluster by obtaining notifications from API server 300A. Controller manager 1326 can try to move the state of the cluster toward the desired state. Exemplary controller 406A and custom resource controller 302A can be managed by controller manager 1326. Other controllers can include replication controllers, endpoint controllers, namespace controllers, and service account controllers. Controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, termination container pool garbage collection, cascade deletion garbage collection, node garbage collection, etc. Controller manager 1326 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0217] The network controller for the SDN architecture described herein can provide cloud networks for computing architectures operating on top of the network infrastructure. Cloud networks can include private clouds for enterprises or service providers, infrastructure as a service (IaaS), and virtual private clouds (VPC) for cloud service providers (CSPs). Private clouds, VPCs, and IaaS use cases may involve references such as Figure 1 The multi-tenant virtualized data center described. In this case, multiple tenants located in the data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is allocated its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; virtual networks). These logical resources are isolated from each other unless explicitly allowed by security policies. Virtual networks located in the data center can also be interconnected with physical IP VPNs or L2VPNs.

[0218] A network controller (or "SDN controller") can provide network function virtualization (NFV) for networks such as business edge networks, broadband user management edge networks, and mobile edge networks. NFV involves the orchestration and management of network functions in virtual machines, containers, or other virtual execution elements rather than physical hardware devices, such as firewalls, intrusion detection or prevention systems (IDS / IPS), deep packet inspection (DPI), caching, wide area network (WAN) optimization, etc.

[0219] The SDN controller manager 1325 includes code that can be executed by the microprocessor 1310. The SDN controller manager 1325 can be one or more computer processes. The SDN controller manager 1325 operates as an interface between orchestration-oriented elements (e.g., scheduler 1322, API server 300A and custom API server 301A, controller manager 1326, and configuration storage 1328). Typically, the SDN controller manager 1325 monitors clusters of new Kubernetes local objects (e.g., container pools and services). The SDN controller manager 1325 can isolate container pools in virtual networks and connect container pools to services, as well as interconnect virtual networks using so-called virtual network routers (virtual network routers are not confused with virtual routers that implement virtual network routers in the form of various import and export strategies of common routing targets described above to facilitate interconnection between virtual networks).

[0220] The SDN controller manager 1325 can run as a container of the master node of the cluster. As described herein, in some cases, using the SDN controller manager 1325 can enable disabling the service agent (e.g., Kubernetes kube-agent) of the slave node so that all container pool connections are achieved using a virtual router.

[0221] The components of the network controller 24 can operate as a CNI for Kubernetes and can support multiple deployment modes. CNI 17, CNI 750 are the compute node interfaces of the entire CNI framework for managing the network of Kubernetes. The deployment modes can be divided into two categories: (1) an SDN architecture cluster as a CNI integrated into the workload Kubernetes cluster; and (2) an SDN architecture cluster as a CNI separated from the workload Kubernetes cluster.

[0222] Integrate with workload Kubernetes clusters

[0223] The components of the network controller 24 (e.g., custom API server 301, custom resource controller 302, SDN controller manager 1325, and control node 232) run in a management Kubernetes cluster on a master node close to the Kubernetes controller components. In this mode, the components of the network controller 24 are actually part of the same Kubernetes cluster as the workload.

[0224] Separation from workload Kubernetes cluster

[0225] Components of the network controller 24 are executed by a separate Kubernetes cluster from the workload Kubernetes cluster.

[0226] The SDN controller manager 1325 can use the controller framework for the orchestration platform to listen to (or otherwise monitor) changes to objects defined in the Kubernetes native API and add annotations to some of these objects. An annotation can be a label or other identifier that indicates the characteristics of the object (e.g., "virtual network green"). The SDN controller manager 1325 is a component of the SDN architecture, that is, it listens to Kubernetes kernel resource (such as, container pools, network policies, services, etc.) events and converts these events into custom resources configured by the SDN architecture as needed. The CNI plugin (e.g., CNI 17, 570) is an SDN architecture component that supports the Kubernetes network plugin standard: container network interface.

[0227] The SDN controller manager 1325 can create a network solution for an application using the REST interface exposed by the aggregate API 402 to define network objects such as virtual networks, virtual network interfaces, and access control policies. For example, the network controller 24 component can implement a network solution in a computing infrastructure by configuring one or more virtual networks and virtual network interfaces in a virtual router. (This is just one example of an SDN configuration).

[0228] The following exemplary deployment configuration for the application consists of a container pool and virtual network information for the container pool:

[0229]

[0230]

[0231] This metadata information may be replicated to each container pool replica created by the controller manager 1326. When the SDN controller manager 1325 is informed of these container pools, the SDN controller manager 1325 may create the virtual networks listed in the annotations ("red-network", "blue-network", and "default / extns-network" in the above embodiment) and create a virtual network interface for each container pool replica (e.g., container pool 202A) for each virtual network using a unique private virtual network address from the virtual network's cluster-wide address block (e.g., 10.0 / 16).

[0232] Contrail is an exemplary network controller architecture. Contrail CNI may be a CNI developed for Contrail. A cloud-native Contrail controller may be an example of a network controller described in this disclosure, such as network controller 24.

[0233] Fig. 7A is a block diagram illustrating the control / routing plane of an underlay network and an overlay network configuration using an SDN architecture according to the techniques of this disclosure. Figure 7B is a block diagram illustrating a configured virtual network connecting a pool of containers using tunnels configured in an underlying network in accordance with the techniques of this disclosure.

[0234] The network controller 24 for the SDN architecture can use a distributed or centralized routing plane architecture. The SDN architecture can use a containerized routing protocol daemon (process).

[0235] In terms of network signaling, the routing plane can work according to a distributed model, where cRPD runs on each compute node in the cluster. Essentially, this means that intelligence is built into the compute nodes and the intelligence involves complex configuration at each node. In this model, route reflectors (RRs) may not be able to make intelligent routing decisions, but are used as relays to reflect routes between nodes. A distributed container routing protocol daemon (cRPD) is a routing protocol process that can be used, where each compute node runs its own instance of the routing daemon. At the same time, a centralized cRPD master instance can be used as a RR to relay routing information between compute nodes. Routing and configuration intelligence is distributed across nodes using RRs at a central location.

[0236] Alternatively, the routing plane can work according to a more centralized model, where components of the network controller run centrally and absorb the intelligence required to process configuration information, build network topology, and program the forwarding plane into virtual routers. The virtual router agent is a local agent that processes the information programmed by the network controller. This design results in facilitating more limited intelligence required at the compute nodes and tends to produce simpler configuration states.

[0237] The centralized control plane provides the following:

[0238] Allows the proxy routing framework to be simpler and more lightweight. Hides the complexity and limitations of BGP from the proxy. The proxy does not need to understand concepts like route distinguishers, route targets, etc. The proxy simply exchanges prefixes and builds its corresponding forwarding information.

[0239] There are indeed more control nodes than routes. Control nodes are built on virtual network concepts and can generate new routes using route replication and reorganization (e.g., supporting features like service chaining and intra-VM routing among other use cases).

[0240] Build a BUM tree for optimal broadcast and multicast forwarding.

[0241] It should be noted that the control plane has a distributed nature in certain aspects. As a control plane supporting distributed functions, it allows each local virtual router agent to publish its local routes and subscribe to configurations based on needs.

[0242] Therefore, it makes sense to think about control plane design from a tool POV and use the tool at hand appropriately. Consider the pros and cons of contrail-bgp and cRPD.

[0243] The following functions may be provided by the cRPD or control node of the network controller 24.

[0244] Routing daemon / process

[0245] The control node and cRPD can act as routing daemons that implement different protocols and have the ability to program routing information in the forwarding plane.

[0246] cRPD implements routing protocols with a rich routing stack including Interior Gateway Protocol (IGP) (e.g., Intermediate System to Intermediate System (IS-IS)), BGP-LU, BGP-CT, SR-MPLS / SRv6, Bidirectional Forwarding Detection (BFD), Path Computation Element Protocol (PCEP), etc. It can also be deployed to provide services such as route reflectors only for the control plane and is popular in Internet routing use cases due to these capabilities.

[0247] The control node 232 also implements routing protocols, but is primarily based on BGP. The control node 232 understands overlay networks. The control node 232 provides a rich feature set when overlaying virtualization and caters to SDN use cases. Overlay features such as virtualization (abstraction using virtual networks) and service chaining are very popular among telecom and cloud providers. In some cases, cRPD may not include support for this overlay functionality. However, the rich feature set of cRPD provides strong support for the underlying network.

[0248] Network Orchestration / Automation

[0249] The routing function is only part of the control node 232. An integral part of the overlay network is orchestration. In addition to providing overlay routing, the control node 232 helps model the orchestration function and provides network automation. The core of the orchestration capability of the control node 232 is the ability to model network virtualization using abstractions based on virtual networks (and related objects), including the VNR described above. The control node 232 interfaces with the configuration node 230 to relay configuration information to the control plane and the data plane. The control node 232 also helps build an overlay tree for multicast layer 2 and layer 3. For example, the control node can build a virtual topology of its cluster for this purpose. Typically, cRPD does not include this orchestration capability.

[0250] High availability and horizontal scalability

[0251] The control node design is more centralized, while cRPD is more distributed. There are cRPD worker nodes running on each computing node. On the other hand, the control node 232 does not run on the computing and can even run on a remote cluster (that is, independent and in some cases geographically far away from the workload cluster). The control node 232 also provides horizontal scalability for HA and runs in active-active mode. The computing load is shared between the control nodes 232. On the other hand, cRPD does not usually provide horizontal scalability. The control node 232 and cRPD can provide a smooth restart to HA and can allow data plane operation in headless mode-wherein, even if the control plane is restarted, the virtual router can also run.

[0252] The control plane should be more than just a routing daemon. The control plane should support overlay routing and network orchestration / automation, and cRPD is just as good as routing protocols in managing underlay routing, however, cRPD generally lacks network orchestration capabilities and does not provide strong support for overlay routing.

[0253] Accordingly, in some examples, the SDN architecture can be FIG. 7A to FIG. 7B The compute nodes shown in Figure 4 have cRPDs on them. Fig. 7AAn SDN architecture 700 is shown that can represent an exemplary implementation SDN architecture 200 or 400. In the SDN architecture 700, the cRPD 324 runs on the compute nodes and provides underlying routing to the forwarding plane while running a centralized (and horizontally scalable) set of control nodes 232 that provide orchestration and overlay services. In some examples, instead of running the cRPD 324 on the compute nodes, a default gateway can be used.

[0254] The cRPD 324 on the computing node provides rich underlying routing to the forwarding plane by interacting with the virtual router agent 514 using an interface 540 (which can be a gRPC interface). The virtual router agent interface can allow programming of routing, configuration of virtual network interfaces for overlay, and configuration of the virtual router 506 by other means. This is further described in detail in U.S. Application No. 17 / 649,632. At the same time, one or more control nodes 232 operate as independent container pools that provide overlay services. Thus, the SDN architecture 700 can obtain rich overlays and orchestration provided by the control node 232 and modern underlying routing of the supplementary control node 232 provided by the cRPD 324 on the computing node. An independent cRPD controller 720 can be used to configure the cRPD 324. The cRPD controller 720 can be a device / element management system, a network management system, an orchestrator, a user interface / CLI, or other controller. The cRPD 324 runs a routing protocol and exchanges routing protocol messages with routers, including other cRPDs 324. Each cRPD 324 may be a centralized routing protocol process and in effect operates as a software-only version of the router control plane.

[0255] The enhanced underlay routing provided by cRPD 324 can replace the default gateway of the forwarding plane and provide a rich routing stack for supported use cases. In some examples where cRPD 324 is not used, virtual router 506 relies on the default gateway of the underlay routing. In some examples, cRPD 324 as an underlay routing process is limited to programming the default inet(6).0 structure with only control plane routing information. In this example, a non-default overlay VRF can be programmed by control node 232.

[0256] In this context, the control node 232 can obtain the custom resource that defines the VNR 52 and instantiate separate routing instances (with corresponding common routing targets). The control node 232 can create import and / or export policies for the common routing target to import and export routes from various virtual networks (e.g., VNs 50A and 50N) to the common routing target. The control node 232 can parse the common routing target to obtain forwarding information that is then pushed to the virtual router 506 via the virtual routing target 514. Using this forwarding information, the virtual router 506 can forward packets between VN 50A and VN 50N (in some interconnection schemes, such as the mesh interconnection scheme described above).

[0257] FIG. 7A to FIG. 7B The dual routing / control plane solution described above is shown in FIG. Fig. 7A In some aspects, cRPD 324 provides underlying routing / forwarding information to virtual router agent 514, similar to how a router control plane programs a router forwarding / data plane.

[0258] like Figure 7B , cRPD 324 exchanges routing information that can be used to create a tunnel through the underlying network 702 of the VRF. Tunnel 710 is an example and connects server 12A to the virtual router 506 of server 12X. Tunnel 710 can represent a segment routing (SR) or SRv6 tunnel, a generic routing encapsulation (GRE) tunnel, and an IP-in-IP tunnel, an LSP, or other tunnel. The control node 232 uses tunnel 710 to create a virtual network 712 that connects server 12A of the VRF attached to the virtual network with the container pool 22 of server 12X.

[0259] As described above, cRPD 324 and virtual router agent 514 can exchange routing information using a gRPC interface, and virtual router agent 514 can program virtual router 506 through configuration using a gRPC interface. It should also be noted that control node 232 can be used for overlay and orchestration, while cRPD 324 can be used to manage the underlying routing protocol. Virtual router agent 514 can use a gRPC interface with cRPD 324 while communicating with the control node and domain name service (DNS) using XMPP.

[0260] Because there may be workers running on various compute nodes, the gRPC model performs well for cRPD 324, and the virtual router proxy 316 acts as a gRPC server that exposes the services of the client (cRPD 324) for programming routing and configuration information (underlying layers). Thus, gRPC is an attractive solution when compared to XMPP. Specifically, gRPC transmits data as a binary stream and does not add overhead when encoding / decoding data sent through gRPC.

[0261] In some examples, the control node 232 can use XMPP to interface with the virtual router proxy 514. Since the virtual router proxy 514 acts as a gRPC server, the cRPD 324 acts as a gRPC client. This means that the client (cRPD) needs to initiate a connection to the server (vRouter proxy). In the SDN architecture 700, the virtual router proxy 514 selects the set of control nodes 232 to which it subscribes (because there are multiple control nodes). In this aspect, the control node 232 acts as a server and the virtual router proxy 514 connects as a client and subscribes for updates.

[0262] The control node 232 needs to select the virtual router agent 514 to which it needs to connect through gRPC and then subscribe as a client. Because the control node 232 does not run on each computing node, this requires the implementation of an algorithm for selecting the virtual router agent 514 to which it can subscribe. Further, the control nodes 232 need to synchronize this information with each other. When a restart occurs, this also complicates the situation, and synchronization is required between the control nodes 232 to select the agent it serves. Features such as smooth restart (GP) and fast convergence have been implemented on top of XMPP. XMPP is already lightweight and effective. Therefore, for communication from the control node 232 to the virtual router agent 514, XMPP may be superior to gRPC.

[0263] The additional enhancements of the control node 232 and its use are as follows. HA and horizontal scalability have three control nodes. As with any routing platform, it is sufficient to have only two control nodes 232 to meet the HA requirements. In many cases, this is advantageous. (However, one or more control nodes 232 can be used). For example, it provides a more deterministic infrastructure and is consistent with the best practices of standard routing. Each virtual router agent 514 is attached to a unique pair of control nodes 232 to avoid randomness. With two control nodes 232, debugging may be simpler. In addition, with only two control nodes 232, edge replication for building a multicast / broadcast tree can be simplified. Currently, because the vRouter agent 316 is only connected to two of the three control nodes, all control nodes may not have a complete picture of the tree at a certain time and rely on BGP to synchronize the state between the control nodes. Because the virtual router agent 316 can randomly select two, the three control nodes 232 exacerbate this situation. If there are only two control nodes 232, each virtual router agent 316 is connected to the same control node. This in turn means that the control node 232 does not need to rely on BGP to synchronize state and have the same picture of the multicast tree.

[0264] The SDN architecture 200 can provide ingress replication as an alternative to edge replication and provide an option for the user. Ingress replication can be considered as a special degenerate case of a general overlay multicast tree. However, in practice, the signaling of the ingress replication tree is simpler than that of the general overlay multicast tree. With ingress replication, each virtual router 21 ends up with a tree with itself as the root and each other vrouter as a leaf. A failed virtual router 21 should not theoretically lead to a reconstruction of the tree. It should be noted that the performance of ingress replication deteriorates due to larger clusters. However, ingress replication performs well for smaller clusters. Further, multicast is not a popular and common requirement for many customers. It is mainly limited to transmitting only the initial broadcast BUM traffic.

[0265] Configuration processing module enhancements

[0266] In a conventional SDN architecture, the network controller handles the orchestration of all use cases. The configuration node converts the intent into a configuration object based on the data model and writes the configuration object to a database (e.g., Cassandra). In some cases, for example, notifications are sent to all clients waiting for configuration via RabbitMQ at the same time.

[0267] The control node not only acts as a BGP speaker, but also has a configuration processing module that reads configuration objects from the database in the following way. First, when the control node starts (or restarts), it connects to the database and reads all configurations directly from the database. Secondly, the control node can also be a message client. When there is an update to the configuration object, the control node receives a message notification that lists the objects that have been updated. This again causes the configuration processing module to read the object from the database.

[0268] The configuration processing module reads the configuration objects of the control plane (BGP related configuration) and the vRouter forwarding plane. The configuration can be stored as a graph with objects as nodes and relationships as links. This graph can then be downloaded to the client (BGP / cRPD and / or vRouter agent).

[0269] According to the technology of the present disclosure, in some examples, the API server and message service of the conventional configuration are replaced by the kubeapi-server (API server 300 and custom API server 301) and the previous Cassandra database is replaced by etcd in Kubernetes. Due to this change, clients interested in configuration objects can directly make monitoring on the etcd database to obtain updates instead of relying on RabbitMQ notifications.

[0270] Controller Orchestration of CRPD

[0271] BGP configuration may be provided to cRPD 324. In some examples, cRPD controller 720 may be a Kubernetes controller that caters to a Kubernetes controller that develops its own controller that caters to the Kubernetes space and implements the CRDs needed to orchestrate and provision cRPD 324.

[0272] Distributed configuration processing

[0273] As mentioned before in this section, the configuration processing module can be part of the control node 232. The configuration processing module reads the configuration directly from the database, converts the data into JSON format, and stores it in its local IFMAP database as a graph with objects as nodes and the relationships between them as links. The graph is then downloaded to the interested virtual router agent 514 on the computing node via XMPP. The virtual router agent 514 also builds an IFMAP-based dependency graph locally to store these objects.

[0274] By having the virtual router proxy 514 monitor directly on the etcd server in the API server 300, the need for IFMAP as an intermediate module and the need to store the dependency graph can be avoided. The cRPD 324 running on the compute nodes can use the same model. This will avoid the need for an IFMAP-XMPP configuration channel. The Kubernetes configuration client (for controlling the nodes 232) can be used as part of this configuration. The virtual router proxy can also use this client.

[0275] However, this can increase the number of clients reading configuration from the etcd server, especially in clusters with hundreds of compute nodes. Adding more monitors eventually causes the write rate to drop and the event rate to drop to suboptimal values. etcd's gRPC proxy replays from one server monitor to multiple client monitors. The gRPC proxy merges multiple client monitors (c-monitors) for the same key or range into a single monitor (s-monitor) connected to the etcd server. The proxy broadcasts all events from the s-monitor to its c-monitor. Assuming N clients monitor the same key, a gRPC proxy can reduce the monitoring load on the etcd server from N to 1. Users can deploy multiple gRPC proxies to further distribute the server load. These clients share a server monitor; the proxy effectively offloads resources from the core cluster. With the addition of proxies, etcd can serve a million events per second.

[0276] DNS / Naming in SDN Architecture

[0277] In the previous architecture, DNS services were provided by contrail-dns and contrail-naming processes working together to provide DNS services to VMs in the network. Naming was used as a DNS server that provided an implementation of the BIND protocol, contrail-dns received updates from vrouter-agent and pushed these records to naming.

[0278] The system supports four DNS modes, and IPAM configuration can select the required DNS mode.

[0279] 1. None - No DNS support for the VM.

[0280] 2. Default DNS Server - DNS resolution for the VM is done based on the name server configuration in the server infrastructure. When the VM gets a DHCP response, the subnet default gateway is configured as the DNS server for the VM. DNS requests sent by the VM to the default gateway are resolved via the (fabric) name server configured on the corresponding compute node, and the response is sent back to the VM.

[0281] 3. Tenant DNS Servers - Tenants can use their own DNS servers using this mode. A list of servers can be configured in IPAM, and then sent as DNS servers in the DHCP response to the VM. DNS requests sent by the VM are routed like any other data packets based on the available routing information.

[0282] 4. Virtual DNS Server - In this mode, the system supports virtual DNS servers, thereby providing a DNS server to resolve DNS requests from VMs. A person skilled in the art can define multiple virtual domain name servers under each domain in the system. Each virtual domain name server is an authoritative server for the configured DNS domain.

[0283] The SDN architecture described here is effective in the DNS service it provides. Customers in the cloud-native world benefit from a variety of DNS services. However, with the move to a next-generation Kubernetes-based architecture, the SDN architecture can replace the core DNS with any DNS service.

[0284] Data plane

[0285] The data plane consists of two components: a virtual router agent 514 (aka agent) and a virtual router forwarding plane 506 (also known as a DPDK vRouter / kernel vRouter). The agent 514 in the SDN architecture solution is responsible for managing the data plane components. The agent 514 establishes an XMPP neighbor relationship with the two control nodes 232, and then exchanges routing information with the two control nodes 232. The vRouter agent 514 also dynamically generates flow entries and injects the flow entries into the virtual router 506. This gives instructions to the virtual router 506 on how to forward packets.

[0286] The responsibilities of the agent 514 may include: interfacing with the control node 232 to obtain configurations. Converting the received configurations into a form that the data path can understand (e.g., converting a data model from IFMAP into a data model used by the data path). Interfacing with the control node 232 to manage routing. And collecting statistics from the data path and exporting the statistics to a monitoring solution.

[0287] The virtual router 506 implements data plane functionality that can allow virtual network interfaces to be associated with VRFs. Each VRF has its own forwarding and flow tables, while the MPLS and VXLAN tables are global within the virtual router 506. The forwarding table can contain routes for the IP and MAC addresses of the destination and the IP to MAC association is used to provide proxy ARP capabilities. The label value in the MPLS table is selected by the virtual router 506 when the VM / container interface is started and is only important locally to the vRouter. The VXLAN network identifier is global to all VRFs of the same virtual network in different virtual routers 506 within a domain.

[0288] In some examples, each virtual network has a default gateway address assigned to it, and each VM or container interface receives this address in the DHCP response it receives when it is initialized. When a workload sends a packet to an address outside of its subnet, the ARP for the MAC will correspond to the IP address of the gateway, and the virtual router 506 will respond with its own MAC address. Thus, the virtual router 506 can support a fully distributed default gateway function for all virtual networks.

[0289] The following is an example of packet flow forwarding implemented by virtual router 506.

[0290] Packet flow between VM / container interfaces in the same subnet.

[0291] The worker node can be a VM or a container interface. In some examples, packet processing is performed as follows:

[0292] The VMI / Container Interface needs to send a packet to VM2, so the virtual router 506 first queries its own DNS cache for the IP address, but because this is the first packet, there is no entry.

[0293] When its interface comes up, VM1 sends a DNS request to the DNS server address supplied in the DHCP response.

[0294] The virtual router 506 intercepts the DNS request and forwards it to the DNS server running in the SDN architecture controller.

[0295] The DNS server in the controller responds with the IP address of VM2.

[0296] Virtual router 506 sends the DNS response to VM1.

[0297] VM1 needs to form an Ethernet frame and therefore needs the MAC address of VM2. VM1 checks its own ARP cache, but, because this is the first packet, no entry exists.

[0298] VM1 sends an ARP request.

[0299] • Virtual router 506 intercepts the ARP request and looks up the MAC address of IP-VM2 in its own forwarding table and finds the association in the L2 / L3 routing where the controller sent the MAC address for VM2.

[0300] Virtual router 506 sends an ARP reply to VM1 using the MAC address of VM2.

[0301] A TCP timeout occurred in VM1's network stack.

[0302] VM1's network stack retries to send the packet, and this time, finds VM2's MAC address in the ARP cache and is able to form an Ethernet frame and send it out.

[0303] Virtual router 506 queries VM2's MAC address and finds the encapsulation route. Virtual router 506 constructs the outer header and sends the generated packet to server S2.

[0304] • Virtual router 506 on server S2 decapsulates the packet and looks up the MPLS label to identify the virtual interface to send the original Ethernet frame into. The Ethernet frame is sent into the interface and is received by VM2.

[0305] Packet flow between VMs in different subnets

[0306] In some examples, the order of sending packets to destinations in different subnets is the same, except that virtual router 506 responds as the default gateway. VM1 sends packets in Ethernet frames with the MAC address of the default gateway, which IP address was provided in the DHCP response supplied by virtual router 506 when VM1 booted. When VM1 makes an ARP request for the gateway IP address, virtual router 506 responds with its own MAC address. When VM1 sends an Ethernet frame using the gateway MAC address, virtual router 506 uses the destination IP address of the packet in the frame to look up the forwarding table in the VRF to find a route to the host running the destination via the encapsulation tunnel.

[0307] Fig.10 is a diagram showing a first example of how a virtual network router according to various aspects of the technology described in this disclosure may be configured to implement a mesh interconnection between virtual networks. Fig.10In an embodiment, virtual networks (“VNs”) 1500A (also shown as “VN 1”) and VN 1500B (also shown as “VN 2”) are defined and implemented as custom resources within Kubernetes (or some other orchestration platform), and then implemented as VNs within an SDN architecture such as SDN architecture 700. VN 1500A, VN 1500B can be instances of custom resources and have corresponding customer API servers 301 and custom resource controllers 302.

[0308] The network controller 24 can allow an administrator to define VN 1500A as a custom resource that provides network connectivity to container pool 1502A (also shown as "container pool-1"), and to define VN 1500B as a custom resource that provides network connectivity to container pool 1502B (also shown as "container pool-2").

[0309] like Fig.10 As further shown in the embodiment of , VNs 1500A and 1500N ("VN 1500") are defined within the same namespace 1504 (also shown as "Namespace-1"), indicating that VNs 1500 exist within a common network namespace (and are thereby not isolated from each other). Namespaces, at least in the context of Kubernetes, provide a mechanism for isolating grouped resources within a single cluster, such as container pools 1502A and 1502B ("Container Pool 1502") and VN 1500. As shown in the upper left, VNs 1500 are not interconnected and do not provide network connectivity between each other as indicated by the ping status of container pool-2 (Container Pool 1502A) as "failed" and by the ping status of container pool-1 (Container Pool 1502B) as "failed".

[0310] To provide network connectivity between the container pools 1502, the administrator can interconnect the VNs 1500 to each other by instantiating VNRs 1506 within the same namespace 1504 (again, shown as "namespace-1"). As described above, VNRs 1506 can represent custom resources that the administrator instantiates with an exemplary name of "VNR-web" with a network connection type of "mesh". To specify the VNs to be interconnected, the administrator can assign a label to each VN 1500 as "vn:web" and then specify a mesh connection between the VNs 1500 with a selection label of "vn:web".

[0311] The network controller 24 (e.g., the custom resource controller 302) may process the custom resource to generate the export and import policies described above, i.e., to indicate that all routes of VN 1500A are exported to VN 1500B and all routes of VN 1500B are exported to VN 1500A via VNR 1506, and all routes of VN 1500B are imported to VN 1500A and all routes of VN 1500A are imported to VN 1500B via VNR 1506. Because a mesh is created by exchanging all routes of each VN between each VN, this import / export policy is often referred to as symmetric.

[0312] The VNR 1506 may be "installed" by creating a common routing target (shown as "vnr-RT" where RT represents the routing target) in the context of a common routing instance (RI, shown as "vnr-RI"), where import / export policies are defined to import all routes to the vnrRT and export all routes from each VN 1500. Once these policies are defined and installed within the SDN architecture 700 associated with the VN 1500 (e.g., in the routing plane), routes are imported and exported to the VRFs of each vnr-RT and parsed to generate forwarding information. Instances of the vnr-RI and vnr-RT may be instances of custom resources for SDN architecture configuration.

[0313] The control node 232 may generate forwarding information in the form of configuration data and interface with the virtual router agent 514 to provide the configuration data to the virtual router agent 514, which may install the forwarding information to the virtual router 506 to implement the VN 1500 (e.g., Figure 5 Once installed, the forwarding information causes virtual router 506 and other virtual routers in other computing nodes in the container pool with VN 1500 to be correctly used for inter-VN traffic. Fig.10 As shown in the embodiment, through the indication of ping "passed" of container pool-2 (container pool 1502A) and ping "passed" of container pool-1 (container pool 1502B), these virtual routers between container pools 1502 can use the leaked routing information to correctly forward the ping.

[0314] Fig.11 is a diagram illustrating a second example in which a virtual network router according to various aspects of the technology described in this disclosure may be configured to implement mesh interconnections between virtual networks. Fig.11 The embodiment shown in Fig.10The embodiment shown in is constructed in that VNR 1506 provides a mesh interconnection between VNs 1500 to allow container pools 1502 in the same namespace 1504 to communicate symmetrically with each other.

[0315] exist Fig.11 In the embodiment of , VNs 1500C and 1500D (also shown as "VN 3" and "VN 4", respectively) are defined in the same namespace 1504 and provide network connectivity for corresponding container pools 1502C and 1502D (also shown as "Container Pool-3" and "Container Pool-4"). To add VNs 1500C and 1500D to the mesh provided by 1506, the administrator can add a "vn:web" tag to VNs 1500C and 1500D.

[0316] After adding the labels corresponding to the meshed VNR 1506, the network controller 24 may add VN 1500C and VN 1500D to VNR 1506 by generating import and export policies to export and import all routes between VN 1500C and VN 1500A, VN 1500B, and VN 1500D via the VNR common routing target ("vnr-RT") and to export and import all routes between VN 1500D and VN 1500A-1500C again via the VNR common routing target ("vnr-RT"). That is, the control node 232 may only need to generate policies to import routes from the vnr-RT to VN 1500C and VN 1500D and export routes from VN 1500C and VN 1500D to the vnr-RT. In some cases, by using a common routing target for VNR 1506, routes can be automatically exported between all VNs 1500A-1500D without having to update any existing policies (such as the import / export policy for the existing VN 1500A and VN 1500B).

[0317] In any case, the control node 232 can resolve the route into forwarding information and interface with the virtual router agent 514 to communicate with the virtual router agent 514 through the same protocol as above. Fig.10The forwarding information is installed in a similar manner as described above. After the forwarding information is installed, each container pool 1502A-1502D may have a mesh network connection with each other container pool 1502A-1502D, as shown by the "passed" ping status for each other container pool 1502A-1502D (e.g., container pool 1502A represented as "container pool-1" has a "passed" ping status for each other container pool 1502B-1502D represented by "container pool-2, container pool-3, container pool-4", etc. and for each other container pool 1502B-1502D). Therefore, Fig.11 It is shown that adding additional VNs to a mesh of VNs interconnected using VNRs can be accomplished by simply adding the appropriate label to the VN instance (here, "vn:web").

[0318] Fig. 12A and Fig. 12B is a diagram showing a third example in which a virtual network router according to various aspects of the technology described in the present disclosure may be configured to implement a mesh connection between virtual networks. Fig.11 In the embodiment, VN 1500C and VN 1500D are coupled to Fig. 12A VNR 1506B in the embodiment of the present invention is used to provide a mesh network connection between VN 1500C and VN 1500D (on the right side), wherein: Fig.11 The VNR 1506 in the embodiment is Fig. 12A 1506A to provide a mesh network connection between VN 1500A and VN 1500B (left side).

[0319] VNR 1506A has a unique routing instance ("vnr-RI") in the context of namespace 1504 that is different from the routing instance of VNR 1506B (shown as "vnr-2-RI"). VNR 1506A provides vnr-RT, while VNR 1506B provides vnr-2-RT (although the RTs may be the same given the different / unique routing instances). Further, VNR 1506A has a "vn:web" label option that selects VN 1500A and VN 1500B, while VNR 1506B has a "vn:db" label option, i.e., the label assigned to VN 1500C and VN 1500D. Thus, container pools 1502A and 1502DB do not have network connectivity with container pools 1502C and 1502D (indicated by a "failed" ping status), and container pools 1502C and 1502D do not have network connectivity with container pools 1502A and 1502B (indicated by a "failed" ping status).

[0320] To make the container pool 1502A-1502D have a mesh network connection (because VNR 1506A and VNR 1506B are of type "mesh"), the administrator can add a label selection statement to select the labels of the other VNR 1506A and VNR 1506B. Fig. 12B In the embodiment, VNR 1506A now includes a label selection statement of "vnr:db", and the network controller 24 converts the label selection statement into the import and export policy of vnr-db-RT, thereby importing routes to the vnr-db-RT of VNR 1506B and exporting routes from the vnr-db-RT of VNR 1506B. Similarly, VNR 1506B now includes a label selection statement of "vnr:web", and the network controller 24 can convert the label selection statement into importing routes to the vnr-web-RT of VNR 1506A and exporting routes from the vnr-web-RT of VNR 1506A. All container pools of VNs connected to VNR 1506A ("VNR-web") and VNR 1506B ("VNR-db") can use VNRs and labels to communicate with each other in this way by causing VNRs 1506A and 1506B to import / export each other's routes.

[0321] Again, the network controller 24 (e.g., custom resource controller 302) can deploy these policies in the routing plane, and then the network controller can perform routing information exchange. The control node 232 can parse the exchanged routing information into forwarding information and interface with the virtual router agent 514 to communicate with the virtual router agent 514 through the above reference. Fig.10 The forwarding information is installed in virtual router 506 in a similar manner as described above. Thus, a mesh connection between container pools 1502A-1502D is established, as evidenced by a successful "passed" ping status of each of the other container pools 1502A-1502D.

[0322] Fig.13A and Fig. 13B is a diagram showing a fourth example in which a virtual network router according to various aspects of the technology described in this disclosure may be configured to implement mesh connections between virtual networks. Fig.13A In the embodiment of Fig. 12AThe structure is similar to the embodiment shown in , but VNs 1500A and 1500B, container pools 1502A and 1502B, and VNR 1506A are located in a first namespace 1504A (shown as "Namespace-1"), while VNs 1500C and 1500D, container pools 1502C and 1502D, and VNR 1506B are located in a second different namespace 1504B (shown as "Namespace-2").

[0323] Assuming that namespaces 1504A and 1504B isolate VN 1500A / 1500B, container pools 1502A / 1502B, and VNR 1506A from VN 1500C / 1500D, container pools 1502C / 1502D, and VNR 1506B, authorization to import routing information across namespaces 1504A and 1504B must exist because each namespace 1504A and 1504B is managed by a separate group, rather than sharing management authority across the two namespaces 1504A and 1504B. Therefore, a request can be sent to the administrators of the two namespaces 1504A and 1504B to confirm permission to import and export routing information between VNR 1506A and VNR 1506B.

[0324] Furthermore, due to the isolation provided by namespaces 1504A and 1504B relative to other namespaces, only VNR 1506A and VNR 1506B are permitted to import and export routing information across namespaces 1504A and 1504B. This restriction on routing information between namespaces enforces the authorization described above and avoids misconfigurations that could result in the exchange of routing information, i.e., permitting packets with sensitive or other secure information to be transmitted between namespaces 1504A and 1504B (which could result in data compliance violations, invalidation of contractual obligations, or other malicious information that could result in a violation of the isolation provided by namespaces 1504A and 1504B).

[0325] Therefore, assuming authorization is provided by the administrators of the two namespaces 1504A and 1504B, the control node 232 can convert VNRs 1506A and 1506B into the above referenced Fig. 12B The import and export policies described in the embodiments of the present invention facilitate the generation of forwarding information to provide a mesh connection between the container pools 1502A-1502D. Fig. 13B The result of this conversion is shown with the policy installation and resolution of the exchanged routing information into forwarding information that can be installed in virtual router 506 to enable mesh networking between the namespaces.

[0326] Fig.14is a diagram showing a fifth example of a virtual network router that can be configured to implement hub-and-spoke interconnections between virtual networks according to various aspects of the technology described in this disclosure. Fig.14 In the embodiment of , VN 1500A-1500C is located in namespace 1504 and provides network connection for corresponding container pool 1502A-1502C. VN 1500A and VN 1500B are each assigned a VN label of "vn:web" and VN 1500C is assigned a VN label of "vn:db".

[0327] In addition, the administrator can interact with the network controller 24 to instantiate custom resources in the form of VNR 1516A and VNR 1516B. VNR 1516A is a "hub" type VNR that serves as the center in a hub-and-spoke network configuration. A hub-and-spoke network configuration is an exchange in which the center VNR 1516A imports all routing information from any branch VNR coupled to the center VNR 1516A and the branch VNRs coupled to the center VNR 1516A import routing information from the center VNR 1516A. VNR 1516B is configured as a "branch" type VNR that serves as a branch in a hub-and-spoke network configuration. Branch VNR 1516B can export all routing information to the center VNR 1516A, but import all routing information from the center VNR 1516A. However, branch VNs do not receive any routing information from other branch VNs.

[0328] The administrator may define the central VNR 1516A to select the label “vn:db”, and the administrator may define the branch VNR 1516B to select the label “vn:web.” As described above, the label selector may indicate which of the VNs 1500A-1500C is coupled to which of the central VNR 1516A and branch VNR 1516B.

[0329] exist Fig.14 In the embodiment, the control node 232 generates a central common routing target ("hub-RT" or "vnr-hub-RT") for the central VNR 1516A (and spoke-RI or vnr-spoke-RI) and generates a branch common routing target ("spoke-RT" or "vnr-spoke-RT") for the branch VNR 1516B (and hub-RI or vnr-hub-RI). Then, the control node 232 can generate an export policy for the central VNR 1516A to guide any VN 1500A-1500C (in this embodiment, VN 1500C) with the label "vn:db" to export routing information to the vnr-hub-RT.

[0330] The control node 232 may also generate an export policy for the branch VNR 1516B to indicate that the branch VNR 1516B should export all routing information to the vnr-hub-RT. The control node 232 may also generate an import policy for the hub VNR 1516A to indicate that the hub VNR 1516A should import all routing information from any branch VNR, such as the vnr-spoke-RT of the branch VNR 1516B. The control node 232 may also generate import and export policies indicating that VN 1500A and VN 1500B should export all routing information to the vnr-spoke-RT.

[0331] Once the policy is configured, the control node 232 can exchange routing information according to the policy, thereby parsing the routing information and generating forwarding information. The control node 232 can install the forwarding information into the virtual router 506 via the virtual router agent 514 to implement a hub-and-spoke connection, wherein each container pool 1502C can communicate with each container pool 1502A and 1502B (because the center of the connected container pool can reach any branch container pool) and the container pools 1502A and 1502B can communicate with the central container pool 1502C, but because the branch container pools 1502A and 1502B cannot communicate with each other according to each hub-and-spoke network connection, the branch container pools 1502A and 1502B do not communicate with each other.

[0332] Fig.15 is a diagram illustrating a sixth example in which a virtual network router according to various aspects of the techniques described in this disclosure may be configured to implement hub-and-spoke interconnections between virtual networks. Fig.15 The embodiments shown in Fig.14 However, in this example, the central VNR 1516A, VN 1500C, and container pool 1502C are located in a different namespace 1504B (shown as "namespace-2"), while the branch VNR 1516B, VN 1500A and VN 1500B, and container pools 1502A and 1502B are located in a first namespace 1504A (shown as "namespace-1").

[0333] In this example, VNs 1500A-1500C can only interconnect with VNRs 1516A and 1516B in the same namespace, and only branch VNRs 1516B and hub VNRs 1516A can communicate across namespaces 1504A and 1504B for the reasons described above. Assuming authorization is given by the administrators of both namespaces 1504A and 1504B, it is possible to communicate in the same manner as described above. Fig.14Achieve in a manner similar (if not substantially similar) to that described Fig.15 The hub-and-spoke connection is shown at the bottom.

[0334] Fig.16 is a diagram illustrating a seventh example in which a virtual network router may be configured to implement hub-and-spoke interconnections between virtual networks in accordance with various aspects of the techniques described in this disclosure. Fig.16 The embodiments shown in Fig.14 The embodiment shown in is similar to that shown in , however, an additional mesh VNR 1506 is provided to allow modification of the hub-and-spoke interconnection between VNs 1500A-1500D.

[0335] As described in more detail above, Fig.16 In an embodiment of the present invention, VNs 1500A-1500D are initially unable to communicate with each other (based on a "failed" ping status) until hub VNR 1516A and spoke VNR 1516B are instantiated, deployed as policies, exchange routing information, and then resolve and install forwarding information into virtual router 516. This enables VN 1500C and VN 1500D to export routing information to a common hub-RT ("vnr-hub-RT"), while VN 1500A and VN 1500B export routing information to a common spoke-RT ("vnr-spoke-RT"). Spoke VNR 1516B can export all routing information to the vnr-hub-RT, which enables hub container pools 1502C and 1502D to communicate with each other spoke container pool 1502A and 1502B. The branch container pools 1502A and 1502B are able to communicate with each other central container pool 1502C and 1502D, but not with each other branch container pool 1502A and 1502B.

[0336] In order to enable the central container pools 1502C and 1502D to communicate directly with each other, the administrator can instantiate the mesh VNR 1506 to be able to export routing information to the same reference as above. Fig.10 Thus, since all routing information is exchanged between VN 1500C and VN 1500D via Mesh-RT, central container pools 1502C and 1502D can communicate directly with each other.

[0337] 17A to 17D is a flowchart illustrating operations performed when establishing one or more virtual network routers within an SDN architecture according to various aspects of the technology described in the present disclosure. Fig.17AIn an embodiment, a VNR controller of the network controller 24 (e.g., a custom resource controller 302, such as, Figure 6 The custom resource controller 302A shown in the embodiment of FIG. 3 may initially identify (or in other words, monitor the Ectd database to identify) the creation of VN 50A with the name VN-1 and the label vn=web and VN 50N with the name VN-2 and the label vn=web as custom resources in the Ectd database associated with the network controller 24 (e.g., Figure 6 Configuration storage 1328) shown in the embodiment of the network controller 24, wherein the controller manager of the network controller 24 (e.g., controller manager 1326) calls the scheduler of the network controller 24 (e.g., scheduler 1322) to create VN 50A and VN 50N (1700, 1702, 1704).

[0338] Via the kube-api of the network controller 24 (again, may refer to Figure 6 The administrator can instantiate VNR 52A to interconnect VN 50A and VN 50N, specifying the type as mesh and the label as "vn:web". Kube-api updates the Etcd database to store the new custom resource created 52A and states that the creation of VNR 52A is pending in the Etcd database (1706, 1708, 1710).

[0339] Next, the scheduler of the network controller 24 may receive notification from the Etcd database that the new custom resource has been defined, and then interface with the controller manager to notify the controller manager of the VNR creation request (1712, 1714, 1716). The controller manager may interface with the VNR controller to request VNR coordination to create an RI for VNR 52A, thereby providing the new RI to kube-API (1718), so that kube-api can update Etcd to reflect the allocation of the new RI to VNR 52A (1720).

[0340] The Etcd database may then interface with the scheduler to indicate the presence of a VNR-RI creation event (1722), which in turn interfaces with the controller manager to notify the controller manager of the VNR-RI creation (1724). The controller manager then interfaces with the RI controller of the network controller 24 to request VNR-RI coordination (1726).

[0341] Next, refer to Fig. 17BIn the embodiment, the RI controller of the network controller 24 can perform coordination of VNR-RI to create RT for VNR-1-RI, thereby reporting VNR-1-RI back to kube-api (1728). kube-api can update Etcd to indicate that VNR-1-RT has been created, which can report to Kube-api that VNR-1-RT is pending and then successfully created (1730, 1732). Kube-api can then interface with the VNR controller to indicate that VNR-1-RI has been created, which in turn can obtain VNR-1-RT from RI (1736, 1738).

[0342] Next, the VNR controller can interface with Kube-api to identify a list of VN 50s with label vn=web (1740), where Kube-api can return the names VN-1 and VN-2 to VN 50A and VN 50N (1742). Again, the VNR controller can interface with Kube-api to obtain the RIs of VN-1 and VN-2 (1744), which continues to interface with Kube-api to patch VNR-1-RT in VN-1-RI, VN-2-RI states (1746). Kube-api then updates the Etcd of the VN-RI patch event (1748). Etcd then notifies the scheduler of the patch event (1750), thereby notifying the controller manager of the patch event (1752).

[0343] refer to Fig. 17C In the embodiment, the controller manager requests the RI controller to perform VN-RI coordination in response to a patching event (1754), and the RI controller performs coordination on VN-RI before requesting Kube-api to patch VNR-1-RT in the VN-1-RI and VN-2-RI states (1756). In response to the patching request, Kube-api updates the VN-1-RI in Etcd to include policies for importing and exporting to VNR-1-RT (a common routing target for VNRs), and Kube-API updates VN-2-RI to include policies for importing and exporting to VNR-1-RT (a common routing target for VNRs) (1758). Etcd reports the status of the patching back to Kube-API (1760). In response to the patching status of VN-1-RI and VN-2-RI, Kube-api interfaces with the RI controller to patch VNR-1-RT at VNR-RI (1762), wherein the RI controller reports the patching status back to Kube-api (1764).

[0344] Next, refer to Fig.17DIn an embodiment, Kube-api interfaces with Etcd to update Etcd for the patch event, and Etcd interfaces with the scheduler to notify the scheduler of the patch event (1766, 1768). The scheduler then interfaces with the controller manager to notify the controller manager of the VNR-RI patch, and the controller manager in turn requests VNR-RI coordination from the VNR controller (1770, 1772). The RI controller patches VNR-1-RT at VNR-RI and notifies Kube-api of the patch (1774, 1776). Kube-api updates Etcd for the patch (1778) and interfaces with the RI controller to set the VNR-1 status to indicate "success", so Kube-api updates Etcd when the VNR-1 status is successful (1780).

[0345] The following is the detailed design of the VNR API solution (as a YAML file):

[0346] API Type

[0347] type VirtualNetworkRouterSpec struct{

[0348] / / Common spec fields

[0349] CommonSpec`json:",inline"protobuf:"bytes,1,opt,name=commonSpec"`

[0350] / / Type of VirtualNetworkRouter.valid types-mesh,spoke,hub

[0351] Type VirtualNetworkRouterType`json:"type,omitempty"

[0352] protobuf:"bytes,2,opt,name=type"`

[0353] / / Select VirtualNetworks to which this VNR's RT be shared

[0354] VirtualNetworkSelector*metav1.LabelSelector

[0355] `json:"virtualNetworkSelector,omitempty"

[0356] protobuf:"bytes,3,opt,name=virtualNetworkSelector"`

[0357] / / Import Router targets from other virtualnetworkrouters

[0358] Import ImportVirtualNetworkRouter`json:"import,omitempty"

[0359] protobuf:"bytes,4,opt,name=import"`

[0360] }

[0361] type ImportVirtualNetworkRouter struct{

[0362] VirtualNetworkRouters[]VirtualNetworkRouterEntry

[0363] `json:"virtualNetworkRouters,omitempty"

[0364] protobuf:"bytes,1,opt,name=virtualNetworkRouters"`

[0365] }

[0366] type VirtualNetworkRouterEntry struct{

[0367] VirtualNetworkRouterSelector*metav1.LabelSelector

[0368] `json:"virtualNetworkRouterSelector,omitempty"

[0369] protobuf:"bytes,1,opt,name=virtualNetworkRouterSelector"`

[0370] NamespaceSelector*metav1.LabelSelector

[0371] `json:"namespaceSelector,omitempty"

[0372] protobuf:"bytes,2,opt,name=namespaceSelector"`

[0373] }

[0374] VirtualNetworkRouter Yamls-Mesh VNR

[0375] apiVersion:core.contrail.juniper.net / v1alpha1

[0376] kind:VirtualNetworkRouter

[0377] metadata:

[0378] namespace:frontend

[0379] name:vnr-1

[0380] annotations:

[0381] core.juniper.net / display-name:vnr-1

[0382] labels:

[0383] vnr:web

[0384] ns:frontend

[0385] spec:

[0386] type:mesh

[0387] virtualNetworkSelector:

[0388] matchLabels:

[0389] vn:web

[0390] import:

[0391] virtualNetworkRouters:

[0392] -virtualNetworkRouterSelector:

[0393] matchLabels:

[0394] vnr:db

[0395] namespaceSelector:

[0396] matchLabels:

[0397] ns:backend

[0398] VirtualNetworkRouter Yamls-Spoke VNR

[0399] apiVersion:core.contrail.juniper.net / v1alpha1

[0400] kind:VirtualNetworkRouter

[0401] metadata:

[0402] namespace:frontend

[0403] name:vnr-1

[0404] annotations:

[0405] core.juniper.net / display-name:vnr-1

[0406] labels:

[0407] vnrgroup:spokes

[0408] ns:frontend

[0409] spec:

[0410] type:spoke

[0411] virtualNetworkSelector:

[0412] matchLabels:

[0413] vngroup:spokes

[0414] import:

[0415] virtualNetworkRouters:

[0416] -virtualNetworkRouterSelector:

[0417] matchLabels:

[0418] vnrgroup:hubs

[0419] namespaceSelector:

[0420] matchLabels:

[0421] ns:backend

[0422] VirtualNetworkRouter Yaml-Hub VNR

[0423] apiVersion:core.contrail.juniper.net / v1alpha1

[0424] kind:VirtualNetworkRouter

[0425] metadata:

[0426] namespace:backend

[0427] name:vnr-2

[0428] annotations:

[0429] core.juniper.net / display-name:vnr-2

[0430] labels:

[0431] vnrgroup:hubs

[0432] ns:backend

[0433] spec:

[0434] type:hub

[0435] virtualNetworkSelector:

[0436] matchLabels:

[0437] vngroup:hubs

[0438] import:

[0439] virtualNetworkRouters:

[0440] -virtualNetworkRouterSelector:

[0441] matchLabels:

[0442] vnrgroup:spokes

[0443] namespaceSelector:

[0444] matchLabels:

[0445] ns:frontend

[0446] Fig.18 is to illustrate the various aspects of the technology described herein when performing Figure 1 Another flow chart of the operation of the computer architecture shown in an embodiment of the present invention. The network controller 24 can be configured to interconnect a first virtual network (e.g., VN 50A) and a second virtual network (e.g., VN 50N) operating within the SDN architecture system represented by the data center 10. The network controller 24 can be configured to define a logical abstraction of one or more policies to perform the interconnection via one or more VNRs 52 (e.g., VNR 52A).

[0447] Policies may include import and export policies regarding routing information maintained by virtual routers (in this embodiment, may refer to VN 50A and VN 50N). That is, Kubernetes may be extended via a custom resource representing VNR 52A to convert VNR 52A into one or more import and export policies deployed relative to VN 50A and VN 50N to configure mutual communication via the distribution of routing information between VN 50A and VN 50N. Once configured, VN 50A may then export routing information (e.g., routing representing VN 50A) to VN 50N and import routing information (e.g., routing representing VN 50N) to VN 50A. Similarly, VN 50N may export routing information (e.g., routing representing VN 50N) to VN 50A and import routing information (e.g., routing representing VN 50A) to VN 50N.

[0448] Abstraction may hide the underlying routing configuration to implement this routing disclosure, such as defining routing targets for importing and exporting routing information for routing instances that implement VN 50A and VN 50N. Alternatively, network controller 24 may convert VNR 52A to a common routing target and configure communication of routing information via the common routing target for routing instances that implement VN 50A and VN 50N (in this embodiment).

[0449] To achieve mesh connection, the network controller 24 may configure the import and export of routing instances of VN 50A, VN 50N, and VNR 52A with routing targets associated with VN 50A, VN 50N, and VNR 52A. To achieve hub-and-spoke connection, the network controller 24 may configure the export of routing instances associated with VN 50A and VN 50N to export routing information to routing instances associated with VNR 52A (serving as a hub), and configure the routing instances of VNR 52A to import routing information to routing instances associated with VN 50A and VN 50N. In this hub-and-spoke connection, VN 50A and VN 50N may not communicate directly with each other.

[0450] In this regard, the network controller 24 may process a request to create, for example, VNR 52A, wherein VNR 52A represents a logical abstraction of one or more policies that cause one or more imports and exports of routing information to occur between a first virtual network (e.g., VN 50A) and a second virtual network (e.g., VN 50N) (1800). The network controller 24 may then configure VN 50A and VN 50N according to the one or more policies to enable one or more imports and exports of routing information between VN 50A and VN 50N via VNR 52A (1802).

[0451] Likewise, various aspects of the technology can implement the following examples.

[0452] Example 1. A network controller for a software defined network (SDN) architecture system, the network controller comprising: a processing circuit; a configuration node configured to be executed by the processing circuit; and a control node configured to be executed by the processing circuit; wherein the configuration node is executable to process a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within the SDN architecture system, the virtual network router representing a logical abstraction of one or more policies, the one or more policies causing one or more of the importing and exporting of routing information between the first virtual network and the second virtual network to occur; and wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of the importing and exporting of routing information between the first virtual network and the second virtual network via the virtual network router.

[0453] Example 2. A network controller according to Example 1, wherein the request includes a label associated with the first virtual network and the second virtual network; and wherein the configuration node identifies the first virtual network and the second virtual network based on the label to configure a routing instance corresponding to the virtual network router according to the one or more policies to enable import and export of routing information between the first virtual network and the second virtual network.

[0454] Example 3. A network controller according to any one of Examples 1 and 2, wherein the request indicates that the virtual network router is a mesh virtual network router; and wherein the one or more policies represented by the mesh virtual network router include symmetric import and export policies that cause routing information to be imported and exported between the first virtual network and the second virtual network.

[0455] Example 4. A network controller according to any one of Examples 1 to 3, wherein the request indicates that the virtual network router is a central virtual network router, the first virtual network is a first branch virtual network, and the second virtual network is a second branch virtual network; and wherein the one or more policies represented by the central virtual network router include an asymmetric import and export policy that causes routing information to be exported from the first branch virtual network and the second branch virtual network to the virtual network router, but routing information is not imported between the first branch virtual network and the second branch virtual network.

[0456] Example 5. A network controller according to any one of Examples 1 to 4, wherein the first virtual network is associated with a first namespace; wherein the second virtual network is associated with a second namespace; wherein the virtual network router is a first virtual network router; wherein the one or more policies include a first policy and a second policy; wherein the first virtual network router represents a first policy for importing and exporting routing information between the first virtual network and the second virtual network router; and wherein the second virtual network router represents a second policy for importing and exporting routing information between the second virtual network and the first virtual network router.

[0457] Example 6. The network controller of example 5, wherein the first virtual network router must obtain authorization to interconnect the first virtual network router with the second virtual network router before deploying the first policy.

[0458] Example 7. A network controller according to any one of Examples 1 to 6, wherein the network controller supports a container orchestration system.

[0459] Example 8. A network controller according to any one of Examples 1 to 7, wherein the virtual network router is an instance of a custom resource configured for an SDN architecture for an SDN architecture; and wherein the configuration node includes a custom resource controller that coordinates the state of the SDN architecture into an expected state of the virtual network router; and wherein the control node is configured to deploy the virtual network router to achieve the expected state of the virtual network router.

[0460] Example 9. A network controller according to any one of Examples 1 to 8, wherein a virtual network router is implemented using a common routing instance and a common routing target; wherein a first virtual network is implemented using a first routing instance and a first routing target; wherein a second virtual network is implemented using a second routing instance and a second routing target; and wherein a control node is configured to configure the import and export of the first routing instance, the second routing instance, and the common routing instance with the first routing target, the second routing target, and the common routing target to achieve a mesh connection.

[0461] Example 10. A network controller according to any one of Examples 1 to 9, wherein a virtual network router is implemented using a common routing instance and a common routing target; wherein a first virtual network is implemented using a first routing instance and a first routing target; wherein a second virtual network is implemented using a second routing instance and a second routing target; wherein a third virtual network is implemented using a third routing instance and a third routing target; and wherein a control node is configured to configure the import and export of the first routing instance, the second routing instance, the third routing instance, and the common routing instance with the first routing target, the second routing target, the third routing target, and the common routing target to achieve a hub-and-spoke connection.

[0462] Example 11. A network controller according to any one of Examples 1 to 10, wherein the first virtual network is implemented using one or more first virtual routers in the first plurality of computing nodes; and wherein the second virtual network is implemented using one or more second virtual routers in the second plurality of computing nodes.

[0463] Example 12. The network controller of Example 11, wherein the first plurality of computing nodes is a first Kubernetes cluster; and wherein the second plurality of computing nodes is a second Kubernetes cluster.

[0464] Example 13. A network controller according to any one of Examples 1 to 12, wherein a control node parses routing information into forwarding information for one or more virtual networks in a first virtual network and a second virtual network and installs the forwarding information in a virtual router supporting interconnection between the first virtual network and the second virtual network.

[0465] Example 14. A method, comprising: processing, by a network controller, a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within an SDN architecture system, the virtual network router represents a logical abstraction of one or more policies, the one or more policies causing one or more of the importing and exporting of routing information between the first virtual network and the second virtual network to occur; and configuring, by the network controller, the first virtual network and the second virtual network according to the one or more policies to enable one or more of the importing and exporting of routing information between the first virtual network and the second virtual network via the virtual network router.

[0466] Example 15. A method according to Example 14, wherein the request includes a label associated with the first virtual network and the second virtual network; and wherein processing the request includes: identifying the first virtual network and the second virtual network based on the label to configure a routing instance corresponding to the virtual network router according to the one or more policies to enable importing and exporting routing information between the first virtual network and the second virtual network.

[0467] Example 16. A method according to any one of Examples 14 and 15, wherein the request indicates that the virtual network router is a mesh virtual network router; and wherein the one or more policies represented by the mesh virtual network router include symmetric import and export policies that cause the import and export of routing information between the first virtual network and the second virtual network.

[0468] Example 17. A method according to any one of Examples 14 and 16, wherein the request indicates that the virtual network router is a central virtual network router, the first virtual network is a first branch virtual network, and the second virtual network is a second branch virtual network; and wherein the one or more policies represented by the central virtual network router include an asymmetric import and export policy that causes routing information to be exported from the first branch virtual network and the second branch virtual network to the virtual network router, but routing information is not imported between the first branch virtual network and the second branch virtual network.

[0469] Example 18. A method according to any one of Examples 14 to 17, wherein the first virtual network is associated with a first namespace; wherein the second virtual network is associated with a second namespace; wherein the virtual network router is a first virtual network router; wherein the one or more policies include a first policy and a second policy; wherein the first virtual network router represents a first policy for importing and exporting routing information between the first virtual network and the second virtual network router; and wherein the second virtual network router represents a second policy for importing and exporting routing information between the second virtual network and the first virtual network router.

[0470] Example 19. The method of example 18, wherein the first virtual network router must obtain authorization to interconnect the first virtual network router with the second virtual network router before deploying the first policy.

[0471] Example 20. A method according to any one of Examples 14 to 19, wherein the network controller supports a container orchestration system.

[0472] Example 21. A method according to any one of Examples 14 to 20, wherein the virtual network router is an instance of a custom resource of an SDN architecture configuration for an SDN architecture; and wherein processing the request includes: coordinating the state of the SDN architecture to an expected state of the virtual network router; and deploying the virtual network router to achieve the expected state of the virtual network router.

[0473] Example 22. A method according to any one of Examples 14 to 21, wherein a virtual network router is implemented using a common routing instance and a common routing target; wherein a first virtual network is implemented using a first routing instance and a first routing target; wherein a second virtual network is implemented using a second routing instance and a second routing target; and wherein processing a request includes: configuring the import and export of the first routing instance, the second routing instance, and the common routing instance with the first routing target, the second routing target, and the common routing target to achieve a mesh connection.

[0474] Example 23. A method according to any one of Examples 14 to 22, wherein a virtual network router is implemented using a common routing instance and a common routing target; wherein a first virtual network is implemented using a first routing instance and a first routing target; wherein a second virtual network is implemented using a second routing instance and a second routing target; wherein a third virtual network is implemented using a third routing instance and a third routing target; and wherein processing a request includes: configuring the import and export of the first routing instance, the second routing instance, the third routing instance, and the common routing instance with the first routing target, the second routing target, the third routing target, and the common routing target to achieve a hub-and-spoke connection.

[0475] Example 24. A method according to any one of Examples 14 to 23, wherein the first virtual network is implemented using one or more first virtual routers in the first plurality of computing nodes; and wherein the second virtual network is implemented using one or more second virtual routers in the second plurality of computing nodes.

[0476] Example 25. The method of Example 24, wherein the first plurality of computing nodes is a first Kubernetes cluster; and wherein the second plurality of computing nodes is a second Kubernetes cluster.

[0477] Example 26. The method according to any one of Examples 14 to 25 further includes: parsing the exchanged routing information into forwarding information for one or more virtual networks in the first virtual network and the second virtual network and installing the forwarding information in a virtual router supporting the interconnection between the first virtual network and the second virtual network.

[0478] Example 27. A non-transitory computer-readable medium comprising instructions for causing a processing circuit of a network controller to: process a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within an SDN architecture system, the virtual network router representing a logical abstraction of one or more policies that cause one or more of the import and export of routing information between the first virtual network and the second virtual network; and configure the first virtual network and the second virtual network according to the one or more policies to enable one or more of the import and export of routing information between the first virtual network and the second virtual network via the virtual network router. The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or may be implemented separately as discrete, but interoperable or logic devices or other hardware devices. In some cases, the various features of the electronic circuit may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0479] If implemented in hardware, the present disclosure may be directed to an apparatus such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or in addition, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium that includes instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store the instructions that are executed by a processor.

[0480] The computer-readable medium may constitute part of a computer program product including packaging materials. The computer-readable medium may include computer data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0481] In some examples, computer-readable storage media may include non-volatile media. The term "non-volatile" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In a specific example, a non-volatile storage medium may store data that changes over time (e.g., stored in RAM or cache).

[0482] The code or instructions may be software and / or firmware executed by a processing circuit including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, as used herein, the term "processor" may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A network controller for a software defined network (SDN) architecture system, the network controller comprising: Processing circuit; a configuration node configured to be executed by the processing circuit; as well as a control node configured to be executed by the processing circuit; wherein the configuration node is executable to process a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within the SDN architecture system, the virtual network router represents a logical abstraction of one or more policies, the one or more policies causing one or more of importing and exporting routing information between the first virtual network and the second virtual network; wherein the request indicates that the virtual network router is a mesh virtual network router or a central virtual network router, the first virtual network is a first branch virtual network, and the second virtual network is a second branch virtual network; and wherein the one or more policies represented by the mesh virtual network router include symmetric import and export policies, the symmetric import and export policies causing import and export of the routing information between the first virtual network and the second virtual network; wherein the one or more policies represented by the central virtual network router include an asymmetric import and export policy, the asymmetric import and export policy causing the routing information from the first branch virtual network and the second branch virtual network to be exported to the virtual network router, but not importing the routing information between the first branch virtual network and the second branch virtual network; and The control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of importing and exporting the routing information between the first virtual network and the second virtual network via the virtual network router.

2. The network controller according to claim 1, in, The request includes tags associated with the first virtual network and the second virtual network; and The configuration node identifies the first virtual network and the second virtual network based on the label to configure the routing instance corresponding to the virtual network router according to the one or more policies, so as to import and export the routing information between the first virtual network and the second virtual network.

3. The network controller according to claim 1, in, The first virtual network is associated with a first namespace; Wherein, the second virtual network is associated with a second namespace; Wherein, the virtual network router is a first virtual network router; wherein the plurality of strategies among the one or more strategies include a first strategy and a second strategy; wherein the first virtual network router represents the first policy for importing and exporting the routing information between the first virtual network and the second virtual network router; and The second virtual network router represents the second policy for importing and exporting the routing information between the second virtual network and the first virtual network router.

4. The network controller according to claim 3, wherein: The first virtual network router must obtain authorization to interconnect the first virtual network router and the second virtual network router before deploying the first policy.

5. The network controller according to claim 1, wherein: The network controller supports a container orchestration system.

6. The network controller according to any one of claims 1 to 5, in, The virtual network router is an instance of a custom resource of an SDN architecture configuration for the SDN architecture system; and wherein the configuration node comprises a custom resource controller that coordinates the state of the SDN architecture system to the expected state of the virtual network router; and The control node is configured to deploy the virtual network router to achieve the expected state of the virtual network router.

7. The network controller according to any one of claims 1 to 5, in, Implementing the virtual network router using a common routing instance and a common routing target; wherein the first virtual network is implemented using a first routing instance and a first routing target; wherein the second virtual network is implemented using a second routing instance and a second routing target; and The control node is configured to configure the import and export of the first routing instance, the second routing instance, and the common routing instance with the first routing target, the second routing target, and the common routing target to achieve a mesh connection.

8. The network controller according to any one of claims 1 to 5, in, Implementing the virtual network router using a common routing instance and a common routing target; wherein the first virtual network is implemented using a first routing instance and a first routing target; wherein the second virtual network is implemented using a second routing instance and a second routing target; wherein a third virtual network is implemented using a third routing instance and a third routing target; and Among them, the control node is configured to configure the import and export of the first routing instance, the second routing instance, the third routing instance, and the common routing instance with the first routing target, the second routing target, the third routing target, and the common routing target to achieve a hub-and-spoke connection.

9. The network controller according to any one of claims 1 to 5, in, implementing the first virtual network using one or more first virtual routers in a first plurality of computing nodes; and The second virtual network is implemented using one or more second virtual routers in the second plurality of computing nodes.

10. The network controller according to claim 9, in, The first plurality of compute nodes is a first Kubernetes cluster; and The second plurality of computing nodes is a second Kubernetes cluster.

11. The network controller according to any one of claims 1 to 5, wherein: The control node parses the routing information into forwarding information for one or more of the first virtual network and the second virtual network, and installs the forwarding information in a virtual router supporting interconnection between the first virtual network and the second virtual network.

12. A software defined network method, comprising: Processing, by a network controller, a request to create a virtual network router, wherein the virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within a software defined network (SDN) architecture system, the virtual network router representing a logical abstraction of one or more policies causing one or more of importing and exporting routing information between the first virtual network and the second virtual network; wherein the request indicates that the virtual network router is a mesh virtual network router or a central virtual network router, the first virtual network is a first branch virtual network, and the second virtual network is a second branch virtual network; and wherein the one or more policies represented by the mesh virtual network router include symmetric import and export policies, the symmetric import and export policies causing import and export of the routing information between the first virtual network and the second virtual network; wherein the one or more policies represented by the central virtual network router include an asymmetric import and export policy, the asymmetric import and export policy causing routing information from the first branch virtual network and the second branch virtual network to be exported to the virtual network router, but not importing the routing information between the first branch virtual network and the second branch virtual network; and The network controller configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of importing and exporting the routing information between the first virtual network and the second virtual network via the virtual network router.

13. The method according to claim 12, in, The request includes tags associated with the first virtual network and the second virtual network; and Among them, processing the request includes: identifying the first virtual network and the second virtual network based on the label, configuring the routing instance corresponding to the virtual network router according to the one or more policies, so as to import and export the routing information between the first virtual network and the second virtual network.

14. The method according to any one of claims 12 and 13, in, The first virtual network is associated with a first namespace; Wherein, the second virtual network is associated with a second namespace; Wherein, the virtual network router is a first virtual network router; wherein the plurality of strategies among the one or more strategies include a first strategy and a second strategy; wherein the first virtual network router represents the first policy for importing and exporting the routing information between the first virtual network and the second virtual network router; and The second virtual network router represents the second policy for importing and exporting the routing information between the second virtual network and the first virtual network router.

15. The method according to claim 14, wherein: The first virtual network router must obtain authorization to interconnect the first virtual network router and the second virtual network router before deploying the first policy.

16. A non-transitory computer-readable medium comprising instructions for causing a processing circuit of a network controller to: Processes a request to create a virtual network router, where: The virtual network router is configured to cause the network controller to interconnect a first virtual network and a second virtual network operating within a software defined network (SDN) architecture system, the virtual network router representing a logical abstraction of one or more policies causing one or more of importing and exporting routing information between the first virtual network and the second virtual network; wherein the request indicates that the virtual network router is a mesh virtual network router or a central virtual network router, the first virtual network is a first branch virtual network, and the second virtual network is a second branch virtual network; and wherein the one or more policies represented by the mesh virtual network router include symmetric import and export policies, the symmetric import and export policies causing import and export of the routing information between the first virtual network and the second virtual network; wherein the one or more policies represented by the central virtual network router include an asymmetric import and export policy, the asymmetric import and export policy causing routing information from the first branch virtual network and the second branch virtual network to be exported to the virtual network router, but not importing the routing information between the first branch virtual network and the second branch virtual network; and The first virtual network and the second virtual network are configured according to the one or more policies to enable one or more of importing and exporting the routing information between the first virtual network and the second virtual network via the virtual network router.

Images