Computer network system and computer networking method for network service discovery

Abstract

The invention discloses a computer network system and a computer networking method for network service discovery. The computer network system includes processing circuitry accessible to a storage device, the processing circuitry configured to encode, by a network controller executing in a software defined network (SDN), one or more attributes with information identifying a network service, where the one or more attributes conform to a routing protocol. The processing circuitry is further configured to generate, by the network controller, an advertisement in a first network cluster executing within a container orchestration platform of the SDN, where the advertisement conforms to a routing protocol and includes one or more attributes. The processing circuitry is further configured to broadcast, by the network controller, an advertisement to a second network cluster executing within the container orchestration platform of the SDN according to the routing protocol.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Application No. 18 / 399,888, filed on December 29, 2023, and U.S. Provisional Application No. 63 / 506,322, filed on June 5, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to computer networks. Background Art

[0004] In a typical cloud data center environment, a large number of interconnected servers typically provide computing and / or storage capabilities to run various applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide subscriber support equipment located in various physical hosting facilities for infrastructures spread across the globe.

[0005] Cloud computing infrastructure that manages the deployment and infrastructure for application execution may involve two main roles: (1) orchestration - used to automatically deploy, scale, and operate applications across a cluster of hosts and provide computing infrastructure, which may include virtual machines (VMs) or container-centric computing infrastructure; and (2) network management - used to create virtual networks in the network infrastructure to enable communication between applications running on virtual execution environments (such as pods or VMs) and between applications running on traditional (such as physical) environments. Software-defined networking facilitates network management.

[0006] A multi-cloud environment refers to the use of multiple clouds to provide computing and storage services. An enterprise can utilize internal computing and / or storage services (e.g., an internal cloud) as well as one or more external clouds, such as clouds hosted by third-party providers. Examples of clouds include private clouds, public clouds, or hybrid public / private clouds, which allow for easy scalability while allowing for different levels of control and security. Based on the types of applications being executed and the other needs of the enterprise, an enterprise can utilize one or more of private clouds, public clouds, or hybrid public / private clouds. Summary of the invention

[0007] Techniques for advertising network service information with attributes, such as labels specifying a Border Gateway Protocol (BGP) extended community, customized BGP attributes, or other extensible routing protocol attributes, to enhance communication between multiple network clusters are disclosed. In this manner, attributes can be encoded with network service information to serve applications by enabling communication between virtual actuators from different network clusters, thereby improving network management. By encoding attributes with network service information, network administrators can directly configure virtual actuators of remote clusters as endpoints of a service without using a DNS server or service mesh.

[0008] A network cluster (also referred to herein as a "cluster") may execute a network service (also referred to herein as a "service") and notify information of the network service to a remote network cluster in the form of attributes, such as a customized BGP attribute or an extended community (e.g., a BGP community or an extended community). By notifying the remote network cluster of the network service information, endpoints within the remote network cluster may communicate with the network service without relying on the IP address of the network service, using a domain name system (DNS), or maintaining a complex service mesh network to interconnect network clusters to achieve the purpose of service delivery and consumption.

[0009] In some cases, endpoints of a remote cluster can communicate with network services of a network cluster by using a DNS server. However, the DNS server may have a long time to live (TTL) for endpoint references, which may delay the operation of the SDN when creating or removing endpoints (e.g., long before the long TTL), which may cause the DNS server to provide inaccurate information because the DNS server still keeps these endpoints active when they have been removed. Administrators of the SDN also cannot control the DNS server and provide less autonomy in customizing the SDN. Alternative approaches may include creating and managing service meshes based on proprietary protocols. However, service meshes can be complex and difficult to scale with large SDNs. The technology described herein provides a robust and lightweight technology for communication between endpoints of network services located in remote network clusters.

[0010] These technologies can provide one or more technical advantages for implementing practical applications. For example, these technologies can provide network administrators with the ability to automatically configure virtual executive elements as endpoints of network services, regardless of which cluster contains virtual executive elements and network services. These technologies can effectively and reliably process data by utilizing reliable protocols to allow the control planes of multiple network clusters to communicate directly at the IP level, rather than allowing the control planes of multiple clusters to communicate through upstream routers with a large number of next hops. The technology described herein utilizes a complete protocol to effectively add virtual executive elements residing in multiple clusters as endpoints of network services without the need for external hardware, such as DNS servers or compatible intermediate routers, and without the need to develop complex service meshes. In addition, intermediate devices (e.g., intermediate routers, route reflectors, etc.) may need to support announcements including network layer reachability information (NLRI) encoded with network service information (e.g., ports, protocols, fully qualified domain names). Regardless of whether the intermediate device supports the attribute categories described herein (e.g., BGP extended communities, customized optional transferable BGP attributes, etc.), these technologies retain network service information.

[0011] In one example, a computing device includes a processing circuit capable of accessing a storage device, the processing circuit being configured to encode one or more attributes by a network controller executing in a software defined network (SDN) with information identifying a network service, wherein the one or more attributes conform to a routing protocol. The processing circuit is further configured to generate an announcement by the network controller in a first network cluster executing in a container orchestration platform of the SDN, wherein the announcement conforms to the routing protocol and includes the one or more attributes. The processing circuit is further configured to broadcast the announcement by the network controller to a second network cluster executing in the container orchestration platform of the SDN according to the routing protocol.

[0012] In another example, a method includes encoding, by a network controller executing in a software defined network (SDN), one or more attributes with information identifying a network service, wherein the one or more attributes conform to a routing protocol. The method may also include generating, by the network controller, an announcement in a first network cluster executing in a container orchestration platform of the SDN, wherein the announcement conforms to the routing protocol and includes the one or more attributes. The method may also include broadcasting, by the network controller, the announcement to a second network cluster executing in a container orchestration platform of the SDN according to the routing protocol.

[0013] In another example, a computer-readable storage medium comprising instructions, which when executed are configured to cause a processing circuit of a network system to encode one or more attributes by a network controller executing in a software defined network (SDN) with information identifying a network service, wherein the one or more attributes conform to a routing protocol. The instructions may also cause the processing circuit to generate a notification by the network controller in a first network cluster executing in a container orchestration platform of the SDN, wherein the notification conforms to the routing protocol and includes the one or more attributes. The instructions may also cause the processing circuit to broadcast the notification by the network controller to a second network cluster executing in a container orchestration platform of the SDN according to the routing protocol.

[0014] In yet another example, a computing system includes a processing circuit capable of accessing a storage device, the processing circuit being configured to receive a notification by a network controller executed in a software defined network (SDN), wherein the notification complies with a routing protocol and includes one or more attributes having information identifying a network service. The processing circuit may also be configured to extract the information identifying the network service by processing the one or more attributes by the network controller. The processing circuit may also be configured to generate a network service directory by the network controller based on the information identifying the network service. The processing circuit may also be configured to add one or more virtual execution elements by the network controller and the network service directory as endpoints of the network service. The processing circuit may also be configured to implement a network policy to forward network traffic between endpoints of the network service by one or more virtual routers executed in the SDN.

[0015] In another example, a method includes receiving a notification by a network controller executed in a software defined network (SDN), wherein the notification complies with a routing protocol and includes one or more attributes having information identifying a network service. The method may also include extracting, by the network controller, the information identifying the network service by processing the one or more attributes. The method may also include generating, by the network controller, a network service directory based on the information identifying the network service. The method may also include adding, by the network controller and the network service directory, one or more virtual execution elements as endpoints of the network service. The method may also include implementing a network policy to forward network traffic between endpoints of the network service by one or more virtual routers executed in the SDN.

[0016] In another example, a computer-readable storage medium comprising instructions, which when executed are configured to cause a processing circuit of a network system to receive a notification by a network controller executed in a software defined network (SDN), wherein the notification complies with a routing protocol and includes one or more attributes having information identifying a network service. The instructions may also cause the processing circuit to extract information identifying the network service by processing the one or more attributes by the network controller. The instructions may also cause the processing circuit to generate a network service directory based on the information identifying the network service by the network controller. The instructions may also cause the processing circuit to add one or more virtual execution elements by the network controller and the network service directory as endpoints of the network service. The instructions may also cause the processing circuit to implement a network policy to forward network traffic between endpoints of the network service by one or more virtual routers executed in the SDN.

[0017] In another example, a computing system includes a processing circuit capable of accessing a storage device, the processing circuit being configured to encode one or more attributes with information identifying a network service by a first network controller executed in a software defined network SDN, wherein the one or more attributes conform to a routing protocol. The processing circuit may also be configured to generate an announcement by the first network controller in a first network cluster executed in a container orchestration platform of the SDN, wherein the announcement conforms to a routing protocol and includes one or more attributes. The processing circuit may also be configured to broadcast an announcement by the first network controller to a second network cluster executed in a container orchestration platform of the SDN according to a routing protocol. The processing circuit may also be configured to extract information identifying a network service by processing one or more attributes by a second network controller executed in the SDN. The processing circuit may also be configured to generate a network service directory in a second network cluster by the second network controller. The processing circuit may also be configured to add one or more virtual execution elements executed on the second network cluster as endpoints of a network service by the second network controller using the network service directory. The processing circuit may also be configured to implement a network policy to forward network traffic between endpoints of a network service by one or more virtual routers executed in the SDN.

[0018] In another example, a method includes encoding one or more attributes with information identifying a network service by a first network controller executed in a software defined network SDN, wherein the one or more attributes conform to a routing protocol. The method may also include generating an announcement by the first network controller in a first network cluster executed in a container orchestration platform of the SDN, wherein the announcement conforms to a routing protocol and includes one or more attributes. The method may also include broadcasting the announcement to a second network cluster executed in a container orchestration platform of the SDN according to the routing protocol by the first network controller. The method may also include extracting information identifying the network service by processing one or more attributes by a second network controller executed in the SDN. The method may also include generating a network service directory in a second network cluster by the second network controller. The method may also include adding one or more virtual execution elements executed on the second network cluster as endpoints of the network service by the second network controller using the network service directory. The method may also include implementing a network policy to forward network traffic between endpoints of the network service by one or more virtual routers executed in the SDN.

[0019] In another example, a computer-readable storage medium including instructions, which when executed is configured to cause a processing circuit of a network system to encode one or more attributes with information identifying a network service by a first network controller executed in a software defined network (SDN), wherein the one or more attributes conform to a routing protocol. The instructions may also cause the processing circuit to generate an announcement by the first network controller in a first network cluster executed in a container orchestration platform of the SDN, wherein the announcement conforms to the routing protocol and includes one or more attributes. The instructions may also cause the processing circuit to broadcast the announcement to a second network cluster executed in a container orchestration platform of the SDN according to the routing protocol by the first network controller. The instructions may also cause the processing circuit to extract information identifying the network service by processing one or more attributes by a second network controller executed in the SDN. The instructions may also cause the processing circuit to generate a network service directory in the second network cluster by the second network controller. The instructions may also cause the processing circuit to add one or more virtual execution elements executed on the second network cluster using the network service directory by the second network controller as endpoints of the network service. The instructions may also cause the processing circuit to include implementing a network policy to forward network traffic between endpoints of the network service by one or more virtual routers executed in the SDN.

[0020] The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these technologies will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1is a block diagram illustrating an example computing infrastructure in which examples of the techniques described herein may be implemented.

[0022] Figure 2 is a block diagram illustrating an example of a cloud-native SDN architecture for cloud-native networking according to techniques of this disclosure.

[0023] Figure 3 is a conceptual diagram illustrating an example network with multiple network clusters according to one or more aspects of the present disclosure.

[0024] Figure 4 is a block diagram illustrating an example network with multiple network clusters according to one or more aspects of the present disclosure.

[0025] Figure 5 is a block diagram illustrating an example network according to one or more aspects of the present disclosure.

[0026] Figure 6 is a conceptual diagram illustrating an example advertisement with example network layer reachability information and a set of communities in accordance with one or more aspects of the present disclosure.

[0027] Figure 7 is a flow chart illustrating an exemplary process for assigning virtual executive elements of a remote network cluster as endpoints of a network service having a routing protocol in accordance with the techniques of this disclosure.

[0028] Like reference characters refer to like elements in the drawings and the description. DETAILED DESCRIPTION

[0029] In general, the techniques described herein enable efficient and dynamic communication between virtual execution elements across multiple network clusters (also referred to herein as "clusters"). In some cases, a network service (also referred to herein as a "service") of one network cluster (e.g., AWS network services, MongoDB, etc.) (e.g., methods that expose a network application running as one or more virtual execution elements) can communicate with virtual execution elements (e.g., pods or VMs) of a remote cluster to easily manage the utilization of resources used by the cluster, thereby maximizing efficiency and reducing execution costs. DNS servers are typically used, but require additional orchestration and do not allow users to control the time-to-live configuration.

[0030] The technology described herein integrates network service information in advertisements broadcast between multiple clusters. These advertisements may conform to a network routing protocol (e.g., a border gateway routing protocol), which is typically used to announce routes (and other routing specific information) in or between networks. Given that network routing protocols have been used in networks and have undergone extensive testing and troubleshooting to provide consistent operation in a variety of different network topologies, while also having well-defined software and / or hardware implementation suites, network routing protocols can provide lightweight and efficient (e.g., in terms of computing utilization) advertisements of service information between network clusters.

[0031] Figure 1 is a block diagram of an exemplary computing infrastructure 8 showing an example of the techniques described herein that can be implemented. Current implementations of software defined network (SDN) architectures for virtual networks present challenges for cloud native adoption, for example, due to the complexity of lifecycle management, mandatory high resource analysis components, scale limitations of configuration modules, and no command line interface (CLI)-based interfaces (similar to kubectl). The computing infrastructure 8 includes a cloud native SDN architecture system described herein that addresses these challenges and is modernized for the telecom cloud native era. Example use cases for cloud native SDN architectures include 5G mobile networks and cloud and enterprise cloud native use cases. The SDN architecture may include data plane elements implemented in computing nodes (e.g., servers 12) and network devices such as routers or switches, and the SDN architecture may also include an SDN controller (e.g., network controller 24) for creating and managing virtual networks. The SDN architecture configuration and control plane are designed to scale out cloud native software with a container-based microservice architecture that supports in-service upgrades.

[0032] Therefore, SDN architecture components are microservices, and compared with existing network controllers, SDN architecture uses a basic container orchestration platform to manage the life cycle of SDN architecture components. Use a container orchestration platform to call out SDN architecture components; SDN architecture uses cloud native monitoring tools, which can be integrated with cloud native options provided by customers; SDN architecture provides resource declaration methods for SDN architecture objects (i.e., custom resources) using aggregated APIs. SDN architecture upgrades can follow cloud native patterns, and SDN architecture can leverage Kubernetes constructs such as Multus, authentication and authorization, cluster APIs, Kube Alliance, KubeVirt, and Kata containers. SDN architecture can support Data Plane Development Kit (DPDK) pods, and SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies.

[0033] For service providers and enterprises, the SDN architecture automates network resource provisioning and orchestration to dynamically create highly scalable virtual networks and link virtualized network functions (VNFs) and physical network functions (PNFs) to form differentiated service chains on demand. The SDN architecture can be integrated with orchestration platforms such as Kubernetes, OpenShift, Mesos, OpenStack, VMware vSphere (e.g., orchestrator 23), and can also be integrated with service provider operations support systems / traffic support systems (OSS / BSS).

[0034] Typically, one or more data centers 10 provide an operating environment for applications and services at a customer site 11 (illustrated as "Customer 11"), which has one or more customer networks coupled to the data center through a service provider network 7. For example, each data center 10 may host infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 15, which may represent one or more networks managed by other providers and may therefore form part of a large-scale public network infrastructure (e.g., the Internet). The public network 15 may represent, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0035] Although customer sites 11 and public networks 15 are primarily illustrated and described as edge networks of service provider network 7, in some examples, one or more customer sites 11 and public networks 15 may be tenant networks within any one of data centers 10. For example, data center 10 may host multiple tenants (customers), each tenant being associated with one or more virtual private networks (VPNs), each of which may implement one customer site 11.

[0036] The service provider network 7 provides packet-based connectivity to connected customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement multi-protocol label switching (MPLS) forwarding, and in this case may be referred to as an MPLS network or MPLS backbone. In some cases, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0037] In some examples, each of the data centers 10 may represent one of many geographically distributed network data centers, which may be connected to each other via a service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1 As shown in the example of , data center 10 may include facilities for providing network services to customers. Customers of a service provider may be collective entities, such as businesses, governments, or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although illustrated as an independent edge network of a service provider network 7, elements of data center 10 (e.g., one or more physical network functions (PNFs) or virtualized network functions (VNFs)) may be included within the kernel of service provider network 7.

[0038] In this example, data center 10 includes storage and / or computing servers (or "nodes") interconnected via a switch fabric 14 provided by one or more layers of physical network switches and routers, with servers 12A-12X (herein "servers 12") being depicted as coupled to top-of-rack switches 16A-16N. Servers 12 are computing devices and may also be referred to herein as "computing nodes," "hosts," or "host devices." Although in Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10 may include many additional servers coupled to other TOR switches 16 in data center 10.

[0039] The switch fabric 14 in the illustrated example includes interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") that are coupled to a distribution layer of chassis (or "spine" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices such as firewalls, intrusion detection and / or intrusion prevention devices, servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as, for example, physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0040] In this example, the TOR switch 16 and the chassis switch 18 provide redundant (multi-homed) connections to the IP structure 20 and the service provider network 7 for the server 12. The chassis switch 18 aggregates traffic flows and provides connections between the TOR switches 16. The TOR switch 16 can be a network device that provides Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functions. The TOR switch 16 and the chassis switch 18 can each include a memory and one or more processors, and can execute one or more software processes. The chassis switch 18 is coupled to the IP structure 20, which can perform Layer 3 routing to route network traffic between the data center 10 and the customer site 11 through the service provider network 7. The switching architecture of the data center 10 is only an example. For example, other switching architectures can have more or fewer switching layers. The IP structure 20 can include one or more gateway routers.

[0041] The term "packet flow", "traffic flow", or simply "flow" refers to a group of packets originating from a specific source device or endpoint and sent to a specific destination device or endpoint. A single packet flow can be identified by a 5-tuple: for example, <source network address, destination network address, source port, destination port, protocol>. This 5-tuple generally identifies the packet flow to which the received packets correspond. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for the packet.

[0042] Servers 12 may each represent a computing server or a storage server. For example, each server 12 may represent a computing device, such as an x86 processor-based server, which is configured to operate according to the techniques described herein. Servers 12 may provide a network function virtualization infrastructure (NFVI) for the NFV architecture.

[0043] Any of the servers 12 may configure a virtual execution element (e.g., a pod or virtual machine) by virtualizing the server's resources to provide some measure of isolation between one or more processes (applications) executing on the server. "Hypervisor-based" or "hardware-level" or "platform" virtualization refers to the creation of virtual machines, each of which includes a client operating system for executing one or more processes. Typically, a virtual machine provides a virtualization / client operating system for executing applications in an isolated virtual environment. Because the virtual machines are virtualized from the physical hardware of the host server, the executed applications are isolated from both the host's hardware and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0044] A virtual network is a logical structure implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., one of data centers 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly allowed by security policy.

[0045] Virtual networks can use the data center 10 gateway router ( Figure 1 The virtual network can be connected to and extended over physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and Ethernet virtual private network (EVPN) networks (not shown). The virtual network can also be used to implement network function virtualization (NFV) and service chaining.

[0046] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks, namely a physical underlying network consisting of an IP structure 20 and a switching structure 14 and a virtual overlay network. The role of the physical underlying network is to provide an "IP structure" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0047] As further described below with respect to virtual router 21 (illustrated and also referred to herein as "vRouter 21"), the virtual routers running in server 12 create a virtual overlay network on top of the physical underlying network using a mesh of dynamic "tunnels" between them. These overlay tunnels may be, for example, MPLS tunnels over GRE / UDP or VXLAN tunnels, or NVGRE tunnels. The underlying physical routers and switches may not store any per-tenant state for the virtual machines or other virtual executives, such as any media access control (MAC) addresses, IP addresses, or policies. The forwarding tables of the underlying physical routers and switches may, for example, contain only the IP prefix or MAC address of the physical server 12. (A gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0048] The virtual router 21 of the server 12 typically contains per-tenant state. For example, a separate forwarding table (routing instance) may be included for each virtual network. The forwarding table contains the IP prefixes (in the case of a layer 3 overlay) or MAC addresses (in the case of a layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., container pods). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that exist locally on the server 12 (i.e., there is at least one virtual execution element on the server 12).

[0049] “Container-based” or “operating system” virtualization refers to virtualizing an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers, such as those provided by the open source Docker container application or CoreOS Rkt (“Rocket”). Like a virtual machine, each container is virtualized and can remain isolated from the host and other containers. However, unlike a virtual machine, each container can omit a separate operating system and instead provide a suite of applications and application-specific libraries. Typically, a container is executed by a host as an isolated userspace instance and can share an operating system and common libraries with other containers executing on the host. Therefore, containers may require less processing power, storage, and network resources than virtual machines (“VMs”). A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks.

[0050] In some examples, containers are managed by their host kernel to allow limiting and prioritizing resources (CPU, memory, block I / O, network, etc.) without the need to launch any virtual machines, in some cases using namespace isolation capabilities that allow complete isolation of an application's (e.g., a given container's) view of an operating environment, including process trees, networks, user identifiers, and mounted file systems. In some examples, containers can be deployed in accordance with Linux Containers (LXC), an operating system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

[0051] Server 12 hosts virtual network endpoints for one or more virtual networks that operate on a physical network represented herein by IP fabric 20 and switch fabric 14. Although primarily described with respect to a data center based switched network, other physical networks such as service provider network 7 may serve as the basis for one or more virtual networks.

[0052] Each server 12 can host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a set of one or more containers (e.g., pods), or another virtual execution element, such as a layer 3 endpoint of a virtual network. The term "virtual execution element" includes virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" may also include a pod of one or more containers. A virtual execution element may represent an application workload. As Figure 1 As shown, server 12A hosts a virtual network endpoint in the form of a Pod 22 having one or more containers. However, given the hardware resource limitations of server 12, server 12 can execute as many virtual execution elements as possible. Each virtual network endpoint can use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint can use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0053] Each server 12 includes at least one network interface card (NIC) 13, each NIC including at least one interface for exchanging packets with a TOR switch 16 via a communications link. For example, server 12A includes NIC 13A. Any NIC 13 can provide one or more virtual hardware components 21 for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) described in the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to present one or more virtual network interfaces as "virtual functions" for use by various endpoints executing on the server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are examples of virtual hardware components 21.

[0054] As another example, one or more servers 12 may implement Virtio, which is a paravirtualization framework that can be used, for example, for a Linux operating system, which provides emulated NIC functionality as a virtual hardware component to provide a virtual network interface to a virtual network endpoint. As another example, one or more servers 12 may implement an open vSwitch to perform distributed virtual multi-layer switching between one or more virtual NICs (vNICs) for a hosted virtual machine, wherein such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some cases, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some cases, the virtual hardware component is an SR-IOV virtual function. In some examples, any server in the server 12 may implement a Linux bridge that simulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For a Docker implementation of a container hosted by a server, a Linux bridge or other operating system bridge that executes on the server and exchanges packets between containers may be referred to as a "Docker bridge." The term “virtual router” as used herein may include a Contrail or Tungsten Fabric virtual router, an Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other device and / or software that resides on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12.

[0055] Any NIC 13 may include an internal device switch to exchange data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch may be a virtual Ethernet bridge (VEB) to switch between SR-IOV virtual functions and, accordingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a client operating system. The internal device switch may also be referred to as a NIC switch, and for SR-IOV implementations, may also be referred to as an SR-IOV NIC switch. The virtual hardware component associated with the NIC 13A may be associated with a Layer 2 destination address, which may be assigned by the NIC 13A or a software process responsible for configuring the NIC 13A. Physical hardware components (or "physical functions" for SR-IOV implementations) are also associated with a Layer 2 destination address.

[0056] One or more servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide a virtual network interface and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) for an Internet Protocol virtual private network (IP-VPN). For example, a packet received by the virtual router 21 of the server 12A from the underlying physical network structure of the data center 10 (i.e., the IP structure 20 and the switching structure 14) may include an outer header to allow the physical network structure to tunnel the payload or "inner packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the network interface card 13A of the server, but also a virtual network identifier, such as a VxLAN label or a multi-protocol label switching (MPLS) label that identifies a virtual network and a corresponding routing instance executed by the virtual router 21. The inner packet includes an inner header with a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0057] The virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the tunnel encapsulation header of the packet, and forwards the packet to the appropriate destination virtual network endpoint of the packet. For example, for server 12A, for each packet outbound from the virtual network endpoint (e.g., Pod 22A) hosted by server 12A, virtual router 21A appends a tunnel encapsulation header indicating the virtual network of the packet to generate an encapsulated or "tunnel" packet, and virtual router 21A outputs the encapsulated packet to a physical destination computing device, such as another server 12, via the overlay tunnel of the virtual network. As used herein, the virtual router 21 can perform the operation of a tunnel endpoint to encapsulate an internal packet originating from a virtual network endpoint to generate a tunnel packet, and decapsulate the tunnel packet to obtain an internal packet for routing to other virtual network endpoints.

[0058] In some examples, virtual router 21 may be kernel-based and executed as part of the kernel of the operating system of server 12A.

[0059] In some examples, the virtual router 21 can be a virtual router that supports the Data Plane Development Kit (DPDK). In such an example, the virtual router 21 uses DPDK as a data plane. In this mode, the virtual router 21 runs as a user space application linked to the DPDK library (not shown). This is a performance version of the virtual router and is typically used by telecommunications companies, where VNFs are typically DPDK-based applications. The performance of the virtual router 21 as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the polling mode driver (PMDs) of the DPDK instead of the interrupt-based driver of the Linux kernel.

[0060] A User I / O (UIO) kernel module, such as vfio or uio_pci_generic, can be used to expose the registers of the physical network interface into user space so that they can be accessed by the DPDK PMD. When the NIC 13A is bound to the UIO driver, it is moved from the Linux kernel space to the user space and is therefore no longer managed or visible by the Linux operating system. Therefore, it is the DPDK application (i.e., virtual router 21A in this example) that fully manages the NIC 13. This includes packet polling, packet processing, and packet forwarding. The user packet processing steps can be performed by the virtual router 21DPDK data plane, where the kernel (where the kernel is not in Figure 1 There is limited or no involvement of the DPDK vRouter (shown in ). This "polling mode" nature makes the virtual router 21DPDK data plane packet processing / forwarding more efficient than interrupt mode, especially when the packet rate is high. There are limited or no interrupts and context switches during packet I / O. For more details on the DPDK vRouter example, see "DAY ONE: CONTRAIL DPDK v ROUTER", 2021, Kiran KN et al., Juniper Networks, Inc., the entire contents of which are incorporated herein by reference.

[0061] The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration system that provides a container-centric infrastructure for automating the deployment, scaling, and operation of containers to provide a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provisioning, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements to host servers available to the orchestration platform. Container orchestration can facilitate container coordination and refers to the deployment, management, scaling, and configuration of containers to host servers through a container orchestration platform. Examples of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0062] The elements of the automation platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. A cluster-based framework can be used to deploy containers to a virtualized environment, wherein a cluster master node of a network cluster manages the deployment and operation of containers to one or more cluster child nodes of the network cluster. The terms "master node" and "child node" used herein include different orchestration platform terms for distinguishing between the primary management element of a network cluster and similar devices of the primary container hosting device of the network cluster. For example, the Kubernetes platform uses the terms "cluster master node" and "child node", while the Docker Swarm platform refers to the cluster manager and cluster nodes.

[0063] Orchestrator 23 and network controller 24 may be executed on separate computing devices, or on the same computing device. Each of orchestrator 23 and network controller 24 may be a distributed application executed on one or more computing devices. Orchestrator 23 and network controller 24 may implement a corresponding master node for one or more clusters, each cluster having one or more child nodes (also referred to as "computing nodes") implemented by a corresponding server 12.

[0064] In general, network controller 24 controls the network configuration of the data center 10 structure, for example, to establish one or more virtual networks for packetized communications between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator. More information regarding example operations of the network controller 24 operating in conjunction with other devices in the data center 10 or other software defined networks may be found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET Flows,” and in U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, and entitled “TUNNELED PACKET AGGREGATION FOR VIRTUAL NETWORKS,” each of which is incorporated herein by reference as if fully set forth herein.

[0065] exist Figure 1 In the example of , the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides a computing infrastructure, which may include a container-centric computing infrastructure. The orchestrator 23 and, in some cases, the network controller 24 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualized infrastructure for the container management platform. Figure 3 Describe example components of the Kubernetes orchestration system.

[0066] In one example, Pod 22 is a Kubernetes pod and is an example of a virtual network endpoint. A pod is one or more logically related containers ( Figure 1 pods are groups of containers (not shown), shared storage for containers, and options for how to run containers. In the case of instantiation for execution, a pod may alternatively be referred to as a "pod replica." Each container of a pod 22 is an example of a virtual execution element. The containers of a pod are always located on a single server, co-scheduled, and run in a shared context. The shared context of a pod may be a set of Linux namespaces, cgroups, and other isolation aspects.

[0067] Within the context of a pod, a single application may have further sub-isolations applied. Typically, containers in a pod have a common IP address and port space and are able to detect each other through localhost. Because they have a shared context, containers in a pod can also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different pods have different IP addresses and cannot communicate through IPC without a configuration to enable this functionality. Instead, containers that are members of different pods typically communicate with each other through the pod IP address.

[0068] Server 12A includes container platform 19 for running containerized applications, such as applications of Pod 22. Container platform 19 receives a request from orchestrator 23 to obtain a container and hosts the container in server 12A. Container platform 19 obtains and executes the container.

[0069] The container network interface (CNI) 17 configures a virtual network interface for a virtual network endpoint. The orchestrator 23 and the container platform 19 use the CNI 17 to manage the networking of pods, including the Pod 22. For example, the CNI 17 creates a virtual network interface to connect the pod to the virtual router 21, and enables the containers of such pods to communicate with other virtual network endpoints on the virtual network via the virtual network interface. For example, the CNI 17 may insert a virtual network interface for a virtual network into the network namespace for the container in the Pod 22, and configure (or request configuration) a virtual network interface for the virtual network in the virtual router 21, so that the virtual router 21 is configured to send packets received from the virtual network via the virtual network interface to the container of the Pod 22, and to send packets received from the container of the Pod 22 on the virtual network via the virtual network interface to the virtual network. The CNI 17 may allocate a network address (e.g., a virtual IP address of the virtual network) and may establish a route for the virtual network interface.

[0070] In Kubernetes, by default, all pods can communicate with all other pods without using network address translation (NAT). In some cases, the orchestrator 23 and the network controller 24 create a service virtual network and a pod virtual network shared by all namespaces, and allocate service and pod network addresses from these namespaces, respectively. In some cases, all pods in all namespaces generated in the Kubernetes cluster can communicate with each other, and the network addresses of all pods can be allocated from the pod subnet specified by the orchestrator 23. When a user creates an isolated namespace for a pod, the orchestrator 23 and the network controller 24 can create a new pod virtual network and a new shared service virtual network for the new isolated namespace. The pods in the isolated namespace generated in the Kubernetes cluster extract network addresses from the new pod virtual network, and the corresponding services of these pods extract network addresses from the new service virtual network.

[0071] CNI 17 may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17 may at least partially conform to the container network interface (CNI) specification or the rkt networking solution. CNI 17 may represent Contrail, Open Contrail, Multus, Calico, cRPD, or other CNI. Alternatively, CNI 17 may be referred to as a network plugin or CNI plugin or CNI instance. For example, Multus CNI may call a separate CNI to establish different virtual network interfaces for Pod 22.

[0072] The CNI 17 may be invoked by an orchestrator 23. For the purposes of the CNI specification, a container may be considered a synonym for a Linux network namespace. Which unit this corresponds to depends on the specific container runtime implementation: for example, in an implementation of the application container specification such as rkt, each pod runs in a unique network namespace. However, in Docker, a network namespace typically exists within each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and can communicate with each other. This can be a single container, a machine / server (real or virtual), or other network device such as a router. Containers can be conceptually added to or removed from one or more networks. The CNI specification specifies a number of considerations for conformance plugins ("CNI plugins").

[0073] Pod 22 includes one or more containers. In some examples, Pod 22 includes a containerized DPDK workload that is designed to use DPDK to accelerate packet processing, for example, by exchanging data with other components using DPDK libraries. In some examples, virtual router 21 can be executed as a containerized DPDK workload.

[0074] Pod 22 is configured with a virtual network interface 26 for sending and receiving packets through virtual router 21. Virtual network interface 26 may be a default interface for Pod 22. Pod 22 may implement virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while virtual router 21 may implement virtual network interface 26 as a tap interface, a virtual user interface, or other types of interfaces.

[0075] Pod 22 and virtual router 21 use virtual network interface 26 to exchange data packets. Virtual network interface 26 can be a DPDK interface. Pod 22 and virtual router 21 can use vhost to establish virtual network interface 26. Pod 22 can operate according to the aggregation model. Pod 22 can use virtual devices, such as virtio devices with vhost user adapters, for user space container process communication of virtual network interface 26.

[0076] CNI 17 can be combined with Figure 1 One or more other components shown configure a virtual network interface 26 for the Pod 22. Any container of the Pod 22 can utilize (ie, share) the virtual network interface 26 of the Pod 22.

[0077] The virtual network interface 26 may represent a virtual Ethernet ("veth") pair, where each end of the pair is a separate device (e.g., a Linux / Unix device), one end of the pair is assigned to the Pod 22, and one end of the pair is assigned to the virtual router 21. A veth pair or one end of a veth pair is sometimes referred to as a "port". The virtual network interface may represent a macvlan network with media access control (MAC) addresses assigned to the Pod 22 and the virtual router 21 for communication between the containers of the Pod 22 and the virtual router 21. For example, the virtual network interface may alternatively be referred to as a virtual machine interface (VMI), a pod interface, a container network interface, a tap interface, a veth interface, or simply a network interface (in a particular context).

[0078] exist Figure 1In the example server 12A of FIG. 1 , Pod 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for an application deployment that specifies a virtual network and specifies that Pod 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. Orchestrator 23 may receive the configuration data from, for example, a user, an operator / administrator, or other computing system.

[0079] As part of the process of creating a Pod 22, orchestrator 23 requests network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). Pod 22 may have a different virtual network interface for each virtual network to which it belongs. For example, virtual network interface 26 may be a virtual network interface for a particular virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks.

[0080] The network controller 24 processes the request to generate interface configuration data for the virtual network interface of the Pod 22. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0081] Network controller 24 sends the interface configuration data to server 12A, and more specifically, in some cases, to virtual router 21. To configure the virtual network interface for Pod 22, orchestrator 23 may call CNI 17. CNI 17 obtains the interface configuration data from virtual router 21 and processes it. CNI 17 creates each virtual network interface specified in the interface configuration data. For example, CNI 17 may connect one end of a veth pair that implements management interface 26 to virtual router 21, and may connect the other end of the same veth pair to Pod 22, which may implement it using a virtual user.

[0082] The following is example interface configuration data for Pod 22 for virtual network interface 26 .

[0083] [{

[0084] / / Virtual network interface 26

[0085] "id":"fe4bab62-a716-11e8-abd5-0cc47a698428",

[0086] "instance-id":"fe3edca5-a716-11e8-822c-0cc47a698428",

[0087] "ip-address":"10.47.255.250",

[0088] "plen":12,

[0089] "vn-id":"56dda39c-5e99-4a28-855e-6ce378982888",

[0090] "vm-project-id":"00000000-0000-0000-0000-000000000000",

[0091] "mac-address":"02:fe:4b:ab:62:a7",

[0092] "system-name":"tapeth0fe3edca",

[0093] "rx-vlan-id":65535,

[0094] "tx-vlan-id":65535,

[0095] "vhostuser-mode":0,

[0096] “v6-ip-address”:“::“,

[0097] “v6-plen”:,

[0098] “v6-dns-server”:“::”,

[0099] “v6-gateway”:“::”,

[0100] "dns-server":"10.47.255.253",

[0101] "gateway":"10.47.255.254",

[0102] "author":" / usr / bin / contrail-vrouter-agent",

[0103] "time":"426404:56:19.863169"

[0104] }]

[0105] A traditional CNI plugin is called by a container platform / runtime, receives an Add command from the container platform to add a container to a single virtual network, and such a plugin may subsequently be called to receive a Del(ete) command from the container / runtime and remove the container from the virtual network. The term "call" may refer to instantiating a software component or module in memory into executable code for execution by a processing circuit.

[0106] The network controller 24 is a cloud-native distributed network controller for software-defined networking (SDN), which is implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself can be implemented using one or more cloud-native component microservices. Each control node 32 itself can be implemented using one or more cloud-native component microservices.

[0107] In some examples, configuration node 30 may be implemented by extending a native orchestration platform to support customized resources for an orchestration platform for software-defined networking, and more specifically, by providing a northbound interface to the orchestration platform by, for example, configuring virtual network interfaces for virtual execution elements, configuring an underlying network connecting server 12, and configuring overlay routing functions including overlay tunnels for virtual networks and overlay trees for multicast layer 2 and layer 3 to support intent-driven / declarative creation and management of virtual networks.

[0108] As Figure 1 As part of the SDN architecture shown, the network controller 24 can be multi-tenant aware and support multi-tenancy of the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined networking constructs and advanced networking features such as virtual networking, BGPaaS, networking policies, service chaining, and other telecommunications features. The network controller 24 can support network isolation using virtual network constructs and support layer 3 networking.

[0109] To interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual router 21) import and export policies defined using virtual network router (VNR) resources. Virtual network router resources can be used to define connectivity between virtual networks by configuring the import and export of routing information between various routing instances used to implement virtual networks in the SDN architecture. A single network controller 24 can support multiple Kubernetes clusters, and VNR therefore allows connecting multiple virtual networks in a namespace, virtual networks in different namespaces, Kubernetes clusters, and across Kubernetes clusters. VNR can also be extended to support virtual network connections across multiple instances of the network controller 24. Optionally, VNR may be referred to herein as a virtual network policy (VNP) or a virtual network topology.

[0110] like Figure 1 As shown in the example of , the network controller 24 can maintain configuration data (e.g., configuration 30) representing virtual networks (VNs) 50A-50N ("VN50") (which represent policies and other configuration data for establishing VN 50 within the data center 10 through physical underlying networks and / or virtual routers (e.g., virtual router 21) ("vRouter21")). The network controller 24 can also maintain configuration data (e.g., configuration 30) representing virtual network routers (VNRs) 52A-52N ("VNRs 52") (which can be implemented at least in part using policies and other configuration data for establishing interconnectivity between VNs 50).

[0111] A user, such as an administrator, can interact with the UI 60 of the network controller 24 to define VNs 50 and VNRs 52. In some cases, the UI 60 represents a graphical user interface (GUI) that facilitates input of configuration data that defines the VNs 50 and VNRs 52. In other instances, the UI 60 can represent a command line interface (CLI) or other types of interfaces. Assuming that the UI 60 represents a graphical user interface, the administrator can define the VN 50 by arranging graphical elements representing different pods (e.g., Pods 22) to associate the pods with the VNs 50, wherein any VN 50 enables communication between one or more pods assigned to the VN.

[0112] In this regard, an administrator may understand Kubernetes or other orchestration platforms, but not fully understand the underlying infrastructure supporting VN 50. Some controller architectures, such as Contrail, can configure VN 50 based on network protocols that are similar (if not substantially similar) to routing protocols in traditional physical networks. For example, Contrail can leverage the concepts of the Border Gateway Protocol (BGP), which is a routing protocol used to communicate routing information within so-called autonomous systems (ASEs) and sometimes between ASEs.

[0113] There are different versions of BGP, such as internal BGP (iBGP) for transmitting routing information within an ASE, and external BGP (eBGP) for transmitting routing information between ASEs. Each version of BGP can also include multi-protocol BGP (MP-BGP), such as MP-eBGP or MP-iBGP. ASes may be related to the concept of projects in Contrail, which is also similar to namespaces in Kubernetes. In each instance of AS, project, and namespace, an AS (similar to projects and namespaces) can represent a collection of one or more networks (e.g., one or more of VN 50) that can share routing information, thereby facilitating interconnection between networks (or in this instance, VN 50).

[0114] Administrators may also use one or more optional BGP communities when implementing various versions of BGP. A BGP community may be a group of destinations that share common attributes. A BGP community may be a transitive BGP attribute that is sent to and interpreted by a BGP peer. A BGP community may be represented as a label attached to a BGP route advertised between multiple BGP peers. A BGP community may include information about path attributes in a BGP update message (e.g., an announcement). A BGP community may include information that may identify community members and enable administrators to perform operations on a group of community members without having to describe each community member in detail. BGP communities may be used to trigger routing decisions, such as accept, reject, prefer, or redistribute. BGP communities may also take various forms, such as a BGP extended community. A BGP extended community may be a BGP community that allows flexibility for large software-defined networks.

[0115] In its simplest form, a VNR 52 represents a logical abstraction of a set of routers in the context of Kubernetes, where a VNR 52 may be defined as a custom resource to facilitate interconnection between VNs 50. Given that Kubernetes administrators may not fully understand the complex routing information propagation according to complex routing protocols such as BGP, various aspects of cloud native network technology may help abstract the underlying routing protocol (or complementary processes of Contrail or other controller architectures) into a VNR 52.

[0116] That is, rather than resorting to defining how routing occurs between two or more VNs 50, an administrator can define one or more VNRs 52 to interconnect the VNs 50 without having to manually develop and deploy extensive policies and / or routing instance configurations to enable the exchange of routing information between such VNs 50. Instead, an administrator (who may have little knowledge of routing protocols) can define custom resources (e.g., one or more of the VNRs 52) using familiar Kubernetes syntax / semantics (or even simply by dragging graphical elements and specifying the interconnections between, for example, a graphical element representing VNR 52A and, again, for example, graphical elements representing VNs 50A and 50N).

[0117] In this regard, administrators can use Figure 1 The logical abstraction shown in the example as VNR50 is used to easily interconnect VN 50, so that the network controller 24 can convert VNR50 into an underlying routing target to automatically (meaning little or possibly no human intervention) cause routing information of VN 50A and 50N to be exchanged and enable communication (meaning the exchange of packets or other data) between VN 50A and 50N.

[0118] Assuming that administrators can configure VNR50 using familiar Kubernetes syntax / semantics instead of configuring complex configuration data that conforms to routing protocol syntax / semantics, the network controller 24 can facilitate a better user experience while also facilitating more efficient operation of the data center 8 itself. That is, having administrators enter configuration data that is unfamiliar to these administrators may result in misconfigurations that waste the underlying resources of the data center 8 (in terms of processing cycles, memory, bus bandwidth, etc. and associated power), while also delaying the correct implementation of the network topology (which may prevent packets and other data from being successfully routed between VN 50). This delay can frustrate not only the administrator, but also the customers associated with the VN 50, who may require the rapid operation of the VN 50 to achieve traffic goals. By enabling administrators to easily facilitate communication between VN 50 using the logical abstraction shown as VNR50, the data center 8 itself can experience more efficient operation (in terms of the above-mentioned computing resources including processor cycles, memory, bus bandwidth, and associated power), while providing a better user experience for administrators and customers.

[0119] The network controller 24 (representing the SDN architecture system of the data center 10) includes processing circuits that implement configuration nodes and control nodes. The network controller 24 can be configured to interconnect a first virtual network (e.g., VN 50A) and a second virtual network (e.g., VN 50N) operating within the SDN architecture system represented by the data center 10. The network controller 24 can be configured to define a logical abstraction of one or more policies to perform such interconnection via one or more VNRs 52 (e.g., VNR 52A).

[0120] Policies may include import and export policies regarding routing information maintained by virtual networks (in this example, virtual networks may refer to VN 50A and 50N). That is, Kubernetes may be extended via a custom resource representing VNR 52A to convert VNR 52A into one or more import and export policies deployed with respect to VN 50A and VN 50N, thereby configuring intercommunication via routing information distribution between VN 50A and VN 50N. Once configured, VN 50A may export routing information (e.g., routing representing VN 50A) to VN 50N, and import routing information (e.g., routing representing VN 50N) to VN 50A. Similarly, VN 50N may export routing information (e.g., routing representing VN 50N) to VN 50A, and import routing information (e.g., routing representing VN 50A) to VN 50N.

[0121] The abstraction may hide the underlying routing configuration to implement such routing leakage, such as defining routing information import and export to routing targets for routing instances that implement VN 50A and VN 50N. Alternatively, network controller 24 may convert VNR 52A to a common routing target and configure communication of routing information via the common routing target for routing instances that implement VN 50A and VN 50N (in this example).

[0122] To achieve mesh connectivity, the network controller 24 may configure the import and export of routing instances of VN 50A, VN 50N, and VNR 52A using routing targets associated with VN 50A, VN 50N, and VNR 52A. To achieve hub-and-spoke connectivity, the network controller 24 may configure the export of routing instances associated with VN 50A and VN 50N to export routing information to routing instances associated with VNR 52A (acting as a hub), and configure the routing instances of VNR 52A to import routing information to routing instances associated with VN 50A and VN 50N. In such hub-and-spoke connectivity, VN 50A and VN 50N may not communicate directly with each other.

[0123] In addition, the network controller 24 can use network policies to implement multi-layer security. The default behavior of Kubernetes is to let pods communicate with each other. To apply network security policies, the SDN architecture implemented by the network controller 24 and the virtual router 21 can run as the CNI of Kubernetes through the CNI 17. For Layer 3, isolation occurs at the network level and virtual networks run at Layer 3. Virtual networks are connected through policies. Kubernetes native network policies provide security at Layer 4. The SDN architecture may support Kubernetes network policies. Kubernetes network policies operate on Kubernetes namespace boundaries. The SDN architecture can add customized resources for enhanced network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta-tags to apply granular security policies in an extensible manner.) For Layer 4+, in some examples, the SDN architecture can support integration with containerized security devices and / or Istio, and can provide encryption support.

[0124] As Figure 1As part of the SDN architecture shown, the network controller 24 can support multi-cluster deployments, which is important for telco cloud and high-end enterprise use cases. For example, the SDN architecture can support multiple Kubernetes clusters. The cluster API can be used to support lifecycle management of Kubernetes clusters. KubefedV2 can be used to configure node 30 federation across Kubernetes clusters. The cluster API and KubefedV2 are optional components to support a single instance of the network controller 24 that supports multiple Kubernetes clusters.

[0125] SDN architectures can provide insights into infrastructure, clusters, and applications using web user interfaces and telemetry components. Telemetry nodes can be cloud-native and include microservices to support insights.

[0126] As a result of the above features and other features to be described elsewhere in this document, the computing infrastructure 8 implements a cloud-native SDN architecture and can present one or more of the following technical advantages. For example, the network controller 24 is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrades of the various component microservices of the configuration node 30 and the control node 32 (as well as any other components of other examples of the network controller described in this disclosure). These technologies can further enable optional cloud-native monitoring (telemetry) and user interfaces, high-performance data planes for containers using DPDK-based virtual routers connected to DPDK-enabled pods, and cloud-native configuration management that leverages the configuration framework of existing orchestration platforms (such as Kubernetes or Openstack) in some cases. As a cloud-native architecture, the network controller 24 is an extensible and elastic architecture to address and support multiple clusters. In some cases, the network controller 24 can also support the scalability and performance requirements of key performance indicators (KPIs).

[0127] An SDN architecture having the features and technical advantages as described herein can be used to implement a cloud-native telecom cloud to support, for example, 5G mobile networks (and subsequent generations) and edge computing, as well as an enterprise Kubernetes platform including, for example, high-performance cloud-native application hosting. Telecom cloud applications are rapidly moving towards a containerized, cloud-native approach. 5G fixed and mobile networks are driving the need to deploy workloads as microservices with significant decomposition, particularly in the 5G Next Generation RAN (5GNR). The 5G Next Generation Core (5GNC) is likely to be deployed as a set of microservice-based applications, corresponding to each of the different components described by 3GPP. When viewed as a group of microservices delivering applications, the 5GNC is likely to be a highly complex combination of pods with complex network, security, and policy requirements. The cloud-native SDN architecture described herein has well-defined network, security, and policy structures that can be used for this use case. The network controller 24 can provide relevant APIs that can create these complex structures.

[0128] Similarly, the user plane function (UPF) in the 5GNC will be an ultra-high-performance application. It can be delivered as a set of highly distributed, high-performance pods. The SDN architecture described in this article is capable of providing a very high throughput data plane (in terms of bits per node (bps) and packets per second (pps)). Integration with the DPDK virtual router, eBPF, and SmartNIC with the latest performance enhancements will help achieve the required throughput. The DPDK-based virtual router is described in more detail in U.S. application No. 17 / 649,632, entitled “CONTAINERIZED ROUTER WITH VIRTUALNETWORKING,” filed on February 1, 2022, the entire contents of which are incorporated herein by reference.

[0129] High performance processing may also be relevant to GiLAN as workloads migrate from more traditional virtualized workloads to containerized microservices. In the UPF and data plane of GiLAN services, such as GiLAN firewall, intrusion detection and prevention, virtualized IP Multimedia Subsystem (vIMS) voice / video, etc., throughput will be high and sustained in both bps and pps. For the control plane of 5GNC functions, such as access and mobility management function (AMF), session management function (SMF), etc., and for some GiLAN services (such as IMS), while the absolute traffic in terms of bps may be modest, the advantage of small packets means that pps will remain high. In some examples, as implemented on server 12, the SDN controller and data plane provide millions of packets per second to each virtual router 21. In 5G radio access network (RAN), in order to get rid of the proprietary vertically integrated RAN stack provided by traditional radio vendors, Open RAN separates RAN hardware and software in many components, including non-RT radio intelligent controller (RIC), near real-time RIC, centralized unit (CU) control plane and user plane (CU-CP and CU-UP), distributed unit (DU) and radio unit (RU). The software components are deployed on commodity server architecture, supplemented by programmable accelerators when necessary. The SDN architecture described in this article can support O-RAN specifications.

[0130] Edge computing is likely to target two different use cases. The first will serve as support for containerized telecom infrastructure (such as 5G RAN, UPF, security functions), while the second will be used for containerized service workloads, both from telecom companies and third parties such as suppliers or enterprise customers. In both cases, edge computing is actually a special case of GiLAN, where traffic is interrupted at highly distributed locations for special processing. In many cases, these locations will have limited resources (power, cooling, space).

[0131] The SDN architecture described in this article can be well suited to support requirements for a very lightweight footprint, can support compute and storage resources in sites that are far from the associated control functions, and can be location-aware in the way workloads and storage are deployed. Some sites may have only one or two compute nodes, delivering a very specific set of services to a highly localized set of users or other services. There may be a hierarchy of sites, with central sites densely connected with many paths, regional sites multi-pathed with two to four uplink paths, and remote edge sites may be connected to only one or two upstream sites.

[0132] This requires great flexibility in how the SDN architecture is deployed and how (and where) the tunnel traffic in the overlay is terminated and bound to the core transport network (SRv6, MPLS, etc.). Similarly, in sites hosting telco cloud infrastructure workloads, the SDN architecture described in this article can support that specialized hardware (GPUs, SmartNICs, etc.) is required for high-performance workloads. There may also be workloads that require SR-IOV. Therefore, the SDN architecture can also support the creation of VTEPs at the ToR and linking them back to the overlay as VXLAN.

[0133] Expect to see a mix of fully distributed Kubernetes microclusters, where each site runs its own master server, and SDN architectures that may support scenarios like remote computing.

[0134] For use cases involving enterprise Kubernetes platforms, high-performance cloud-native applications support financial services platforms, online gaming services, and managed application service providers. The cloud platforms that deliver these applications must provide high performance, failure resistance, high security, and visibility. Applications hosted on these platforms tend to be developed in-house. Application developers and platform owners work with infrastructure teams to deploy and operate the organization's application instances. These applications tend to require high throughput (>20Gbps per server) and low latency. Some applications may also use multicast to send signaling or payload traffic. Additional hardware and network infrastructure can be leveraged to ensure availability. Applications and microservices will be partitioned using namespaces in the cluster. In high-security environments, isolation between namespaces is critical. While a default deny policy is the standard state of affairs in zero-trust application deployment environments, additional network segmentation using virtual routing and forwarding instances (VRFs) adds an extra layer of security and allows the use of overlapping network ranges. Overlapping network ranges are a key requirement for managed application hosting environments that tend to standardize a set of reachable endpoints for all hosting customers.

[0135] Complex microservices-based applications tend to utilize sophisticated network filters. The SDN architecture described in this article can provide high-performance firewall filtering at scale. This filtering can exhibit consistent forwarding performance with less latency degradation regardless of rule set length or sequence. Some customers may also face the same regulatory pressures as telecommunications companies for application separation, not only at the network layer, but also in the kernel. Finance departments, as well as others, require data plane encryption, especially when running on public clouds. In some examples, the SDN architecture described in this article can include features for meeting these requirements.

[0136] In some examples, when automated through an application dev / test / staging / production continuous integration / continuous development (CI / CD) pipeline, the SDN architecture can provide a GitOps-friendly UX for strict change management control, auditing, and the reliability of making changes in production multiple times or even hundreds of times per day.

[0137] Administrators of SDN may wish to add virtual execution elements (e.g., pods or VMs) of remote network clusters as endpoints of network services (e.g., exposing components of the backend of network applications to the container orchestration platform of the SDN so that the endpoints of network services run network applications) to maximize efficiency and reduce execution costs. The technology described herein can allow administrators to maximize the efficiency of adding and removing virtual execution elements as network service endpoints by avoiding the creation of duplicate services that may be relied upon by implementing DNS servers. The technology also allows administrators to reduce execution costs by dynamically adding virtual execution elements of any cluster (e.g., adding virtual execution elements of clusters with cheaper processing or memory overhead) as the ability to serve network endpoints, and enables network traffic to be forwarded between endpoints of network services through virtual routers rather than through DNS servers or a series of upstream and downstream routers (e.g., service mesh networks). Administrators can also use virtual routers to load balance network traffic and processing requirements of network applications between virtual execution elements residing in various clusters added as network service endpoints. The technology enables administrators to establish communications between endpoints of network services at the Internet Protocol (IP) level, regardless of which cluster the endpoints are located in, by assigning IP addresses directly to virtual actuators assigned as endpoints based on a reliable protocol, without relying on DNS servers or developing complex service meshes.

[0138] Network administrators can use DNS servers to manage the IP addresses assigned to virtual executive elements added as network service endpoints. However, DNS servers reduce the control of network administrators over the dynamic characteristics of software-defined networks, such as constantly creating and deleting virtual executive elements. DNS servers have lifetime parameters for virtual executive element entries, which network administrators cannot customize. Network administrators can consider developing service mesh networks instead, to orchestrate the addition and removal of virtual executive elements as network service endpoints through a series of next hops between router networks. However, developing service mesh networks is complex and inefficient, and difficult to manage. Compared with using DNS servers or service mesh networks, the technology described herein uses a well-established routing protocol that is effectively executed. Technology reduces computing overhead (e.g., consumed processing cycles, memory, memory bus bandwidth, or other computing resources associated with power consumption) to allow reliable and effective service announcements, while improving the operation of the underlying endpoints running network applications.

[0139] Service information can be included directly in the advertised network layer reachability information (NLRI), as further described in detail in U.S. application Ser. No. 18 / 193,583, filed on Mar. 30, 2023, and entitled “NETWORK SERVICE DIRECTORY,” which is incorporated herein by reference in its entirety. However, encoding service information in the advertised NLRI requires compatibility of intermediate routers, which may impact the scalability of networks with multiple clusters supporting various network applications. The techniques described herein encode service information in attributes and resolve which endpoints will be assigned to a network service at the control plane level. In this way, network administrators can more efficiently add virtual execution elements from remote clusters as endpoints to network services at scale.

[0140] In some examples, orchestrator 23 may use network controller 24 to create replicas of a network service (e.g., network service 58) in multiple network clusters (e.g., network clusters 170A and 170B). Network controller 24 may add a virtual executive of a remote cluster (e.g., Pod 22X) as an endpoint of a replica of a network controller of network service 58 created in network cluster 170B (not shown) by assigning an IP address to a virtual executive associated with the replica of network service 58 created in network cluster 170B. In this example, network controller 24 may store the IP addresses of the endpoints of network service 58 and the IP addresses of the endpoints of the replicas of network service 58 created in multiple network clusters in a DNS server to allow network service 58 to implement policies associated with forwarding network traffic of network applications running on the endpoints of network service 58. However, the DNS server may have a long time to live (TTL) for endpoint references, which may delay the operation of the SDN as endpoints are created or removed to accommodate the dynamic nature of network traffic loads. For example, the network controller 24 may have added Pod 22X as an endpoint of the network service 58 and stored the assigned IP address in the DNS server. However, Pod 22X may be deleted, and the network service 58 will have to wait for the DNS server to be updated in order to effectively implement policies such as load balancing network traffic for network applications. Administrators of the SDN have no control over the DNS server and have less autonomy in customizing the SDN.

[0141] In other examples, administrators of SDNs can develop and maintain service mesh networks based on proprietary protocols to add virtual execution elements of remote clusters (e.g., Pod 22X) as endpoints of network services (e.g., network services 58). The service mesh network will include a series of upstream and downstream routers (e.g., chassis switches 18 and TOR switches 16), which transmit endpoint creation requests for network services 58 and enable endpoints of network services 58 to communicate (e.g., forward network traffic) through a series of next hops. However, service mesh networks can be complex and difficult to maintain with large SDNs. By constantly relaying requests and network traffic with incremental next hops between upstream and downstream routers, service mesh networks also require more computing and network resources. Compared with using a DNS server or a service mesh network, the technology described herein uses a well-established routing protocol that is effectively executed. The technology reduces computing overhead (e.g., consumed processing cycles, memory, memory bus bandwidth, or other computing resources associated with power consumption) to allow reliable and efficient service announcements while improving the operation of the underlying endpoints running network applications.

[0142] In another example, a network cluster (e.g., network cluster 170A) can execute a network service (e.g., NS 58) and notify the remote network cluster (e.g., network cluster 170B) of the information of the network service to add the virtual execution element (e.g., Pod 22X) of the remote network cluster as an endpoint of the network service. The orchestrator 23 can establish a first network cluster 170A to include a virtual network 50A, a virtual network router 52A, a chassis switch 18A, a TOR switch 16A, and a server 12A. The orchestrator 23 can establish a second network cluster 170B to include a virtual network 50N, a virtual network router 52N, a chassis switch 18M, a TOR switch 16N, and a server 12X. In addition, the orchestrator 23 can add Pod 22A as an endpoint of the network service 58 executed in the first network cluster 170A, wherein the network service 58 exposes the backend of the network application running on the endpoint of the network service 58 to the container orchestration platform of the SDN. In some examples, control node 32 of first network cluster 170A may generate an advertisement including information of network service 58 (e.g., the FQDN of the network service, the port used by the network service, and the protocol used by the network service). In some examples, the first network cluster and the second network cluster may share the same hardware elements.

[0143] Rather than relying on an external DNS server or developing and maintaining a complex service mesh to add the Pod 22X as an endpoint of the network service 58, a control node of the controller 32 distributed to the first network cluster 170A can generate an advertisement with information about the network service 58 and send the advertisement to the second network cluster 170B to add the Pod 22X as an endpoint of the network service 58. The control node of the controller 32 distributed to the first network cluster 170A can generate an advertisement with information about the network service 58, in accordance with a network routing protocol (e.g., Border Gateway Routing Protocol) (and other routing specific information) that is typically used to advertise routes in or between networks. The first network cluster 170A and the second network cluster 170B can be directly or indirectly peered together through an intermediate BGP router and exchange routing information (e.g., IP addresses) to establish communication between the virtual executive elements of the first network cluster 170A (e.g., Pod 22A) and the virtual executive elements of the second network cluster 170B (e.g., Pod 22X) added as endpoints of the network service 58.

[0144] However, including the network layer reachability information (NLRI) of the network service 58 in the BGP advertisement may require that the intermediate routers be configured as BGP routers. By including the advertisement with the information of the network service 58 in the BGP advertisement, the network administrator can limit which virtual execution elements (e.g., Pod 22X) of the remote cluster (e.g., cluster 170B) can be added as endpoints to the network service 58 based on whether the intermediate router (e.g., route reflector) connecting cluster 170A to the remote cluster is a configured BGP router.

[0145] According to the techniques described herein, the control node 32 of the cluster 170A can encode the information of the network service 58 with an extensible routing protocol attribute. In some cases, the control node 32 can encode the information of the network service 58 in a custom BGP service attribute or a BGP extended community attribute. The custom BGP service attribute or the BGP extended community attribute can be an optional transitive BGP attribute that includes a bit array or vector that can be announced with other network layer reachability information. The vector included in the optional transitive BGP attribute only needs to be understood by the receiving BGP peer (e.g., cluster 170B). Intermediate routers between BGP peers do not need to understand the information included by the BGP optional transitive attribute. In this way, by encoding the information of the network service 58 with the BGP optional transitive attribute, the BGP peer (e.g., cluster 170B) receives the port, protocol, and fully qualified domain name of the network service 58 without configuring the intermediate router (e.g., route reflector) as a BGP router. These techniques can reduce the operational costs associated with updating intermediate router techniques.

[0146] In operation, network cluster 170A may encode information about network service 58 with extensible routing protocol attributes (e.g., BGP extended communities or customized BGP service attributes). Network cluster 170A may include the fully qualified domain name (FQDN) of network service 58 and the port and protocol used by network service 58 in the advertisement encoded with the optional transitive BGP attributes. Network cluster 170A may use a BGP controller of control node 32 to generate BGP attribute values ​​configured to relay information about network service 58. Figure 6 Further details of network cluster 170A utilizing BGP extended community attributes to encode information for network services 58 are described in greater detail throughout the specification.

[0147] In some cases, network cluster 170A may encode information of network service 58 in a custom BGP attribute. Network cluster 170A may encode information of network service 58 in an optional transitive BGP attribute. In this way, network cluster 170A may advertise information of network service 58 to other network clusters without changing the information of network service 58 to conform to predefined attributes. Network cluster 170A may use custom BGP service attributes to advertise information of network service 58. Network cluster 170A may use custom BGP service attributes to advertise information regardless of whether the custom attributes are supported by each BGP implementation. Examples of custom BGP service attributes that include network service information (e.g., port, protocol, fully qualified domain name) are as follows:

[0148] struct Port:public ParseObject{

[0149] uint32_t service_id;

[0150] uint16_t port;

[0151] uint8_t protocol;

[0152] uint32_t name0;

[0153] uint32_t name1;

[0154] uint32_t name2;

[0155] uint32_t name3;

[0156] explicitPort();

[0157] static std::string ToString(Port*port);

[0158] static Port*FromString(std::string name);

[0159] };

[0160] struct ServiceSpec:public BgpAttribute{

[0161] static const int kSize=-1;

[0162] static const uint8_t kFlags=Optional|Transitive;

[0163] ServiceSpec():BgpAttribute(Services,kFlags){}

[0164] explicit ServiceSpec(const BgpAttribute&rhs):BgpAttribute(rhs){}

[0165] std::vector<Port*>ports;

[0166] struct PortValue:public ParseObject{

[0167] uint32_t service_id;

[0168] uint16_t port;

[0169] uint8_t protocol;

[0170] uint32_t name0;

[0171] uint32_t name1;

[0172] uint32_t name2;

[0173] uint32_t name3;

[0174] };

[0175] std::vector<PortValue*>port_values;

[0176] virtual int CompareTo(const BgpAttribute&rhs_attr)const{

[0177] int ret=BgpAttribute::CompareTo(rhs_attr);

[0178] if (ret!=0)

[0179] return ret;

[0180] KEY_COMPARE(ports,static_cast<const ServiceSpec&> (rhs_attr).ports);

[0181] return 0;

[0182] }

[0183] virtual void ToCanonical(BgpAttr*attr);

[0184] virtual std::string ToString()const;

[0185] virtual size_t EncodeLength()const;

[0186] };

[0187] The network cluster 170A may generate an advertisement that complies with BGP and includes an optional transferable BGP attribute that is configured to specify information of the network service 58 (e.g., a set of BGP extended community attributes or a custom BGP attribute). The network cluster 170A may generate an advertisement that complies with any routing protocol that allows extensible attributes that can be used to transfer the network service 58 between multiple network clusters. The network cluster 170A may use a BGP controller of a control node of the control 32 to generate an advertisement according to a BGP protocol (e.g., MP-BGP). The network cluster 170A may send the generated advertisement to the network cluster 170B. The network cluster 170A may use one or more intermediate routers to send the generated advertisement to the network cluster 170B, regardless of whether the intermediate router is configured as a BGP router. In this way, the network cluster 170B can add the virtual execution element of the network cluster 170B as an endpoint of the network service 58 without requiring a DNS server, a service grid, or a compatible BGP router.

[0188] The network cluster 170B may receive the announcement via an intermediate router and / or a BGP controller of a control node of the control 32 distributed to the network cluster 170B. The network cluster 170B may generate a network service directory 62 based on the information of the network service 58 included in the optional transitive BGP attributes (e.g., BGP extended community set or custom BGP attributes) generated from the network cluster 170A. The network cluster 170B may generate the network service directory 62 using a configuration node of the configuration 30 assigned to the network cluster 170B. The network cluster 170B may also use the network controller 24 to receive and process the announcement and network service information included in the BGP attributes sent from the network cluster 170A. The network controller 24 may generate the network service directory 62 in the virtual network 50N executed in the network cluster 170B. The network controller 24 may also generate the network service directory 62 using an API (e.g., kube-api). The network service directory 62 may be a copy of the network service 58 in terms of the fully qualified domain name, port, and protocol at which the network service directory 62 is instantiated.

[0189] The network controller 24 may use the network service catalog 62 to add one or more virtual executive elements of the network cluster 170B (e.g., Pod 22X) as endpoints of the network service 58 executed on the network cluster 170B to establish communication between the endpoints of the network service 58 via the virtual router 21. The network controller 24 may use the network service catalog 62 to allocate an IP address to the Pod 22X of the second network cluster 170B based on the information of the network service 58 included in the advertisement received by the second network cluster 170B. The virtual router 21 may include a routing table that stores the IP addresses allocated to the Pod 22A and the Pod 22X when the Pod 22A and the Pod 22X are added as endpoints of the network service 58, so that the Pod 22A of the network cluster 170A and the Pod 22X of the network cluster 170B can effectively communicate (e.g., load balancing network traffic or processing requirements of network applications exposed by the network service 58). The virtual router 21 storing the IP addresses assigned to the Pod 22X based on the network service directory 62 avoids the need to use a DNS server or a service mesh network, because the Pod 22X can communicate directly with other endpoints (e.g., Pod 22A) of the network service 58 through the virtual router 21. Instead of the network service 58 communicating with the DNS server to load balance the network traffic or processing requirements of the network application between the Pod 22A of the network cluster 170A and the Pod 22X of the network cluster 170B, the network service 58 can load balance the network traffic or processing requirements of the network application between the Pod 22A of the network cluster 170A and the Pod 22X of the network cluster 170B by using the routing table of the assigned IP addresses of the Pod 22A and the Pod 22X stored in the virtual router 21. In addition, the network service 58 does not have to rely on multiple intermediate routers between the network cluster 170A and the network cluster 170B to route the network traffic between the Pod 22A and the Pod 22X, as defined by the service mesh network. The network service 58 only needs to access the IP address stored in the virtual router 21.

[0190] Thus, the technique may enable Pod 22A of a first network cluster 170A and Pod 22X of a second network cluster 170B to communicate directly at an IP level controlled by a virtual router 21, rather than using a DNS server or having a large number of next desired upstream routers defined by a service mesh. The network controller 24 may utilize established protocols to effectively establish Pod 22X of the second network cluster 170B as an endpoint for the network service 58 of the first network cluster 170A, without requiring external hardware such as a DNS server, or without requiring the development of a service mesh. Given that the network routing protocol is already used in networks and has undergone extensive testing and troubleshooting to provide consistent operation in a variety of different network topologies, while also having a well-defined software and / or hardware implementation suite, the network routing protocol may provide lightweight and efficient (e.g., in terms of computational utilization) advertising of information associated with the network service 58 between the first network cluster 170A and a remote network cluster (e.g., the second network cluster 170B). The techniques described herein allow administrators of the SDN to better control which virtual executive elements (eg, Pod 22A and Pod 22X) run network applications exposed by network services 58 without requiring the development of DNS servers or complex service meshes.

[0191] In addition, the technology allows the network service directory 62 to be initialized in any remote cluster, regardless of whether the intermediate router connecting the remote cluster to the cluster 170A is a BGP router configured to process complex NLRI information. Cluster 170A can encode the information of network services 58 using extensible routing protocol attributes. For example, cluster 170A can encode the information of network services 58 in a set of BGP communities of BGP extended community attributes or in custom BGP service attributes. Cluster 170A, which uses optional transferable BGP attributes to relay the information of network services 58, enables cluster 170A to use one or more services to forward traffic associated with an application, and the one or more services can access the virtual execution element of the remote cluster by sending an announcement with a BGP attribute, and the BGP attribute can include information of multiple services. Instead of cluster 170A sending multiple announcements, each announcement including information of a single service, cluster 170A can send information of multiple services in one announcement by using BGP attributes. In this way, network administrators can easily configure the virtual execution element of the remote cluster as an endpoint of multiple network services, thereby also reducing the processing power associated with generating announcements for each service.

[0192] Figure 22 is a block diagram illustrating an example of a cloud native SDN architecture 200 for a cloud native network in accordance with the techniques of the present disclosure. The SDN architecture 200 is illustrated in a manner that abstracts the underlying connections between various components. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A-230N ("configuration nodes" or "config nodes", and collectively referred to as "configuration nodes 230") and control nodes 232A-232K (collectively referred to as "control nodes 232"). The configuration nodes 230 and the control nodes 232 may represent Figure 1 1. The configuration node 230 and the control node 232 are example implementations of the configuration node 30 and the control node 32. The configuration node 230 and the control node 232, although shown as separate from the server 12, can be executed as one or more workloads on the server 12.

[0193] Configuration node 230 provides a northbound representational state transfer (REST) ​​interface to support intent-driven configuration of SDN architecture 200. Example platforms and applications that can be used to push intent to configuration node 230 include virtual machine orchestrator 240 (e.g., Openstack), container orchestrator 242 (e.g., Kubernetes), user interface 242, or other one or more applications 246. In some examples, SDN architecture 200 has Kubernetes as its underlying platform.

[0194] The SDN architecture 200 is divided into a configuration plane, a control plane, and a data plane, as well as an optional telemetry (or analysis) plane. The configuration plane is implemented with horizontally scalable configuration nodes 230, the control plane is implemented with horizontally scalable control nodes 232, the data plane is implemented with computing nodes, and the telemetry plane is implemented with telemetry nodes 260. The SDN architecture can also distribute configuration nodes 230, control nodes 232, and servers 12 across multiple clusters.

[0195] At a high level, the configuration node 230 uses the configuration store 224 to manage the state of the configuration resources of the SDN architecture 200. In general, a configuration resource (or more simply a "resource") is a named object schema that includes data and / or methods that describe a custom resource, and an application programming interface (API) is defined for creating and manipulating data through an API server. kind is the name of the object schema. Configuration resources may include Kubernetes native resources such as Pods, Ingress, ConfigMap, Service, Role, Namespace, Node, Network Policy, or Load Balancer.

[0196] Configuration resources also include custom resources, which are used to extend the Kubernetes platform by defining an application programming interface (API) that may not be available in the default installation of the Kubernetes platform. In the example of the SDN architecture 200, custom resources can describe the physical infrastructure, virtual infrastructure (e.g., VN 50 and / or VNR 52), configuration, and / or other resources of the SDN architecture 200. As part of configuring and operating the SDN architecture 200, various custom resources (e.g., VNR 52 within vRouter 21) can be instantiated. Instantiated resources (whether local or custom) can be referred to as objects or instances of resources, which are persistent entities in the SDN architecture 200 that represent the intent (desired state) and state (actual state) of the SDN architecture 200.

[0197] Configuration node 230 provides an aggregate API for performing operations (i.e., create, read, update, and delete) on configuration resources of SDN architecture 200 in configuration storage 224. Load balancer 226 represents one or more load balancer objects that load balance configuration requests between configuration nodes 230. Configuration storage 224 can represent one or more etcd databases. Configuration node 230 can be implemented using Nginx.

[0198] SDN architecture 200 can provide networking for both Openstack and Kubernetes. Openstack uses a plug-in architecture to support networking. Using the virtual machine orchestrator 240 as Openstack, the Openstack network plug-in driver converts the Openstack configuration objects into SDN architecture 200 configuration objects (resources). The computing nodes run Openstack nova to call out virtual machines.

[0199] With the container orchestrator 242 being Kubernetes, the SDN architecture 200 is used as a Kubernetes CNI. As described above, Kubernetes native resources (pods, services, ingress, external load balancers, etc.) can be supported, and the SDN architecture 200 can support customized resources for Kubernetes for advanced networking and security of the SDN architecture 200.

[0200] Configuration nodes 230 provide REST monitoring to control nodes 232 to monitor configuration resource changes affected by control nodes 232 within the computing infrastructure. Control nodes 232 receive configuration resource data from configuration nodes 230 by observing resources and build a complete configuration graph. A given one of control nodes 232 consumes the configuration resource data associated with the control node and distributes the required configuration to virtual routers 21 (i.e., virtual router agents— Figure 1 2N+1 configuration nodes 230 and 2N control nodes 232.

[0201] The control nodes 232 may be in different network clusters and distribute routes between computing nodes within the corresponding clusters. The control nodes 232 may use MP-BGP to exchange routes between control nodes 232 of different clusters, and the control nodes 232 may be peered with any external BGP-supported gateway or other router. The control nodes 232 may use route reflectors.

[0202] exist Figure 2 In the example of FIG. 1 , configuration node 230A, control node 232A, and server 12A may be deployed as (or in other words, configured to support) a first network cluster 270A. Configuration node 230N, control node 232K, and server 12X may be deployed or otherwise configured to support a second network cluster 270B. In some examples, first network cluster 270A and second network cluster 270B may share hardware components (e.g., components of network controller 24).

[0203] Pod 250 and virtual machine 252 are examples of workloads that can be deployed to computing nodes by container orchestrator 242 or virtual machine orchestrator 240, respectively, and interconnected by SDN architecture 200 using one or more virtual networks. In some examples, pod 250A and virtual machine 252A can be endpoints of network services 258. Network services 258 can be any collection of virtual execution elements added as endpoints for running one or more applications 246. Figure 2 In the example of , network service 258 may initially be configured to include pod 250A and virtual machine 252A as endpoints for running application 246 .

[0204] In operation, the control node 232A of the network cluster 270A may encode information of the network service 258 using one or more BGP attributes. For example, the control node 232A may encode information of the network service 258 in a customized BGP service attribute or a set of BGP extended community attributes. The control node 232A may encode information of the network service 258, such as a service identifier, a port identifier, a protocol identifier, and a fully qualified name identifier associated with the network service 258. In an example where the control node 232A encodes the network service information using BGP extended communities, the control node 232A may encode the information of the network service 258 in a manner that complies with the bit requirements of the BGP extended community attributes. The control node 232A may include a BGP controller or router for encoding information of the network service 258 in a set of BGP extended communities.

[0205] The control node 232A of the network cluster 270A may generate an advertisement including one or more BGP attributes encoded with information about the network service. The control node 232A may generate an advertisement that complies with a BGP protocol (e.g., MP-BGP). The control node 232A may broadcast or send an advertisement (having one or more BGP attributes encoded with information about the network service 258) to the control node 232K of the network cluster 270B according to the protocol (e.g., MP-BGP). The control node 232K of the network cluster 270B may include a second BGP controller or router for receiving and processing advertisements sent from other BGP controllers or routers of a remote network cluster (e.g., a BGP controller or router of the control node 232A).

[0206] The control node 232K may process the BGP attributes to extract the information of the service 258. The control node 232K may relay the extracted information of the network service 258 to the configuration node 230N to generate a network service directory 262. The configuration node 230N may generate the network service directory 262 as a copy of the network service 258. The configuration node 230N may instantiate the network service directory 262 with the obtained information of the network service 258 (e.g., the FQDN of the network service 258, the port used by the network service 258, and the protocol used by the network service 258).

[0207] In some examples, the control node 232K may use a second BGP router or controller of the second network cluster 270B to determine whether the announcement received from the control node 232A includes a label or tag indicating a community of which the second network cluster 270B is a member. If the control node 232K determines that the announcement sent from the control node 232A includes a label or tag of a community of which the second network cluster 270B is not a member, the control node 232K ignores the announcement. In response to the control node 232K determining that the second network cluster 270B is part of the community identified in the label or tag included in the announcement sent from the control node 232A, the control node 232K may relay information of the network service 258 included in the announcement to the configuration node 230N. The configuration node 230N may then generate a network service catalog 262 based on the obtained information of the network service 258. The network service catalog 262 may be, for example, an Atlas Kubernetes operator service.

[0208] The network service catalog 262 may request the network controller 24 to add the pod 250X and / or VM 252X as an endpoint of the network service 258 so that the pod 250A and VM 252A can communicate directly with the pod 250X and / or VM 252X via a virtual router. The network controller 24 may use a mapping controller to add the pod 250X and / or VM 252X as an endpoint of the network service 258 by assigning a floating IP address to the pod 250X and / or VM 252X. The network controller 24 may use a mapping controller to statically or dynamically assign a floating IP address to the pod 250X and / or VM 252X based on the FQDN, port, and protocol of the network service 258 used to instantiate the network service catalog 262. The network controller 24 may include the floating IP address assigned to the pod 250X and / or VM 252X in the routing tables of one or more virtual routers to enable communication between all endpoints (e.g., pod 250A, VM 252A, pod 250X, and VM 252X) running one or more applications 246, regardless of in which network cluster the endpoints are located.

[0209] The techniques described herein allow direct communication between one or more virtual execution elements (e.g., pod 250A, VM 252A, pod 250X, and VM 252X) for running one or more network applications 246 exposed by a network service 258, without relying on external DNS servers or developing complex service mesh networks. By utilizing well-established protocols (e.g., BGP) to advertise information of the network service 258, an administrator of the SDN can effectively add the pod 250X and virtual machine 252X of the second network cluster 270B, for example, as backend endpoints for running one or more network applications 246 exposed by the network service 258, without using a DNS server of the service mesh network. The network controller 24 can add the pod 250X and VM 252X as endpoints of the network service 258 that exposes the one or more network applications 246 based on the information of the network service 258 included in the advertisement for instantiating the network service catalog 262. The network controller 24 does not need to add the pod 250X or VM 252X as an endpoint of the network service 258, where the DNS server or service mesh defines a series of routers connecting the first network cluster 170A and the second network cluster 170B, because the network controller 24 only needs to generate the network service directory 262 as a reference for how IP addresses should be assigned to the pod 250X and VM 252X when adding the pod 250X and VM 252X as endpoints of the network service 258. Once the network controller 24 assigns IP addresses to the pod 250X and VM 252X based on the network service directory 262, the network controller 24 stores the IP addresses in the virtual router to enable direct communication between all endpoints of the network service 258 (e.g., pod 250A, VM 252A, pod 250X, and VM 252X) to load balance network traffic and processing requirements of one or more applications 246 exposed by the network service 258.

[0210] The technology described herein can also utilize the characteristics of optional transitive BGP attributes to effectively implement the communication of service information without configuring intermediate routers as BGP routers. For example, network cluster 270A and network cluster 270B can be communicatively coupled with a series of routers and / or route reflectors. In the case where network cluster 270A sends the information of network service 258 to network cluster 270B by embedding the information of network service 258 in the announcement, the intermediate routers and / or route reflectors will need to be configured to receive the BGP announcement and interpret the NLRI included in the announcement. However, network cluster 270A can encode the information of network service 258 in a set of BGP communities (e.g., an array or vector of bits) or a customized BGP attribute (e.g., an optional transitive BGP attribute) to allow the information of network service 258 to reach network cluster 270B, regardless of how the intermediate routers and / or route reflectors are configured.

[0211] Figure 3 is a conceptual diagram illustrating an example network 300 having multiple network clusters according to one or more aspects of the present disclosure. Figure 3 The example of FIG. 3 shows one or more software defined networks (SDNs) arranged as cloud computing cluster 370A, cluster 370B, and cluster 370C (collectively referred to as "clusters 370," and representing any number of clusters). Each of cloud computing clusters 370 is implemented by a computing infrastructure that may be virtualized to support one or more services implemented by the cluster. For example, one or more of clusters 370 may be provided on multiple servers hosted on a network (e.g., the Internet) to store, manage, and process data, or perform other functions.

[0212] In some examples, one or more of clusters 370 may be located within an enterprise, with some or all of the other clusters 370 being remote. In other examples, some or all of clusters 370 may be remote from the enterprise. Furthermore, in some examples, clusters 370 may all be included within a single data center. In other examples, each of clusters 370 may be deployed within its own data center, or, potentially, one or more of clusters 370 may span multiple data centers or geographic regions. Each cluster 370 also includes a corresponding network 44 and any number of servers (e.g., servers 12A, 12B, and 12C) for providing computing resources. Typically, Figure 3 Each component shown in (e.g., cluster 370, network controller 324 within each cluster 370, and server 12 within each cluster 370) can communicate via one or more networks, which can be or include the Internet or any public or private communication network or other network. Such networks may include one or more networks 44 within cluster 370.

[0213] Figure 3 Included are multiple software-defined network controllers, referred to herein as network controllers 324A, 324B, and 370C (collectively, "network controllers 324"), each located within clusters 370A, 370B, and cluster 130C, respectively. Each network controller 324 configures aspects of their respective cluster 370 and may be implemented by a computing device and / or processing circuitry, whether physical or virtual.

[0214] In some examples, cluster 370 may include one or more configurable services (e.g., service 358A) to expose applications executing on a set of endpoints. Figure 3 In the example of , service 358A may include endpoints 351A to endpoint 351N (collectively referred to as "endpoints 351" and representing any number of endpoints), as well as policies for making endpoints 351 accessible. Endpoint 351 can be a virtual execution element (e.g., a pod or virtual machine) executing on any server (e.g., servers 12A-12C). In some cases, network controller 324A may continuously scan for virtual execution elements executing on any server (e.g., server 12) within cluster 370A that match a selector defined by service 358A. In response to network controller 324A matching a virtual execution element to a selector defined by service 358A, network controller 324A may add the matching virtual execution element as an endpoint of endpoint 351.

[0215] According to one or more aspects of the present disclosure, a network controller 324A (also referred to herein as "NC 324A") may add a virtual executive element executed in a network cluster 370B and / or a network cluster 370C as an endpoint 351 of a network service 358A (also referred to herein as "network service 358" or "service 358") associated with the network controller 324A via a routing protocol (e.g., MP-BGP). For example, the NC 324A may utilize BGP attributes when announcing information of the network service 358A to a remote network cluster (e.g., network cluster 370B and / or network cluster 370C). The NC 324A may encode the information (port, protocol, FQDN) of the network service 358 with optional transitive BGP attributes. For example, the NC 324A may encode the information of the network service 358 in multiple BGP communities such that the encoding complies with the bit requirements of the BGP communities. In another example, the NC 324A may encode the information of the network service 358 in a custom BGP service attribute.

[0216] NC 324A may generate advertisement 366 to include one or more BGP attributes encoded with network service information according to the MP-BGP routing protocol. Advertisement 366 may include BGP attributes as a bit array or vector. For example, advertisement 366 may include a service identifier BGP community as a vector having at least 32 bits, a port identifier BGP community as a vector having at least 32 bits, a protocol identifier BGP community as a vector having at least 32 bits, and a plurality of FQDN BGP communities as a set of vectors each having at least 32 bits. NC 324A may include a BGP controller or router that makes advertisement 366. NC 324A may also include a community tag or label in advertisement 366 using the BGP controller or router to indicate that advertisement 366 is intended to be received by members or peers of the community.

[0217] NC 324A may send advertisement 366 to any of NC 324B of cluster 370B, NC 324C of cluster 370C, and / or NC 324. If the network cluster in which NC 324 is located is not a member or peer of the community identified in the community tag or label included in the advertisement 366, any of NC 324B, NC 324C, and / or NC 324 may ignore the advertisement. If NC 324 is configured as a peer-to-peer network cluster that is a member of the community identified in the community tag or label included in the advertisement 366, NC 324 may use a BGP controller or router to extract service information for network service 358A from a set of BGP communities included in the advertisement 366. In the example where the advertisement 366 includes a custom service BGP attribute, the intermediate BGP routers and / or route reflectors may accept and pass the service BGP attribute regardless of whether the router or route reflector supports the service BGP attribute. NC 324 may instruct the BGP controller to send the information included in advertisement 366 to the mapping controller in NC 324 .

[0218] Each NC 324 that receives the announcement 366 may generate a network service directory (e.g., directory 362B or 362C, collectively referred to as directory 362) based on the information of service 358A extracted from one or more BGP attributes. For example, NC 324B and NC 324C may configure the information of service 358A for directory 362B and directory 362C, respectively. Directory 362 may allocate virtual execution elements in its corresponding cluster as endpoints 351. In some examples, NC 324 may generate directory 362 as a service without a selector.

[0219] The directories 362 executed in the network cluster 370B and the network cluster 370C may add the virtual executives executed in the corresponding network clusters as the endpoints 351 of the service 358A. In some examples, the directory 362 may be a service instance without a selector. In such an example, the directory 362 may add the virtual executives of each corresponding network cluster as the endpoints of the service 358. The NC 324B and / or the NC 324C may use a mapping controller to map the directories 362B and 362C to the network addresses and service ports 358A included in the announcement 366, respectively. The NCs 324B and 324C may use a mapping controller to add the IP addresses of the virtual executives in the corresponding network clusters as the endpoints 351 of the service 358A based on the labels assigned to the directory 362. The NCs 324B and 324C may assign labels to the directory 362 that match the name of the service 358A. Network controllers 324B and 324C may add endpoints to service 358A by instructing directory 362 to assign a floating IP address configured as an endpoint for service 358A to each virtual executive added as an endpoint for service 358A.

[0220] The techniques described herein allow network controller 324A to advertise information of network service 358 to network cluster 370B and network cluster 370C by utilizing optional transitive BGP attributes (e.g., BGP extended communities or custom service BGP attributes). The BGP attributes encoded with the information of network service 358 allow clusters 370B and 370C to add virtual executives as endpoints of endpoints 351 of network service 358 for exposing network applications. Once network controller 324B and / or network controller 324C have added the virtual executives of network cluster 370B and network cluster 370C, respectively, network controller 324A can utilize network service 358 to implement load balancing techniques to direct network traffic of network applications exposed by network service 358 to endpoints 351 assigned to network service 358. For example, NC 324A can use virtual IPs (e.g., kube-vip of a Kubernetes cluster) and load balancers (e.g., Kubernetes load balancing policy settings) to load balance network traffic based on virtual IPs assigned to endpoints 351 stored in virtual routers (e.g., Figure 1 The IP address of each endpoint 351 in the virtual router 21) is used to effectively regulate the ingress and egress traffic of the endpoint 351 running the network application.

[0221] In some examples, the announcement 366 can follow any type of BGP routing protocol (or other type of routing protocol) and allow the service 358A to utilize the attributes included in the BGP routing protocol. For example, the network controller 324A or any other controller within the distributed network can establish a BGP community or extended community associated with the network traffic of the network application disclosed by the service 358A. The network controller 324A can establish a BGP community to include community members (e.g., network cluster 370B and / or network cluster 370C) and allow community members to cooperate when processing network traffic routing. For example, the network controller 324A can establish a BGP community to include community attributes that can be used by routing policies to automatically determine routing decisions. Once the network cluster 370B and / or the network cluster 370C add the virtual execution element as an endpoint of the endpoint 351 based on the announcement 366 and the directory 362, the network controller 324B and the network controller 324C push the routing policy to the newly added endpoint according to the community attributes specified in the announcement 366. The routing policy can use matching conditions to enable import or export statements for specific ingress and egress network traffic. By utilizing BGP communities, the network controller 324A can send information for multiple services in a single advertisement. In this way, the techniques described herein reduce the complexity of configuring virtual executives of remote clusters as endpoints for multiple services.

[0222] Figure 4 is a block diagram illustrating an example network with multiple network clusters according to one or more aspects of the present disclosure. Figure 1 , Figure 2 and Figure 3 As shown in Figure 4 Many of the components shown in FIG. 1 may correspond to those previously described in conjunction with Figure 1 and Figure 2 Like numbered components are described. Figure 4 Such similarly numbered systems, devices, components and items shown in the Figure 1 and Figure 2 The descriptions provided are described in a consistent manner, although in some examples, such systems, devices, components, and items may be directed to alternative implementations having more, fewer, and / or different capabilities.

[0223] exist Figure 4 In the example of , network cluster 370A and network cluster 370B may include control node 432, configuration node 430, and virtual router 421. Control node 432 may represent, for example Figure 1 The control node 32 or Figure 2 As an example, the configuration node 430 may represent Figure 1 Configuration node 30 or Figure 2 Configuration node 230. Figure 4 In the example, virtual router 421 may represent Figure 1 The network cluster 370A can execute applications running on virtual execution elements added as endpoints of the network service 458. The network service 458 can use multiple endpoints, such as Pod 22A-1 to Pod 22A-N to execute applications. In some examples, Pod 22A-1 to Pod 22A-N can be in the virtual network 450A.

[0224] According to the techniques of this disclosure, control node 432A executed in network cluster 370A may encode information of network service 458 using optional transitive BGP attributes. Control node 432A may generate advertisement 466 according to a protocol such as MP-BGP. Advertisement 466 may include network layer reachability information (e.g., routing traffic prefixes, labels, etc.) and a set of BGP communities or custom service BGP attributes, as previously described.

[0225] The control node 432A may send an announcement 466 to the control node 432B. The control node 432B may extract information of the network service 458 from one or more BGP attributes included in the announcement 466. In response to extracting the information of the network service 458 from the one or more BGP attributes, the control node 432B may call a client library (e.g., a Kubernetes client library) executed in the control node 432B. The control node 432B may use the client library to send a service catalog and an endpoint creation request to the configuration node 430B. The configuration node 430B may use an API (e.g., kube-api) to generate a network service catalog 462 based on the information of the network service 458 included in the announcement 466.

[0226] Network service catalog 462 may assign one or more virtual executives (e.g., Pod 22B-1 through Pod 22B-N) executing in network cluster 370B as endpoints of network service 458. In some examples, Pod 22B-1 through Pod 22B-N may be in virtual network 450B. Configuration node 430B may use network service catalog 462 to add Pod 22B-1 through Pod 22B-N (collectively referred to herein as "Pod 22B") of network cluster 370B as endpoints of network service 458 by assigning an IP address to Pod 22B based on information included in advertisement 466. Configuration node 430B may append the community indicated in advertisement 466 to the IP address assigned to Pod 22B added as an endpoint of network service 458. Configuration node 430B may then store the assigned IP address in virtual router 421 to support local direct communication (e.g., local load balancing supported in Kubernetes) without using a DNS server or service mesh.

[0227] In some cases, control node 432B may send a network service catalog and endpoint creation request with a routing policy that maps community attributes for routing decisions (e.g., acceptance, rejection, preference, or redistribution of network traffic). Control node 432 may create a routing policy that is implemented by configuration node 430B. The routing policy may be applied to ingress and egress routes and evaluate network traffic for service 458 based on labels associated with BGP community attributes. For example, control node 432 may create a routing policy to evaluate network traffic routes for service 458 based on matching conditions of import and export statements defined in the routing policy.

[0228] exist Figure 4In an example of , configuration node 430 can implement a routing policy (linking routing tables associated with virtual routers 421A and 421B to create a virtual network service cluster 426). Configuration node 430 can establish virtual network service cluster 426 by linking a virtual routing and forwarding (VRF) table of virtual network 450A stored in virtual router 421A to a VRF table of virtual network 450B stored in virtual router 421B. Configuration node 430 links the VRF table of virtual network 450A to the VRF table of virtual network 450B by including VRF routing instances of virtual network 450A and virtual network 450B in each of virtual router 421A and virtual router 421B. Virtual router 421A and virtual router 421B can then be configured to initiate an Internet Protocol (IP) session between an endpoint in virtual network 450A and an endpoint in virtual network 450B. In some examples, virtual network service cluster 426 can be established by the same protocol (MP-BGP) as used to generate announcement 466. In these examples, once Pods 22B-1 through 22B-N are added as endpoints of network service 458, control nodes 432A and 432B may distribute labeled network traffic routes (e.g., via MP-BGP) of network applications exposed by network service 458 to each other according to, for example, a load balancing policy implemented by network service 458. When control node 432A or 432B receives the labeled network traffic routes, control node 432A or 432B may further distribute the labeled network traffic routes to virtual routers 421A or 421B, respectively, or to a route reflector, for example, via MP-IBGP. Virtual router 421A or 421B may appropriately send network traffic associated with the labeled network traffic routes to service endpoint 458 (e.g., Pod 22A or Pod 22B) based on the VRF tables stored in virtual routers 421A and 421B. Virtual routers 421A and 421B may be configured to support next generation Layer 3 virtual networks, for example, by configuring virtual router 421 with next generation Layer 3 VPN option A, B, or C. For example, virtual router 421 may be configured to maintain network traffic routes for service 458 in a routing information base (RIB) and IP prefixes of endpoints (including endpoints assigned by service directory 462) assigned in a forwarding information base (FIB) with next generation Layer 3 VPN option B. Each of virtual routers 421 may associate network traffic routes for service 458 stored in the RIB with endpoint floating IP prefixes stored in the FIB to send or receive network traffic for service 458 to endpoints of service 458 according to an implemented routing protocol.

[0229] Figure 5is a block diagram illustrating an example network according to one or more aspects of the present disclosure. Figure 5 In the example of FIG. 4 , BGP controller 472 and mapping controller 474 may be one or more network controllers (eg, Figure 1 Network controller 24 or Figure 3 BGP controller 472 and mapping controller 474 may be deployed in a single network cluster (e.g., network cluster 370A) to generate, receive, or perform other networking actions based on advertisements 466 sent from BGP controllers deployed in remote network clusters (e.g., network clusters 370B to 370N).

[0230] In some examples, network clusters 370B through 370N may generate and send advertisements 466A through 466N, respectively (collectively, advertisements 466). Advertisements 466 may include information with network services (e.g., Figure 4 In some examples, advertisement 466 may include a tag or label indicating a community (e.g., a BGP extended community) of which the network cluster generating the advertisement is a member. For example, BGP controller 472 may determine whether advertisement 466 includes a tag or label indicating a community of which network cluster 370A is a member. In response to BGP controller 472 determining that the notification of advertisement 466 includes a tag or label indicating a community of which network cluster 370A is not a member, BGP controller 472 may ignore the advertisement. In response to BGP controller 472 determining that the notification of advertisement 466 includes a tag or label identifying a community of which network cluster 370A is not a member, BGP controller 472 may ignore the advertisement. In response to BGP controller 472 determining that the notification of advertisement 466 includes a tag or label identifying a community of which network cluster 370A is a member, BGP controller 472 may send a message to a configuration node (e.g., Figure 1 The configuration node 30 of the BGP controller 472 may send an instruction to create the network service directory 462. The BGP controller 472 may instruct the configuration node to instantiate the network service directory 462 with the network service information included in the set of BGP communities encoded in the advertisement in the advertisement 466.

[0231] BGP controller 472 may also send instructions to the configuration node to assign virtual executives (e.g., Pod 22A to Pod 22N) as endpoints of a network service executed in a remote network cluster (e.g., network clusters 370B to 370N) that sent the advertisement of advertisement 466. BGP controller 472 may also instruct mapping controller 474 to select which virtual executives (e.g., any of Pods 22A to 22N) are to be added as endpoints of a network service associated with the advertisement of advertisement 466. Mapping controller 474 may assign IP addresses to one or more virtual executives (e.g., Pod 22A to Pod 22N) based on information of the network service included in a set of BGP communities and used to instantiate network service catalog 462. Mapping controller 474 may also store the assigned IP addresses in a routing table of virtual router 421A to enable communication between all endpoints of the network service associated with the advertisement.

[0232] Figure 6 is a conceptual diagram illustrating an example advertisement 466 with example network layer reachability information 468 and a set of BGP communities according to one or more aspects of the present disclosure. In some examples, Figure 6 Network layer reachability information (NLRI) 468 and a set of BGP communities shown in FIG. 464 may be included in advertisement 466. Advertisement 466 may be sent by a BGP controller (e.g., Figure 5 BGP controller 472), control nodes (e.g., Figure 1 control node 32) or network controller ( Figure 1 The announcement 466 may be generated by a network controller 24. The announcement 466 may be based on an established protocol, such as MP-BGP.

[0233] According to the techniques of the present disclosure, advertisement 466 may include an address family identifier, a subsequent address family identifier, the length of the next hop network address, the next hop network address, a reserved portion, and NLRI 468 and BGP community 480. NLRI 468 is a variable length field that lists network layer reachability information of the feasible routes being advertised. NLRI 468 may have semantics identified by a combination of the address family identifier field and the subsequent address family identifier field included in advertisement 466.

[0234] The NLRI 468 may include a length field 492, which may indicate the length (in bits) of the address prefix in addition to the one or more labels. The NLRI 468 may include a label field 494, which may include one or more labels encoded as three octets (e.g., a label stack as defined in MPLS-ENCAPS). The NLRI 468 may include a prefix field 490, which may include the address prefix, followed by enough trailing bits to cause the end of the field to fall on an octet boundary.

[0235] BGP community 480 can be encoded with the information of network service to provide BGP peers that receive the notification 466 context of network service, so as to subsequently add remote virtual execution elements as the endpoints of network service. BGP community 480 can, for example, include a set of BGP extended communities. BGP extended community can include eight octets, which is more than the four octets included in the conventional BGP community. The extra octets included in the BGP extended community provide expansion and flexibility to adapt to scalable networking needs. BGP extended community may also have different formats. For example, BGP extended community can be formatted with two octets of type field and six octets of value field. In some cases, four octets of the value field of BGP extended community can store any 32-bit value. The technology described herein can encode the information of network service in any 32-bit value of the octets in the value field of BGP extended community. Generally, the network controller can receive BGP community 480 in any order. To address this issue, the techniques described herein include appropriate index values ​​in each BGP community of a set of BGP communities 480 to allow a network controller to properly process information of a network service intended to be advertised with an announcement 466 .

[0236] BGP community 480 may include a service identifier BGP community 482. In some cases, depending on the format of the BGP community (e.g., the BGP extended community format described above), the service identifier BGP community 482 may be up to 32 bits. The service identifier BGP community 482 may allocate two bits for the type field, indicating that the service identifier BGP community 482 is a service identifier. The service identifier BGP community 482 may also include up to 30 bits, which may be used to uniquely identify the network service identifier associated with the BGP community 482. The service identifier BGP community 482 may only be required once per routing prefix.

[0237] BGP community 480 may also include port BGP community 484. In some cases, depending on the format of the BGP community (e.g., the BGP extended community format described above), port BGP community 484 may be up to 32 bits. Port BGP community 484 may allocate two bits for the type field, indicating that port BGP community 484 is to identify the port of the network service. Port BGP community 484 may allocate eight bits for the service index field, indicating that port BGP community 484 is associated with service identifier BGP community 482. Port BGP community 484 may allocate up to 22 bits to identify the port associated with the network service intended to be announced with announcement 466. In some examples, port BGP community 484 may only need to allocate 16 bits to properly identify the port associated with the network service. Port BGP community 484 may be required for each route associated with prefix 490.

[0238] BGP community 480 may also include protocol BGP community 486. Protocol BGP community 486 may allocate two bits for the type field, indicating that protocol BGP community 486 is to identify the protocol of the network service. Protocol BGP community 486 may allocate eight bits for the port index field, indicating that protocol BGP community 486 is associated with port BGP community 484. Protocol BGP community 486 may allocate up to 22 bits to identify the protocol associated with the network service intended to be announced with announcement 466. In some examples, protocol BGP community 486 may only need to allocate eight bits to properly identify the protocol associated with the network service. Each route associated with prefix 490 may require protocol BGP community 484.

[0239] The BGP communities 480 may also include FQDN BGP communities 488A-488N. Each FQDN BGP community 488A-488N may allocate two bits for a type field, indicating that each FQDN BGP community 488A-488N is to identify a fully qualified domain name of a network service. Each FQDN BGP community 488A-488N may allocate eight bits for a port index field, indicating that each FQDN BGP community 488A-488N is associated with a port BGP community 484. Each FQDN BGP community 488A-488N may allocate at least three bits for a name index field, indicating the order in which each FQDN BGP community 488A-488N should be processed. For example, FQDN BGP community 488A may be assigned name index field 0, indicating that FQDN BGP community 488A includes a first string of FQDNs of the service, FQDN BGP community 488B may be assigned name index field 1, indicating that FQDN BGP community 488B includes a second string of FQDNs of the service, and so on. Each FQDN BGP community 488A-488N may be assigned up to 19 bits to identify a corresponding portion of the FQDN associated with the network service to be announced with announcement 466. In some examples, each FQDN BGP community 488A-488N may only need to be assigned 18 bits to appropriately identify the protocol associated with the network service. Each FQDN BGP community 488A-488N may be required for each route associated with prefix 490.

[0240] Figure 7 is a flow chart illustrating an exemplary process for assigning virtual executive elements of a remote network cluster as endpoints of a network service having a routing protocol in accordance with the techniques of this disclosure. Figure 7 and Figures 1 to 6 The discussion together is for example purposes only.

[0241] The network controller 24 (or any other network controller described herein, such as the network controller 324A) may encode one or more attributes (e.g., BGP communities 480 or custom BGP service attributes) to include information of the network service 58 (702). The network controller 24 may encode the BGP communities 480 or custom BGP service attributes in a manner that complies with the BGP routing protocol. The network controller 24 may generate an advertisement 466 (704) in the first network cluster 170A that complies with the routing protocol (e.g., MP-BGP). The advertisement 466 may include network layer reachability information (e.g., NLRI 468) with a prefix (e.g., prefix 490) associated with the route. The advertisement 466 may also include the BGP communities 480 or custom BGP service attributes that have been encoded with information of the network service 58. The first network cluster 170A may be executed within a container orchestration platform of a software defined network (SDN). The network service 58 may expose the back end of a network application (e.g., application 246) to the container orchestration platform of the SDN. The network controller 24 may also broadcast the announcement 466 to the second network cluster 170B according to the routing protocol ( 706 ). The second network cluster 170B may also be executed in the container orchestration platform of the SDN.

[0242] The network controller 24 may extract information identifying the network service 58 encoded in one or more attributes (e.g., BGP community 480 or custom BGP service attributes) (708). The network controller 24 may generate a network service catalog 62 based on the information included in the one or more attributes for execution in the second network cluster 170B (710). The network controller 24 may add one or more virtual execution elements (e.g., pod 250X or VM 252X) executed in the second network cluster 170B as endpoints (e.g., endpoint 351) of the network service 58 (712). Endpoint 351 may run a network application (e.g., application 246) that the network service 58 exposes to the container orchestration platform. In some examples, the network service 58 may use the network controller 24 to implement a network policy (e.g., a network traffic load balancing policy or a network firewall policy) to forward network traffic between endpoints of the network service 58, with one or more virtual routers executing in the SDN.

[0243] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. Describing different features as modules or units is intended to highlight different functional aspects and does not necessarily mean that these modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with one or more modules or units may be performed by separate hardware or software components, or integrated in common or separate hardware or software components.

[0244] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium containing instructions, such as a computer-readable storage medium. The instructions embedded or encoded in the computer-readable storage medium may cause a programmable processor or other processor to perform the method, for example, when the instructions are executed. The computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, a magnetic medium, an optical medium, or other computer-readable medium.

Claims

1. A computer network system, comprising a processing circuit capable of accessing a storage device, the processing circuit being configured to: A first network controller executed in a software defined network (SDN) encodes one or more attributes using information identifying a network service, wherein: The one or more attributes are consistent with a routing protocol; Generating, by the first network controller, an advertisement in a first network cluster executed within a container orchestration platform of the SDN, wherein the advertisement complies with the routing protocol and includes the one or more attributes; Broadcasting the notification by the first network controller to a second network cluster executed in the container orchestration platform of the SDN according to the routing protocol; extracting, by a second network controller executed in the SDN, the information identifying the network service by processing the one or more attributes; generating, by the second network controller, a network service directory in the second network cluster; The second network controller uses the network service catalog to add one or more virtual execution elements executed on the second network cluster as endpoints of the network service; and A network policy is implemented to forward network traffic between endpoints of the network service by one or more virtual routers executing in the SDN.

2. The computer network system according to claim 1, wherein: The processing circuitry is further configured to implement a network traffic load balancing network policy to forward network traffic between endpoints of the network service.

3. The computer network system according to claim 2, wherein: The one or more virtual routers include a first virtual router executed in the first network cluster and a second virtual router executed in the second network cluster, and wherein the processing circuit is further configured to: linking a first routing table of the first virtual router to a second routing table of the second virtual router to enable communication between endpoints of the network service; and A load balancer is implemented to regulate network traffic between endpoints of the network service.

4. The computer network system according to claim 1, wherein: The processing circuit is further configured to send the advertisement using a Border Gateway Protocol (BGP) controller executed in the first network cluster.

5. The computer network system according to claim 1, wherein: The first network controller comprises a first control node of a network controller, and wherein the second network controller comprises a second control node of the network controller.

6. The computer network system according to claim 1, wherein: The routing protocol includes the Multi-Protocol Border Gateway Protocol MP-BGP.

7. The computer network system according to claim 1, wherein: The processing circuit is further configured to encode the one or more attributes to include a fully qualified domain name (FQDN) of the network service, a protocol used by the network service, and a port used by the network service.

8. The computer network system according to any one of claims 1 to 7, wherein: The one or more attributes include a set of Border Gateway Protocol (BGP) extended communities.

9. The computer network system according to claim 8, wherein: Each BGP extended community in the set of BGP extended communities includes an index to the network service or an index to other BGP extended communities in the set of BGP extended communities.

10. The computer network system according to any one of claims 1 to 7, wherein: The one or more attributes include a custom service Border Gateway Protocol (BGP) attribute.

11. A computer networking method, comprising: Encoding, by a first network controller executed in a software defined network (SDN), one or more attributes using information identifying a network service, wherein the one or more attributes conform to a routing protocol; Generating, by the first network controller, an advertisement in a first network cluster executed within a container orchestration platform of the SDN, wherein the advertisement complies with the routing protocol and includes the one or more attributes; Broadcasting the notification by the first network controller to a second network cluster executed in the container orchestration platform of the SDN according to the routing protocol; extracting, by a second network controller executed in the SDN, the information identifying the network service by processing the one or more attributes; generating, by the second network controller, a network service directory in the second network cluster; The second network controller uses the network service catalog to add one or more virtual execution elements executed on the second network cluster as endpoints of the network service; and A network policy is implemented to forward network traffic between endpoints of the network service by one or more virtual routers executing in the SDN.

12. The computer networking method of claim 11, further comprising implementing a network traffic load balancing network policy to forward network traffic between endpoints of the network service.

13. The computer networking method according to claim 12, wherein: The one or more virtual routers include a first virtual router executed in the first network cluster and a second virtual router executed in the second network cluster, and wherein the first network controller or the second network controller is further configured to: linking a first routing table of the first virtual router to a second routing table of the second virtual router to enable communication between endpoints of the network service; and A load balancer is implemented to regulate network traffic between endpoints of the network service.

14. The computer networking method according to claim 11, further comprising: The advertisement is sent using a Border Gateway Protocol (BGP) controller executed in the first network cluster.

15. The computer networking method according to claim 11, wherein: The routing protocol includes the Multi-Protocol Border Gateway Protocol MP-BGP.

16. The computer networking method according to claim 11, wherein: Encoding the one or more attributes with the information identifying the network service includes encoding the one or more attributes to include a fully qualified domain name (FQDN) of the network service, a protocol used by the network service, and a port used by the network service.

17. The computer networking method according to any one of claims 11 to 16, wherein: The one or more attributes include a set of Border Gateway Protocol (BGP) extended communities.

18. The computer networking method according to claim 17, wherein: Each BGP extended community in the set of BGP extended communities includes an index to the network service or an index to other BGP extended communities in the set of BGP extended communities.

19. The computer networking method according to any one of claims 11 to 16, wherein: The one or more attributes include a custom service Border Gateway Protocol (BGP) attribute.

20. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to be configured to execute the computer networking method according to any one of claims 11-19 or to be configured as the computer network system according to any one of claims 1-10.

Images