Deployment check for containerized SDN architecture system

Abstract

In general, techniques are described for performing a pre-deployment check to ensure that a computing environment is properly configured for deploying a containerized software defined network (SDN) architecture system, and for performing a post-deployment check to determine an operational state of the containerized SDN architecture system after deployment to the computing environment.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 376,058, filed September 16, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to virtualized computing infrastructure and, more particularly, to the deployment of containerized workloads. Background Art

[0003] In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and / or storage capacity to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as network and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure that spans the globe, utilizing user support equipment located in various physical hosting facilities.

[0004] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have a widely used virtualization environment in which virtual hosts (also referred to herein as virtual execution elements) such as virtual machines or containers are deployed and run on the underlying computing platforms of physical computing devices.

[0005] Virtualization within a data center, or any environment that includes one or more servers, can provide several advantages. One advantage is that virtualization can provide significant improvements in efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, virtualization becomes easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources in, for example, cloud-based computing environments become fungible resources, the provision and management of computing infrastructure becomes easier. As a result, enterprise IT staff often prefer virtualized computing clusters in data centers for their management advantages, in addition to the efficiency and increased return on investment (ROI) that virtualization provides.

[0006] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight, portable execution elements for applications that are isolated from each other and the host. Because containers are not tightly coupled to the host hardware computing environment, applications can be bundled into container images and run as a single, lightweight package on any host or virtual host that supports the underlying container architecture. In this way, containers solve the problem of making software work across diverse computing environments. Containers guarantee consistent operation from one computing environment to another, virtual or physical.

[0007] Due to the inherent lightweight nature of containers, a single host can typically support many more container instances than traditional virtual machines (VMs). Short-lived containers can generally be created and moved more efficiently than VMs, and can also be managed as groups of logically related elements (for some orchestration platforms, such as Kubernetes, such groups are sometimes called "pods"). These container characteristics affect the requirements for container networking solutions: the network should be agile and scalable. In the same computing environment, VMs, containers, and bare metal servers may need to coexist and enable communication between multiple deployments of applications. Container networking should also not be known to work with multiple types of orchestration platforms used to deploy containerized network architectures.

[0008] Managing the deployed computing infrastructure and the infrastructure used for application execution can involve two main roles: (1) orchestration—for automating the deployment, scaling, and operation of applications across a cluster of hosts and providing computing infrastructure, which can include container-centric computing infrastructure; and (2) network management—for creating virtual networks in the network infrastructure to enable packetized communication between applications running on virtual execution environments such as containers or VMs and between applications running in traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the Invention

[0009] In general, techniques are described for performing pre-deployment checks (also referred to as "pre-flight checks" or "pre-flight tests") to ensure that a computing environment is properly configured for deploying a containerized software-defined networking (SDN) architecture system, as well as techniques for performing post-deployment checks (also referred to as "post-flight checks" or "post-flight tests") to determine the operational state of the containerized SDN architecture system after deployment to the computing environment. A containerized SDN architecture system (alternatively, an "SDN architecture" or "cloud-native SDN architecture") for managing and implementing application networking can be deployed to a computing environment. In some examples, the SDN architecture can include data plane elements and network devices (such as routers or switches) implemented in computing nodes, and the SDN architecture can also include a containerized network controller for creating and managing virtual networks. In some examples, the SDN architecture configuration and control plane are designed as scale-out cloud-native software with containerized applications. However, not all elements of the SDN architecture described herein need to be containerized.

[0010] In some aspects, pre-deployment checks and post-deployment checks can be implemented using custom resources of a container orchestration system (also known as a "container orchestrator"). These custom resources can include resources for executing a test suite, and can also include custom resources for executing individual tests of a test suite. Custom resources can be merged with native / built-in Kubernetes resources.

[0011] The techniques described in the present disclosure may have one or more technical advantages that enable at least one practical application. For example, pre-deployment checks and post-deployment checks may ensure that the computing and network environment can successfully execute the workloads that implement the network controller and network data plane, and enable network connectivity between applications deployed to the computing environment (which may themselves be deployed as containerized workloads). By leveraging a container orchestration framework with custom resources, these techniques may use a common approach to ensure the suitability of the computing infrastructure for deploying the network controller and network data plane, and the operability of the network controller and network data plane when deployed to configure network connectivity between workloads in the computing infrastructure. In some examples, the customizable specifications and container images used may allow users to perform custom tests "on the fly" without requiring the vendor of the network controller to release new code versions to support the custom tests.

[0012] In an example, a system includes: a plurality of servers; and a container orchestrator executed on at least one of the plurality of servers and configured to: create a ready custom resource in the container orchestrator, the ready custom resource configured to receive a specification specifying one or more tests for a software-defined network (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; create a ready-test custom resource in the container orchestrator for each of the one or more tests; deploy a corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; set a status for the ready custom resource based on a corresponding status output by the corresponding container image for the one or more tests; and deploy a workload to at least one of the plurality of servers based on the status for the ready custom resource indicating success, wherein the workload implements: a component of the SDN architecture system, or an application that requires network configuration of the workload through the SDN architecture system.

[0013] In an example, a method includes: creating a ready custom resource in a container orchestrator executed on at least one server among a plurality of servers, the ready custom resource configured to receive a specification specifying one or more tests for a software defined networking (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; creating a ready test custom resource in the container orchestrator for each of the one or more tests; deploying the corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; setting a status for the ready custom resource based on a corresponding status output by the corresponding container image for the one or more tests; and deploying a workload to at least one of the plurality of servers based on the status for the ready custom resource indicating success, wherein the workload implements: a component of the SDN architecture system, or at least one of an application requiring network configuration of the workload by the SDN architecture system.

[0014] In an example, a non-transitory computer-readable medium includes instructions that, when executed by a processing circuit, cause the processing circuit to: create a ready custom resource in a container orchestrator executed on at least one server among a plurality of servers, the ready custom resource being configured to receive a specification specifying one or more tests for a software-defined networking (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image being configured to implement the test on the server and output a status for the test; create a ready-test custom resource in the container orchestrator for each of the one or more tests; deploy the corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; set a status for the ready custom resource based on a corresponding status output by the corresponding container image for the one or more tests; and deploy a workload to at least one of the plurality of servers based on the status for the ready custom resource indicating success, wherein the workload implements: at least one of a component of the SDN architecture system or an application requiring network configuration of the workload by the SDN architecture system.

[0015] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram illustrating an exemplary computing infrastructure in which examples of the techniques described herein may be implemented.

[0017] Figures 2A to 2C is a block diagram illustrating different resource states of a computing infrastructure before and after deployment of a containerized SDN architecture system according to the techniques of this disclosure.

[0018] Figure 3 is a block diagram illustrating another view of components of a container orchestrator in accordance with the techniques of this disclosure in greater detail.

[0019] Figure 4 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure.

[0020] Figure 5 is a block diagram of an exemplary computing device operating as a compute node for one or more clusters of a containerized SDN architecture system in accordance with the techniques of this disclosure.

[0021] Figure 6 is a block diagram illustrating an example of a custom controller for custom resources according to the techniques of this disclosure.

[0022] Figure 7A and Figure 7B is a flow chart illustrating example operations for performing pre-deployment and post-deployment checks on a containerized SDN architecture system in accordance with the techniques of this disclosure.

[0023] Figure 8 is a block diagram illustrating a server implementing a containerized network router, to which one or more techniques of this disclosure may be applied.

[0024] Figure 9 is a flowchart illustrating an exemplary mode of operation of a computing system for implementing an SDN architecture system according to the techniques of this disclosure.

[0025] Like reference numerals denote like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0026] Figure 1 is a block diagram illustrating an exemplary computing system 8 in which examples of the techniques described herein may be implemented. Figure 1 In the example shown in , computing system 8 supports a software-defined network (SDN) architecture for virtual and physical networks. However, the techniques described herein can be readily applied to other computing infrastructures and software architectures.

[0027] exist Figure 1 In the example shown in , the computing system 8 includes a cloud native SDN architecture system 200 ("SDN architecture 200"). Exemplary use cases for the cloud native SDN architecture include 5G mobile networks, cloud and enterprise cloud native use cases, distributed application deployment, etc. The SDN architecture may include data plane elements (e.g., servers 12) and network devices (such as routers or switches) implemented in computing nodes, and the SDN architecture 200 includes an SDN controller (e.g., a network controller 24) for creating and managing virtual networks. The SDN architecture configuration and control plane are designed as scale-out cloud native software with a container-based microservices architecture. As described in detail below, the network controller 24 of the SDN architecture system configures the network configuration of the workload deployed to the server 12, including the virtual network interfaces, and programs routing information into the virtual router 21 to implement the virtual network.

[0028] Therefore, the SDN architecture 200 components can be microservices, and compared to existing network controllers, the network controller 24 of the SDN architecture system 200 adopts a basic container orchestration platform (e.g., orchestrator 23) to manage the lifecycle of the SDN architecture components. The container orchestration platform is used to build the SDN architecture 200 components; the SDN architecture uses cloud-native monitoring tools that can be integrated with customer-provided cloud-native options; the SDN architecture uses an aggregated API to provide a declarative resource management method for SDN architecture objects (i.e., custom resources). SDN architecture upgrades can follow cloud-native patterns, and the SDN architecture can utilize Kubernetes constructs such as Multus, Authentication & Authorization, Cluster API, KubeFederation, KubeVirtualt, and Kata containers. The SDN architecture can support a data plane development kit (DPDK) container pool, and the SDN architecture can be extended to support Kubernetes with virtual network policies and global security policies.

[0029] For service providers and enterprises, the SDN architecture automates the provisioning and orchestration of network resources to dynamically create highly scalable virtual networks and link virtualized network functions (VNFs) and physical network functions (PNFs) to form differentiated service chains on demand. The SDN architecture can be integrated with orchestration platforms (e.g., orchestrators 23) such as Kubernetes, OpenShift, Mesos, OpenStack, and VMware vSphere, as well as service provider operations support systems / business support systems (OSS / BSS).

[0030] Typically, one or more data centers 10 provide an operating environment for applications and services at a customer site 11 (shown as "Customer 11"), which has one or more customer networks coupled to the data center via a service provider network 7. For example, each of the data centers 10 may host infrastructure equipment such as network and storage systems, redundant power supplies, and environmental controls. The service provider network 7 is coupled to a public network 15, which may represent one or more networks managed by other providers and, thus, may form part of a larger public network infrastructure (e.g., the Internet). For example, the public network 15 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a Layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet, or some combination thereof, operated by a service provider operating the service provider network 7, an enterprise IP network, or some combination thereof.

[0031] Although customer sites 11 and public network 15 are primarily shown and described as edge networks of service provider network 7, in some examples, one or more of customer sites 11 and public network 15 may be tenant networks of any of data centers 10. For example, data center 10 may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), where each tenant may implement one of customer sites 11.

[0032] The service provider network 7 provides packet-based connectivity to attached customer sites 11, data centers 10, and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement Multi-Protocol Label Switching (MPLS) forwarding and, in such instances, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0033] In some examples, each of data centers 10 may represent one of many geographically distributed network data centers that may be connected to each other via service provider network 7, dedicated network links, dark fiber, or other connections. Figure 1 As shown in the example of , data center 10 may include facilities that provide network services to customers. Customers of the service provider may be collective entities such as businesses and governments or individuals. For example, a network data center may host network services for several businesses and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or supercomputing, etc. Although shown as a separate edge network of the service provider network 7, elements of the data center 10 (such as one or more physical network functions (PNFs) or virtualized network functions (VNFs)) may be included within the core of the service provider network 7.

[0034] In this example, data center 10 includes storage and / or compute servers (or "nodes") interconnected via a switching fabric 14 provided by one or more layers of physical network switches and routers, with servers 12A through 12X (referred to herein as "servers 12") being depicted as coupled to top-of-rack switches 16A through 16N. Servers 12 are computing devices and may also be referred to herein as "compute nodes," "hosts," or "host devices." Although Figure 1 Only server 12A coupled to TOR switch 16A is shown in detail, but data center 10 may include many additional servers coupled to other TOR switches 16 in data center 10.

[0035] In the example shown, the switch fabric 14 includes interconnected top-of-rack (TOR) (or other "leaf-level") switches 16A through 16N (collectively, "TOR switches 16") coupled to a distribution layer of chassis-style (or "backbone-level" or "core-level") switches 18A through 18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 10 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), mobile core network elements, and other PNFs.

[0036] In this example, the TOR switches 16 and the chassis switches 18 provide redundant (multi-homed) connectivity to the servers 12 to the IP fabric 20 and the service provider network 7. The chassis switches 18 aggregate traffic flows and provide connectivity between the TOR switches 16. The TOR switches 16 can be network devices that provide Layer 2 (MAC) and / or Layer 3 (e.g., IP) routing and / or switching functionality. The TOR switches 16 and the chassis switches 18 can each include one or more processors and memories and can execute one or more software processes. The chassis switches 18 are coupled to the IP fabric 20, which can perform Layer 3 routing to route network traffic between the data center 10 and the customer sites 11 through the service provider network 7. The switching architecture of the data center 10 is merely an example. For example, other switching architectures can have more or fewer switching layers. The IP fabric 20 can include one or more gateway routers.

[0037] The term "packet flow," "traffic flow," or simply "flow" refers to a collection of packets originating from a specific source device or endpoint and being sent to a specific destination device or endpoint. For example, a single flow of packets can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, this 5-tuple identifies the packet flow corresponding to the received packets. An n-tuple refers to any n items extracted from a 5-tuple. For example, a 2-tuple for a packet can refer to the combination of <source network address, destination network address> or <source network address, source port> for the packet.

[0038] Servers 12 may each represent a computing server or a storage server. For example, each of servers 12 may represent a computing device configured to operate according to the techniques described herein, such as an x86 processor-based server. Servers 12 may provide a network function virtualization infrastructure (NFVI) for an NFV architecture.

[0039] Any of the servers 12 can be configured with virtual execution elements, such as container pools or virtual machines, by virtualizing the server's resources to provide a degree of isolation between one or more processes (applications) executing on the server. "Hypervisor-based," or "hardware-level," or "platform" virtualization refers to the creation of virtual machines that each include a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is visualized from the physical hardware of the host server, the executing application is isolated from both the host's hardware and the other virtual machines. Each virtual machine can be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0040] A virtual network is a logical construct implemented on top of a physical network. Virtual networks can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., data center 10). Each tenant or application can have one or more virtual networks. Each virtual network can be isolated from all other virtual networks unless explicitly allowed by security policy.

[0041] Virtual networks can use the Data Center 10 Gateway Router ( Figure 1 The virtual network (not shown) is connected to and extends across physical Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Network (L3VPN) and Ethernet Virtual Private Network (EVPN) networks. The virtual network can also be used to implement Network Function Virtualization (NFV) and service chaining.

[0042] Virtual networks can be implemented using a variety of mechanisms. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. Virtual networks can also be implemented using two networks: a physical underlay network consisting of the IP fabric 20 and the switch fabric 14, and a virtual overlay network. The role of the physical underlay network is to provide an "IP fabric" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlay network can provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

[0043] As further described below with respect to virtual router 21A (illustrated as and also referred to herein as "vRouter 21A"), virtual routers 21A through 21X (collectively, "virtual routers 21") running in server 12 are components of the SDN architecture system and are used to create a virtual overlay network on top of a physical underlying network using a dynamic mesh of "tunnels" between them. For example, these overlay tunnels may be MPLS over GRE / UDP tunnels, or VXLAN tunnels, or NVGRE tunnels. For virtual machines or other virtual execution elements, the underlying physical routers and switches may not store any state for each tenant, such as any media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may contain only the IP prefix or MAC address of the physical server 12. (The gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0044] The virtual router 21 of a server 12 typically contains state for each tenant. For example, a virtual router may contain a separate forwarding table (routing instance) for each virtual network. This forwarding table contains the IP prefixes (in the case of a Layer 3 overlay) or MAC addresses (in the case of a Layer 2 overlay) of virtual machines or other virtual execution elements (e.g., a container pool of containers). A single virtual router 21 does not need to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 only needs to contain those routing instances that are local to the server 12 (i.e., it has at least one virtual execution element present on the server 12).

[0045] “Container-based” or “operating system” virtualization refers to the virtualization of an operating system to run multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers such as those provided by the open source DOCKER container application or by CoreOS Rkt (“Rocket”). Like virtual machines, each container is virtualized and can be kept isolated from the host machine and other containers. However, unlike virtual machines, each container can omit a single operating system and instead provide a suite of applications and application-specific libraries. Typically, containers are run by a host as isolated user-space instances and can share an operating system and common libraries with other containers executing on the host. As a result, containers may require less processing power, storage, and network resources than virtual machines. Groups of one or more containers can be configured to share one or more virtual network interfaces for communicating on corresponding virtual networks.

[0046] In some examples, containers are managed by their host kernel to allow, in some cases, resource limits and priorities (CPU, memory, block I / O, network, etc.) to be set without launching any virtual machines using namespace isolation capabilities, which allow applications (e.g., a given container) to have a completely isolated view of the operating environment, including process trees, networks, user identifiers, and mounted file systems. In some examples, containers can be deployed according to Linux Containers (LXC), an operating system-level virtualization method that uses a single Linux kernel to run multiple isolated Linux systems (containers) on a control host.

[0047] Servers 12 host virtual network endpoints of one or more virtual networks operating on top of a physical network represented here by IP fabric 20 and switch fabric 14. Although primarily described with respect to data center-based switched networks, other physical networks such as service provider network 7 may underlie one or more virtual networks.

[0048] Each of the servers 12 can host one or more virtual execution elements, each having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of a virtual network. For example, a virtual network endpoint can be a collection of virtual machines, one or more containers (e.g., a container pool), or another virtual execution element such as a layer 3 endpoint of a virtual network. The term "virtual execution element" covers virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" can also cover a container pool of one or more containers. A virtual execution element can represent an application workload. As Figure 1 As shown in FIG, server 12A hosts a virtual network endpoint in the form of a container pool 22 having one or more containers. However, given the hardware resource limitations of server 12, server 12 may execute as many virtual execution elements as are practical. Each of the virtual network endpoints may use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint may use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 13A to perform packet I / O and receive / send packets on one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

[0049] The servers 12 each include at least one network interface card (NIC) 13, each network interface card including at least one interface for exchanging packets with the TOR switch 16 via a communication link. For example, server 12A includes NIC 13A. Any of the NICs 13 can provide one or more virtual hardware components for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC ("physical function"). For example, in single root I / O virtualization (SR-IOV) described by the Peripheral Component Interface Special Interest Group SR-IOV specification, the PCIe physical function of a network interface card (or "network adapter") is virtualized to provide one or more virtual network interfaces as "virtual functions" for use by corresponding endpoints running on the server 12. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are examples of virtual hardware components. As another example, one or more servers 12 can implement Virtio (e.g., a paravirtualization framework available for the Linux operating system), which provides emulated NIC functionality as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement OpenvSwitch to perform distributed virtual multilayer switching between one or more virtual NICs (vNICs) of hosted virtual machines, where such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function. In some examples, any of the servers 12 may implement a Linux bridge that emulates a hardware bridge and forwards packets between the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For a Docker implementation of containers hosted by a server, a Linux bridge or other operating system bridge that executes on the server and exchanges packets between containers may be referred to as a "Docker bridge." As used herein, the term “virtual router” may encompass a Contrail or Tungsten Fabric virtual router, Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software residing on a host device that performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more of the servers 12.

[0050] Any of the NICs 13 may include an internal device switch that exchanges data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch may be a virtual Ethernet bridge (VEB) that switches between SR-IOV virtual functions and, correspondingly, between endpoints that are configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. Alternatively, the internal device switch may be referred to as a NIC switch, or, for an SR-IOV implementation, an SR-IOV NIC switch. The virtual hardware components associated with the NIC 13A may be associated with a layer 2 target address assigned by the NIC 13A or a software process responsible for configuring the NIC 13A. Physical hardware components (or "physical functions" for SR-IOV implementations) are also associated with a layer 2 target address.

[0051] One or more of the servers 12 may each include a virtual router 21 that executes one or more routing instances for a corresponding virtual network within the data center 10 to provide virtual network interfaces and route packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) of an Internet Protocol-Virtual Private Network (IP-VPN). For example, a packet received by the virtual router 21 of server 12A from the underlying physical network fabric of the data center 10 (i.e., IP fabric 20 and switch fabric 14) may include an outer header to allow the physical network fabric to tunnel the payload, or "inner packet," to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the server's network interface card 13A, but also a virtual network identifier, such as a VxLAN tag or a Multiprotocol Label Switching (MPLS) label, that identifies one of the virtual networks and the corresponding routing instance executed by the virtual router 21. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0052] Virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the tunnel encapsulation header for the packet, and forwards the packet to the appropriate target virtual network endpoint for the packet. For example, with respect to server 12A, for each packet outbound from a virtual network endpoint (e.g., container pool 22) hosted by server 12A, virtual router 21 appends a tunnel encapsulation header indicating the virtual network to the packet to generate an encapsulated or "tunnel" packet, and virtual router 21 outputs the encapsulated packet to a physical target computing device (such as another server in server 12) via the overlay tunnel for the virtual network. As used herein, virtual router 21 can perform the operations of a tunnel endpoint to encapsulate internal packets originating from a virtual network endpoint to generate tunnel packets, and decapsulate tunnel packets to obtain internal packets for routing to other virtual network endpoints.

[0053] In some examples, virtual router 21 may be kernel-based and executed as part of the kernel of the operating system of server 12A.

[0054] In some examples, virtual router 21 can be a virtual router that supports the Data Plane Development Kit (DPDK). In such an example, virtual router 21 uses DPDK as the data plane. In this mode, virtual router 21 runs as a user space application linked to the DPDK library (not shown). This is a performance version of a virtual router and is commonly used by telecommunications companies, where VNFs are typically DPDK-based applications. The performance of virtual router 21 as a DPDK virtual router can achieve a throughput ten times higher than that of a virtual router operating as a kernel-based virtual router. The physical interface is used by the DPDK's poll mode driver (PMD) instead of the Linux kernel's interrupt-based driver.

[0055] The User I / O (UIO) kernel module (such as vfio or uio_pci_generic) can be used to expose the registers of the physical network interface into user space so that they can be accessed through the DPDK PMD. When the NIC 13A is bound to the UIO driver, it is moved from the Linux kernel space to the user space and is therefore no longer managed or visible by the Linux OS. Therefore, the DPDK application (i.e., the virtual router 21A in this example) fully manages the NIC 13. This includes packet polling, packet processing, and packet forwarding. The user packet processing steps can be performed by the virtual router 21DPDK data plane, which is not in the kernel (the kernel is not in the Figure 1The "polling mode" nature of the virtual router 21 DPDK data plane packet processing / forwarding makes it more efficient than interrupt-based mode, especially at high packet rates. During packet I / O, there is limited or no interrupt and context switching.

[0056] The computing system 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration system that provides a container-centric infrastructure to automate the deployment, scaling, and operation of containers, thereby providing a container-centric infrastructure. In the context of a virtualized computing infrastructure, "orchestration" generally refers to the provision, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements by a host server available to the orchestration platform. Specifically, for example, container orchestration allows for container coordination and refers to the deployment, management, scaling, and configuration of containers to host servers by a container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes (container orchestration system), Docker swarm, Meso / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0057] Elements of the automation platform of computing system 8 include at least a server 12, an orchestrator 23, and a network controller 24. Containers can be deployed to a virtualized environment using a cluster-based framework in which a cluster master node of the cluster manages the deployment and operation of containers to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" as used herein encompass different orchestration platform terms that distinguish between the primary management element of the cluster and the analog device of the cluster's primary container hosting device. For example, the Kubernetes platform uses the terms "cluster master node" and "cluster slave node," while the Docker Swarm platform refers to the cluster manager and cluster nodes. The Kubernetes platform recently began using the term "control plane node" to refer to the virtual machine or physical machine that hosts the Kubernetes control plane, and "worker node" or "Kubernetes node" to refer to the virtual machine and physical machine that hosts containerized workloads in the cluster.

[0058] Orchestrator 23 and network controller 24 may run on separate sets of one or more computing devices, or on overlapping sets of one or more computing devices. Each of orchestrator 23 and network controller 24 may be a distributed application executed on one or more computing devices. Orchestrator 23 and network controller 24 may implement respective master nodes for one or more clusters, each of which has one or more slave nodes (also referred to as "compute nodes") implemented by respective servers 12.

[0059] Generally, network controller 24 controls the network configuration of the data center 10 fabric, for example, to establish one or more virtual networks for packetized communications between virtual network endpoints. Network controller 24 provides a logically and, in some cases, physically centralized controller for facilitating the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration input received from orchestrator 23 and / or an administrator / operator. Additional information regarding exemplary operation of the network controller 24 operating in conjunction with other devices of the data center 10 or other software-defined networks is found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS,” and in U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, and entitled “Tunneled Packet Aggregation for Virtual Networks,” each of which is incorporated herein by reference as if fully set forth herein.

[0060] Typically, the orchestrator 23 controls the deployment, scaling, and operation of containers across a cluster of servers 12 and provides computing infrastructure, which may include container-centric computing infrastructure. The orchestrator 23 and, in some cases, the network controller 24 may implement corresponding cluster master nodes for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud may provide virtualized infrastructure to the container management platform. Figure 3 Describe example components of the Kubernetes orchestration system.

[0061] Kubernetes operates using various Kubernetes objects (entities that represent the state of a Kubernetes cluster). Kubernetes objects can include any combination of names, namespaces, labels, annotations, field selectors, and recommended labels. For example, a Kubernetes cluster can include one or more "namespace" objects. Each namespace in a Kubernetes cluster is isolated from other namespaces in the Kubernetes cluster. Namespace objects can include at least one of the organization, security, and performance of the Kubernetes cluster. As an example, a container pool can be associated with a namespace, thereby associating the container pool with the characteristics of the namespace (e.g., a virtual network). This feature can organize multiple newly created container pools by associating them with a common set of characteristics. Namespaces can be created based on namespace specification data that defines the characteristics of the namespace (including the namespace name). In one example, a namespace may be named "Namespace A," and each newly created container pool can be associated with a set of characteristics represented by "Namespace A." In addition, Kubernetes includes a "default" namespace. If a newly created container pool does not specify a namespace, the newly created container pool can be associated with the characteristics of the "default" namespace.

[0062] Namespaces allow a single Kubernetes cluster to be used by multiple users, user teams, or a single user with multiple applications. Furthermore, each user, user team, or application can be isolated from other users of the cluster within a namespace. Thus, each user of a Kubernetes cluster within a namespace operates as if it were a unique user of the Kubernetes cluster. Multiple virtual networks can be associated with a single namespace. Thus, a container pool belonging to a specific namespace can access every virtual network associated with that namespace, including other container pools acting as virtual network endpoints of a virtual network group.

[0063] In one example, the container pool 22 is a Kubernetes container pool and is an example of a virtual network endpoint. A container pool is one or more logically related containers ( Figure 1), shared storage for containers, and a set of options for how to run containers. When instantiated for execution, a container pool may alternatively be referred to as a "container pool replica." Each container of a container pool 22 is an example of a virtual execution element. Containers of a container pool are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of a container pool can be a collection of Linux namespaces, cgroups, and other isolation aspects. Within the context of a container pool, individual applications can also apply sub-isolation. Typically, containers within a container pool have a common IP address and port space and are able to detect each other via the local host. Because they have a shared context, containers within a container pool also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different container pools have different IP addresses, and in the absence of a configuration to enable this feature, containers cannot communicate via IPC. Containers that are members of different container pools typically communicate with each other via the container pool IP address instead.

[0064] Server 12A includes a container platform 19 for running containerized applications, such as applications in container pool 22. Container platform 19 receives a request from orchestrator 23 to obtain and host a container in server 12A. Container platform 19 obtains and runs the container. In some examples, container platform 19 may be, for example, a Docker engine, CRI-O, containerd, or a Mirantis container runtime.

[0065] The container network interface (CNI) 17 configures virtual network interfaces for virtual network endpoints. The orchestrator 23 and the container platform 19 use the CNI 17 to manage networks for container pools, including container pool 22. For example, the CNI 17 creates virtual network interfaces to connect a container pool to a virtual router 21 and enables containers in such a container pool to communicate with other virtual network endpoints on the virtual network via the virtual network interfaces. For example, the CNI 17 may insert a virtual network interface for a virtual network into the network namespace of a container in container pool 22 and configure (or request configuration of) a virtual network interface for the virtual network in virtual router 21 so that virtual router 21 is configured to send packets received from the virtual network via the virtual network interface to containers in container pool 22, and to send packets received from containers in container pool 22 via the virtual network interface to the virtual network. The CNI 17 may assign network addresses (e.g., virtual IP addresses for the virtual networks) and set up routes for the virtual network interfaces. In Kubernetes, by default, all container pools can communicate with all other container pools without using network address translation (NAT). In some cases, the orchestrator 23 and network controller 24 create a service virtual network and a container pool virtual network shared by all namespaces, and allocate service and container pool network addresses from all namespaces, respectively. In some cases, all container pools in all namespaces derived in the Kubernetes cluster can communicate with each other, and network addresses for all container pools can be allocated from the container pool subnet specified by the orchestrator 23. When a user creates an isolated namespace for a container pool, the orchestrator 23 and network controller 24 can create a new container pool virtual network and a new shared service virtual network for the new isolated namespace. Container pools in the isolated namespace derived in the Kubernetes cluster draw network addresses from the new container pool virtual network, and the corresponding services of such container pools draw network addresses from the new service virtual network.

[0066] CNI 17 may represent a library, plugin, module, runtime, or other executable code for server 12A. CNI 17 may conform, at least in part, to the Container Network Interface (CNI) specification or the rkt networking proposal. CNI 17 may represent Contrail, OpenContrail, Multus, Calico, cRPD, or other CNIs. Alternatively, NIC 17 may be referred to as a network plugin, CNI plugin, or CNI instance. For example, a separate CNI may be called by the Multus CNI to establish different virtual network interfaces for container pool 22.

[0067] The CNI 17 can be called by the orchestrator 23. For the purposes of the CNI specification, containers can be considered to be synchronized with the Linux network namespace. The corresponding unit depends on the specific container runtime implementation: for example, in an implementation of the application container specification such as rkt, each container pool runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that can be uniquely addressed and can communicate with each other. This can be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers can be conceptually added to or removed from one or more networks. The CNI specification specifies a series of things that need to be considered by plugins that conform to the specification ("CNI plugins").

[0068] Container pool 22 includes one or more containers. In some examples, container pool 22 includes containerized DPDK workloads designed to use DPDK to accelerate packet processing, for example, by exchanging data with other components using DPDK libraries. In some examples, virtual router 21 can be executed as a containerized DPDK workload.

[0069] Container pool 22 is configured with a virtual network interface 26 for sending and receiving packets with virtual router 21. Virtual network interface 26 may be the default interface of container pool 22. Container pool 22 may implement virtual network interface 26 as an Ethernet interface (e.g., named "eth0"), while virtual router 21 may implement virtual network interface 26 as a tap interface, a virtio-user interface, or other types of interfaces.

[0070] Container pool 22 and virtual router 21 exchange data packets using virtual network interface 26. Virtual network interface 26 can be a DPDK interface. Container pool 22 and virtual router 21 can establish virtual network interface 26 using vhost. Container pool 22 can operate according to an aggregation model. Container pool 22 can use virtual devices, such as virtio devices with vhost user adapters, for communication between userspace container processes on virtual network interface 26.

[0071] CNI 17 can be combined with Figure 1 One or more other components shown in FIG configure a virtual network interface 26 for the container pool 22. Any container of the container pool 22 can utilize, ie, share, the virtual network interface 26 of the container pool 22.

[0072] Virtual network interface 26 may represent a virtual Ethernet ("veth") pair, where each end of the pair is a separate device (e.g., a Linux / Unix device), one end of the pair is assigned to container pool 22, and the other end of the pair is assigned to virtual router 21. Veth pairs or ends of veth pairs are sometimes referred to as "ports." A virtual network interface may represent a macvlan network with media access control (MAC) addresses assigned to container pool 22 and virtual router 21 for communication between containers of container pool 22 and virtual router 21. For example, a virtual network interface may alternatively be referred to as a virtual machine interface (VMI), a container pool interface, a container network interface, a tap interface, a veth interface, or simply a network interface (in certain contexts).

[0073] exist Figure 1 In the exemplary server 12A, container pool 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment, which specifies the virtual network and specifies that container pool 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. For example, orchestrator 23 may receive the configuration data from a user, an operator / administrator, or another machine system.

[0074] As part of the process of creating a container pool 22, the orchestrator 23 requests the network controller 24 to create corresponding virtual network interfaces for one or more virtual networks (indicated in the configuration data). The container pool 22 can be equipped with a different virtual network interface for each virtual network to which it belongs. For example, the virtual network interface 26 can be a virtual network interface for a specific virtual network. Additional virtual network interfaces (not shown) can be configured for other virtual networks. The network controller 24 processes the request to generate interface configuration data for the virtual network interfaces of the container pool 22. The interface configuration data may include a container unique identifier or a container pool unique identifier, and a list or other data structure that specifies network configuration data for configuring the virtual network interface for each virtual network interface. The network configuration data for the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and / or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

[0075] Network controller 24 sends the interface configuration data to server 12A, and more specifically, in some cases, to virtual router 21. To configure virtual network interfaces for container pool 22, orchestrator 23 may invoke CNI 17. CNI 17 obtains the interface configuration data from virtual router 21 and processes it. CNI 17 creates each virtual network interface specified in the interface configuration data. For example, CNI 17 may attach one end of a veth pair implementing management interface 26 to virtual router 21 and the other end of the same veth pair to container pool 22, which may implement it using virtio-user.

[0076] A traditional CNI plugin is invoked by a container platform / runtime to receive an Add command from the container platform to add a container to a single virtual network, and may subsequently invoke such a plugin to receive a Del(ete) command from the container / runtime and remove the container from the virtual network. The term "invoke" may refer to instantiating a software component or module in memory as executable code for execution by processing circuitry.

[0077] Network controller 24 is a cloud-native distributed network controller for software-defined networking (SDN), implemented using one or more configuration nodes 30 that implement a network configuration plane and one or more control nodes 32 that implement a network control plane. Each configuration node 30 itself can be implemented using one or more cloud-native component microservices. Each control node 32 itself can be implemented using one or more cloud-native component microservices. Each of control node 32 and configuration node 30 can be executed by a control plane node and / or a worker node of orchestrator 23. In this exemplary architecture, network controller 24 is conceptual and is generated by the operation of control node 32 and configuration node 30.

[0078] The network configuration plane (configuration node 30) interacts with the control plane components to manage network controller resources. The network configuration plane can use custom resource definitions (CRDs) to manage network controller resources. The network control plane (control node 32) provides core SDN capabilities. The control node 32 can use BGP to interact with peers (such as other control nodes 32 or other controllers and gateway routers). The control node 32 can use XMPP or other protocols to interact with and configure data plane components (such as virtual routers 21) using the configuration interface 256. The SDN architecture system 200 supports a centralized network control plane architecture in which the routing daemon runs centrally within the control node 32 and learns and distributes routes from and to the data plane components (vRouters 21) operating on the worker nodes (servers 12). This centralized architecture facilitates virtual network abstraction, orchestration, and automation.

[0079] The network data plane resides on all nodes and interacts with the containerized workloads to send and receive network traffic. The main component of the network data plane is the vRouter, shown as vRouter 21 in the corresponding server 12 (for ease of illustration, only vRouter 21A of server 12A is shown).

[0080] In some examples, configuration node 30 can be implemented by extending a native orchestration platform to support custom resources for an orchestration platform for software-defined networking, and more specifically, to provide a northbound interface to the orchestration platform to support intent-driven / declarative creation and management of virtual networks, for example, by configuring virtual network interfaces for virtual execution elements, configuring the underlying network connecting server 12, and configuring overlay routing functions, including overlay tunnels for virtual networks and overlay trees for multicast Layer 2 and Layer 3.

[0081] As Figure 1 As part of the SDN architecture shown in , the network controller 24 can be multi-tenant aware and support multi-tenancy for the orchestration platform. For example, the network controller 24 can support Kubernetes role-based access control (RBAC) constructs, local identity access management (IAM), and external IAM integration. The network controller 24 can also support Kubernetes-defined network structures and advanced network functions such as virtual networks, BGPaaS, network policies, service chains, and other telecommunications functions. The network controller 24 can support network isolation using virtual network constructs and support Layer 3 networks.

[0082] To interconnect multiple virtual networks, the network controller 24 can use (and configure in the underlying layer and / or virtual routers—vRouters—21) network policies, called virtual network policies (VPNs), and alternatively referred to herein as virtual network routers or virtual network topologies. VPNs define connectivity policies between virtual networks. A single network controller 24 can support multiple Kubernetes clusters, and thus VPNs allow multiple virtual networks to be connected in namespaces, Kubernetes clusters, and across Kubernetes clusters. VPNs can also be extended to support virtual network connectivity across multiple instances of the network controller 24.

[0083] The network controller 24 can enable multi-layer security using network policies. The default behavior of Kubernetes is for container pools to communicate with each other. To apply network security policies, the SDN architecture implemented by the network controller 24 and virtual router 21 can operate as a CNI for Kubernetes through CNI 17. For Layer 3, isolation occurs at the network level, and virtual networks operate at L3. Virtual networks are connected through policies. Kubernetes native network policies provide security at Layer 4. The SDN architecture can support Kubernetes network policies. Kubernetes network policies operate at Kubernetes namespace boundaries. The SDN architecture can add custom resources for enhanced network policies. The SDN architecture can support application-based security. (In some cases, these security policies can be based on meta-tags to apply granular security policies in an extensible manner.) For Layer 4+, the SDN architecture can, in some examples, support integration with containerized security appliances and / or Istio, and can provide encryption support.

[0084] As Figure 1 As part of the SDN architecture shown in , the network controller 24 can support multi-cluster deployments, which is important for telco cloud and high-end enterprise use cases. For example, the SDN architecture can support multiple Kubernetes clusters. The cluster API can be used to support lifecycle management of Kubernetes clusters. KubefedV2 can be used to federate configuration nodes 32 across Kubernetes clusters. The cluster API and KubefedV2 are optional components for supporting a single instance of the network controller 24 that supports multiple Kubernetes clusters.

[0085] The computing system 8 implements a cloud-native SDN architecture and can present various advantages. For example, the network controller 24 is a cloud-native, lightweight distributed application with a simplified installation space. This also facilitates easier and modular upgrades of the various component microservices of the configuration node 30 and the control node 32 (and any other components of other examples of the network controller described in this disclosure). These technologies can also enable optional cloud-native monitoring (telemetry) and user interfaces, a high-performance data plane for containers using DPDK-based virtual routers connected to a DPDK-enabled container pool, and cloud-native configuration management that in some cases leverages the configuration framework of existing orchestration platforms (such as Kubernetes or Openstack). As a cloud-native architecture, the network controller 24 is a scalable and elastic architecture to address and support multiple clusters. In some cases, the network controller 24 can also support scalability and performance requirements for key performance indicators (KPIs).

[0086] The SDN architecture may require sufficient computing resources and a correctly configured computing system 8 to successfully provide the above-mentioned capabilities, features and advantages. However, in view of the various features and capabilities that different tenants may require, the SDN architecture may be configured differently for different tenants. It may be difficult to determine whether the resources of the computing system 8 provided to the tenant are able to support the tenant's requirements. In addition, it may be difficult to determine whether the tenant or service provider has correctly configured the SDN architecture. In existing systems, a manual process can be used to determine whether there are sufficient resources within the computing system 8 and whether the SDN architecture has been correctly configured to use these resources. This manual process is time-consuming and prone to errors. As a technical advantage over such systems, the technology described herein provides automatic checks to determine whether the computing system 8 has appropriate and sufficient resources to support the SDN architecture before deploying the SDN architecture (pre-deployment check), and automatically checks whether the SDN architecture has been successfully configured and deployed (post-deployment check).

[0087] To this end, and in accordance with the techniques of this disclosure, orchestrator 23 is configured with readiness 62 and readiness-test 63. Readiness 62 defines custom resources for orchestrator 23 that represent the pre-deployment test suite and the post-deployment test suite as a whole. Readiness-test 63 defines custom resources for orchestrator 23 that represent individual pre-deployment and post-deployment tests. Custom resources can be KubernetesCustomResources created using custom resource definitions (CRDs).

[0088] In the example, a new container pool will be created on the host network (e.g., see Figures 2A to 2C ) with a custom controller that listens for events related to either ApplicationReadiness 62 or ApplicationReadinessTest 63. User input for the readiness test 63 can be included in the specification and can specify:

[0089] a. Test list

[0090] i. If the list is empty, no tests will be performed.

[0091] ii. The user can specify custom tests (if any) in this field, which can include a reference to a container image that will be run as a test container to perform the custom tests. (In some examples, the customizable specifications and container images used can allow users to perform custom tests "on the fly," without requiring the network controller vendor to release a new code version to support the custom tests.)

[0092] iii. Each test / container image should conform to standardized output forwarding so that custom controllers can parse the output to interpret the status of the test.

[0093] iv. A node selector for each entry, used to select the node on which to create the job.

[0094] b. The name of the configmap (in the default namespace) containing the installation manifest for the SDN architecture 200 , such as the network controller 24 nodes and the virtual router 21 .

[0095] i. The configmap name can be provided as a container environment variable to the running test container pool.

[0096] The custom controller will install a custom resource of type ApplicationReadinesTest for each test in the test suite (ReadinessTest 63). A job will be launched on the selected node for each test, which will run the test container inside the container pool. (If the custom controller determines that the configuration node 30 does not have an API server available, the container pool may set hostNetwork to true - in which case there is no CNI.) Each custom resource of type ReadinessTest 63 can have a comprehensive status field that will be updated during test execution. The custom controller will compile the final status of each test and set the status of the test suite CR, i.e., Ready 62, to the summary of each test. Reference Figures 2A to 2C An exemplary custom controller, ready controller 249, is described.

[0097] Readiness 62 can be initially run for pre-deployment to verify that the computing and network environment can successfully execute the workload implementing the network controller. If the summary of Readiness 62 for this initial run indicates success, this confirms that the environment can support the installation of the network controller 24 node and virtual router 21. The indication of failure may include a recommendation for reconfiguration of one or more components of the SDN architecture 200. In some cases, orchestrator 23 can automatically deploy the network controller 24 node and virtual router 21 based on the indication of success. Readiness 62 can then be run for a post-deployment check, utilizing different specifications and different types of Readiness Test 63 custom resources appropriate for the post-deployment check, to verify that the network controller 24 node and virtual router 21 can configure appropriate network connectivity between the workloads executing on the worker nodes (servers 12). This involves verifying the operational status of the network controller 24 node and virtual router 21, as well as the readiness of the worker nodes for the workloads. If the summary of Readiness 62 for this next run indicates success, the SDN architecture 200 is ready to support the deployment of workloads for the applications.

[0098] In this way, custom resources can be used to extend the existing container orchestration framework of the orchestrator 23 to use a common solution to ensure the suitability of the computing system 8 for deploying the network controller 24 and the network data plane, and the operability of the network controller 24 and the network data plane when deployed to configure network connectivity between workloads (e.g., the container pool 22) in the computing system 8.

[0099] Additional details of the above-mentioned SDN architecture can be found in U.S. patent application serial number 17 / 657,596, entitled “CLOUD NATIVE SOFTWARE-DEFINED NETWORK ARCHITECTURE,” filed on March 31, 2022, the entire contents of which are incorporated herein by reference.

[0100] Figures 2A to 2C is a block diagram showing different resource states of a computing system 8 before and after deploying a cloud native SDN architecture. Figures 2A to 2C In the example shown in FIG, network controller 24 is shown as being separate from servers 12A through 12X (collectively, “servers 12”), e.g., by executing on a different server than servers 12. However, network controller 24 may execute on one or more of servers 12 and need not be located on its own server.

[0101] Figure 2A 2 is a block diagram illustrating an initial state 201 of resources of computing system 8 according to the techniques of this disclosure. In this initial state, server 12 may be a bare metal server with minimal software installed on it. For example, server 12 may have an operating system and container platform 19 (or "container runtime") installed. Network controller 24 may be implemented on a server separate from server 12, which may have an operating system and container platform 19 installed.

[0102] Figure 2B is a block diagram illustrating a pre-deployment test state 202 of resources of a computing system 8 according to the techniques of this disclosure. Figure 2BIn the example shown in , a pre-deployment check is performed to verify the current environment and discover incompatibilities that may affect the deployment of a containerized network architecture such as an SDN architecture. In some aspects, pre-deployment testing can be implemented using Kubernetes custom resources. The definition of the custom resource can be stored in the configuration storage 224. In some aspects, two custom resource definitions (CRDs) are used, an "ApplicationReadiness" CRD that defines a custom resource (e.g., an instance of Ready 62) that represents the pre-deployment test suite and the post-deployment test suite as a whole, and an "ApplicationReadinessTest" CRD that defines a custom resource (e.g., an instance of ReadyTest 63 that represents a separate pre-deployment test and post-deployment test). An exemplary ApplicationReadiness CRD is provided in Appendix A of U.S. Provisional Application No. 63 / 376,058. An exemplary ApplicationReadinessTest CRD is provided in Appendix B of U.S. Provisional Application No. 63 / 376,058.

[0103] In some aspects, ready container pool 248 can be deployed to a server of computing system 8, such as one of the servers hosting container orchestrator 242. Ready container pool 248 can include a readiness controller 249. Readiness controller 249 can be a Kubernetes custom controller that listens for events related to the ApplicationReadiness custom resource. An exemplary definition of readiness controller 249 is provided in Appendix C of U.S. Provisional Application No. 63 / 376,058.

[0104] The readiness controller 249 can receive as input an "ApplicationReadinessSpec", which includes information specifying a list of tests to be executed, a node selector indicating one or more nodes on which the test job will be created, and the name of a ConfigMap containing the application installation manifest (e.g., files and other resources deployed as part of the application). A ConfigMap is an API object that stores data in key-value pairs and can be used to set configuration data separately from application code. A container pool can use ConfigMaps as environment variables, command-line parameters, or configuration files in a volume. The test list can include an indicator of the container image to be run to execute the tests. The test list can include predefined tests that can be provided by the application provider as well as custom tests that have been developed by a tenant or customer. An exemplary ApplicationReadinessSpec is provided in Appendix D of U.S. Provisional Application No. 63 / 376,058.

[0105] In response to receiving the ApplicationReadinessSpec, the Readiness controller 249 can install custom resources as defined by the ApplicationReadiness CRD. The Readiness controller 249 can initiate a job (e.g., a workload) on each node specified by the ApplicationReadinessSpec, for example, using a node selector. This job can cause the tests contained in the pre-deployment test container pool 260 to execute on the specified server 12. A job will be launched on the selected node for each test and will run the test container within the pre-deployment test container pool 260. Different pre-deployment tests can be implemented using containers (identified using references to container images, as described above for custom tests), which can be deployed using one or more instances of the pre-deployment test container pool 260. For example, a configmap name can be provided as an environment variable to each pre-deployment test container pool 260. These tests can be Readiness Test 63 custom resources of type ApplicationReadinessTest. This job can be a Kubernetes job that creates one or more container pools and will continue to retry execution of the container pools until a specified number of container pools successfully terminate. When a container pool successfully completes, the job tracks the successful completion. When the specified number of successful completions is reached, the task (i.e., job) is complete. Deleting the job will clean up the container pool it created. Pausing the job will delete its active container pool until the job is resumed.

[0106] An exemplary CRD for pre-deployment testing is provided in Appendix E of U.S. Provisional Application No. 63 / 376,058. An exemplary CRD for post-deployment testing is provided in Appendix F of U.S. Provisional Application No. 63 / 376,058.

[0107] Each tested container image in the pre-deployment test container pool 260 can provide output on "standard output" (referred to as "stdout"). In some implementations, the test output of the pre-deployment test container pool 260 is in a standardized format, such as JSON format. The pre-deployment test output can include the following key-value fields:

[0108] Key: 'timestamp', Value: 'string', Field Description: Timestamp in RFC3309 format (Example: 'timestamp: "2022-03-09T00::20::41Z"').

[0109] Key: 'step', Value: 'string', Field Description: The test step that has just been executed, in UpperCamelCase format.

[0110] Key: 'message', Value: 'string', Field Description: Description of this test step

[0111] Key: 'result', Value: 'int', Field Description: An integer value representing the step execution result, 0 represents success, and others represent failure.

[0112] Key: "failure_reason", Value: "string", Field Description: The reason for the failure (if any).

[0113] Pre-deployment test output may be directed to the pre-test log 262. However, this is optional, and the readiness controller 249 may listen for events related to the readiness test 63 custom resource, such as pre-deployment test output.

[0114] Example test output using the above format is as follows:

[0115] #### Example stdout output of the test

[0116] {"timestamp":"2022-03-09T00:20:01Z","step":"TestStarted",

[0117] { "message": "Test execution started", "result": 0, "failure reason": ""}

[0118] {"timestamp":"2022-03-09T00:20:15Z","step":"TestResourcesReady",

[0119] "message": "All test resources prepared", "result": 0, "failure reason": ""}

[0120] {"timestamp":"2022-03-09T00:20:21Z","step":"TestCompleted",

[0121] "message": "Test execution completed successfully", "result": 0,

[0122] "Failure Reason": ""}

[0123] In some aspects, tests follow a convention where each test includes a "TestStarted" step and a "TestCompleted" step. In some aspects, if multiple outputs of a test include the same step key, only the latest output for that step key is used. Additionally, each test can have a comprehensive status field that is updated during test execution.

[0124] When testing of the pre-deployment test container pool 260 is complete, the readiness controller 249 obtains the output of the test (e.g., from the pre-test log 262 or by listening on stdout for all custom resources of type ApplicationReadinessTest) and compiles the final state (Ready 62) of each test suite custom resource into the state of the pre-deployment test suite.

[0125] In some aspects, one or more of the following tests may be defined as a readiness test 63 custom resource and included in the pre-deployment test container pool 260:

[0126] Resources based on application profiles (CPU / memory / disk / disk speed (fio), etc.)

[0127] inotify monitoring limits -> set warning if memory amount is too low or below double the memory amount

[0128] Domain Name System (DNS) check

[0129] Host name resolution within the cluster

[0130] Maximum Transmission Unit (MTU) check / report

[0131] Network Time Protocol (NTP) check

[0132] Supports separate management and control / data channels

[0133] Artifactory dependencies, from one version of the application to another (if any)

[0134] Structural connectivity check

[0135] If separate control and data interfaces exist, ensure that nodes can reach each other via these interfaces

[0136] Check gateway configuration

[0137] OS / kernel version check, associated with application release

[0138] DPDK related checks

[0139] Large page memory

[0140] Grand Unified Boot Loader (GRUB) or other boot loader

[0141] Kubernetes / Distro version check

[0142] Known host port check:

[0143] ·Proxy (8085) and control self-test port (8083), XMPP (5269), BGP (179)

[0144] In some implementations, components of a network architecture such as the SDN architecture 200 may be deployed across multiple computing resource clusters in the computing system 8. In such implementations, pre-deployment testing / checking may also include:

[0145] Container pools and service networks / subnets are unique. Cluster names are unique. Connectivity exists between the master and worker clusters.

[0146] Management connectivity check

[0147] Kubeconfigs of distributed clusters are stored as secrets in the center, ensuring that resources in the center can be obtained from distributed clusters

[0148] DNS check

[0149] In some aspects, pre-deployment testing can be automatically performed when a request is received to deploy components of a network architecture (e.g., an SDN architecture) to computing system 8. In some aspects, pre-deployment testing can be manually invoked, e.g., via a user interface.

[0150] Figure 2C is a block diagram illustrating a post-deployment testing state 203 of resources of computing system 8 in accordance with techniques of this disclosure. Figure 2C The discussion will be presented using the SDN architecture 200 as an example to be deployed. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A to 230N (“configuration nodes” or “config nodes” and collectively referred to as “configuration nodes 230”) and control nodes 232A to 232K (collectively referred to as “control nodes 232”). The configuration nodes 230 and the control nodes 232 may represent Figure 1 1. Although shown as separate from the server 12, the configuration node 230 and the control node 232 may be executed as one or more workloads on the server 12.

[0151] Configuration node 230 provides a northbound, representational state transfer (REST) ​​interface to support intent-driven configuration of SDN architecture 200. Exemplary platforms and applications that can be used to push intent to configuration node 230 include a virtual machine orchestrator 240 (e.g., Openstack), a container orchestrator 242 (e.g., Kubernetes), a user interface 242, or other one or more applications 246. In some examples, SDN architecture 200 uses Kubernetes as its underlying platform.

[0152] The SDN architecture 200 is divided into a configuration plane, a control plane, and a data plane, as well as an optional telemetry (or analysis) plane. The configuration plane is implemented using horizontally scalable configuration nodes 230, the control plane is implemented using horizontally scalable control nodes 232, and the data plane is implemented using compute nodes.

[0153] At a high level, the configuration node 230 uses the configuration store 224 to manage the state of the configuration resources of the SDN architecture 200. Generally, a configuration resource (or more simply, a "resource") is a named object pattern that includes data and / or methods that describe a custom resource and defines an application programming interface (API) for creating and manipulating data through the API server. A kind is the name of the object pattern. Configuration resources can include Kubernetes native resources such as a container pool, an ingress, a Configmap, a service, a role, a namespace, a node, a NetworkPolicy, or a LoadBalancer. According to the technology of the present disclosure, configuration resources also include custom resources, which are used to extend the Kubernetes platform by defining application programming interfaces (APIs) that may not be available in a default installation of the Kubernetes platform. In the example of the SDN architecture 200, custom resources can describe the physical infrastructure, virtual infrastructure, configuration, and / or other resources of the SDN architecture 200. As part of configuring and operating the SDN architecture 200, various custom resources can be instantiated. Instantiated resources (whether native or custom) can be referred to as objects or resource instances. They are persistent entities in the SDN architecture 200 and represent the intent (desired state) and state (actual state) of the SDN architecture 200. The configuration node 230 provides an aggregate API for performing operations (i.e., create, read, update, and delete) on the configuration resources of the SDN architecture 200 in the configuration store 224. The load balancer 226 represents one or more load balancer objects that load balance configuration requests across the configuration nodes 230. The configuration store 224 can represent one or more etcd databases. The configuration node 230 can be implemented using Nginx.

[0154] The SDN architecture 200 can provide networking for OpenStack and Kubernetes. OpenStack uses a plug-in architecture to support networking. Leveraging the virtual machine orchestrator 240 (i.e., OpenStack), the OpenStack network plug-in driver converts OpenStack configuration objects into SDN architecture 200 configuration objects (resources). Compute nodes run OpenStack nova to start virtual machines.

[0155] Utilizing the container orchestrator 242 (i.e., Kubernetes), the SDN architecture 200 serves as a Kubernetes CNI. As described above, Kubernetes native resources (container pools, services, ingress, external load balancers, etc.) can be supported, and the SDN architecture 200 can support Kubernetes' custom resources for advanced networking and security of the SDN architecture 200.

[0156] Configuration nodes 230 provide REST monitoring to control nodes 232 to monitor configuration resource changes affected by control nodes 232 within the computing infrastructure. Control nodes 232 receive configuration resource data from configuration nodes 230 via monitoring resources and build a complete configuration graph. A given control node 232 consumes the configuration resource data associated with the control node and distributes the required configuration to the computing nodes (servers 12) via control interface 254 to reach the control plane aspect of virtual router 21 (i.e., virtual router agent- Figure 1 (not shown). Any compute node 232 may receive only a portion of the graph as needed for processing. Control interface 254 may be XMPP. The number of configuration nodes 230 and control nodes 232 deployed may be a function of the number of clusters supported. To support high availability, the configuration plane may include 2N+1 configuration nodes 230 and 2N control nodes 232.

[0157] Control nodes 232 distribute routes between compute nodes. Control nodes 232 use internal Border Gateway Protocol (iBGP) to exchange routes between them, and can peer with any external BGP-supported gateway or other router. Control nodes 232 can use route reflectors. Using configuration interface 256, control node 232 configures virtual router 21 with routing information to forward traffic between workloads using the overlay / virtual network.

[0158] Component container pool 250 and virtual machines 252 are examples of workloads that can be deployed to compute nodes by virtual machine orchestrator 240 or container orchestrator 242. Component container pool 250 can include elements of SDN architecture 200 and can be interconnected by SDN architecture 200 using one or more virtual networks.

[0159] After deploying the SDN architecture 200 (e.g., deploying the component container pool 250), the readiness controller 249 can perform post-deployment checks specified in the ApplicationReadinessSpec, or in a specification different from the pre-deployment specification, to determine whether the deployed SDN architecture is ready to handle application workloads. As with the pre-deployment readiness test suite, the readiness controller 249 can install custom resources as defined by the ApplicationReadiness CRD. The readiness controller 249 can initiate a job (e.g., a workload) on each server (node) specified by the ApplicationReadinessSpec, for example, using a node selector. This job can cause the tests contained in the post-deployment test container pool 261 to execute on the specified servers 12. A job will be launched on the selected node for each test and will run the test container within the pre-deployment test container pool 260. Different pre-deployment tests can be implemented using containers (identified using references to container images, as described above for custom tests), which can be deployed using one or more instances of the pre-deployment test container pool 260. For example, a configmap name can be provided as an environment variable to each post-deployment test container pool 261. These tests can be Readiness Test 63 custom resources of type ApplicationReadinessTest. The job can be a Kubernetes job that creates one or more container pools and will continue to retry the execution of the container pools until a specified number of container pools have successfully terminated. When a container pool completes successfully, the job tracks successful completions. When the specified number of successful completions is reached, the task (i.e., the job) is complete. Deleting the job will clean up the container pools it created. Pausing the job will delete its active container pools until the job is resumed again.

[0160] The container image for each test in the post-deployment test container pool 261 can provide output on stdout. In some implementations, the test output of the post-deployment test container pool 261 is in a standardized format, such as JSON, and can be similar to those described above with respect to the pre-deployment test container pool 260. Post-deployment test output can be directed to the post-test log 263. However, this is optional, and the readiness controller 249 can listen for events related to the readiness test 63 custom resource, such as post-deployment test output.

[0161] In some aspects, post-deployment testing follows the convention that each test includes a "TestStarted" step and a "TestCompleted" step. In some aspects, if there are multiple outputs of a test that include the same step key, only the latest output for that step key is used. Additionally, each test can have a comprehensive status field that is updated during test execution.

[0162] When the tests for the post-deployment test container pool 261 are completed, the readiness controller 249 obtains the output of the tests (e.g., from the post-test log 262 or by listening on stdout for all custom resources of type ApplicationReadinessTest) and compiles the final state (Ready 62) of each test suite custom resource into the state of the post-deployment test suite.

[0163] In some aspects, one or more of the following tests may be defined as custom resources and included in post-deployment test container pool 261:

[0164] Application state (in this example, the SDN architecture state)

[0165] Container pool to container pool communication (same node, cross-node, cross-Kubernetes cluster)

[0166] Reports round-trip time (RTT) and packet loss between container pools, including single and multiple pings

[0167] TCP large file (1G) transmission

[0168] UDP large file (1G) transmission

[0169] Reports path MTU and verifies it is compatible with its interface

[0170] Test queries to the API server (configuration or operational status)

[0171] Report TCP segment size

[0172] Re-run multi-cluster pre-flight tests to accommodate multi-cluster environments

[0173] Reports all SDN architecture resources that are not in a successful state

[0174] In some aspects, post-deployment testing can be automatically performed after components of a network architecture (e.g., an SDN architecture) are deployed to computing system 8. In some aspects, post-deployment testing can be manually invoked, e.g., via user interface 50.

[0175] Figure 3is a block diagram that shows in more detail another view of the components of a container orchestrator in accordance with the techniques of this disclosure. Custom resources are extensions to the Kubernetes API. A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in "container pool" resource contains a collection of container objects. Custom resources are extensions to the Kubernetes API that are not necessarily available in a default Kubernetes installation. Custom resources represent customizations to a specific Kubernetes installation. Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster administrators can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they would for built-in resources such as container pools.

[0176] exist Figure 3 In the example shown in , container orchestrator 242 includes ready container pool 248, API server 300, custom resource controller 302, configuration store 224, ready container pool 248 including readiness controller 249, and container platform 19. API server 300 can be a Kubernetes API server. Custom resources associated with pre-deployment testing and post-deployment testing are described above.

[0177] The API server 300 is extended with a Ready 62 custom resource and a Ready Test 63 custom resource defined using CRDs. The Readiness Controller 249 can apply logic to execute the pre-deployment test suite, the post-deployment test suite, and the tests themselves. The test logic is implemented as a coordination loop. Figure 6 is a block diagram illustrating an example of a custom controller for custom resources for pre-deployment testing and post-deployment testing according to the techniques of this disclosure. Custom controller 814 may represent an exemplary instance of readiness controller 249. Figure 6 , a custom controller 814 can be associated with a custom resource 818. Custom controller 814 can include a coordinator 816 that includes logic to perform a reconcile loop in which custom controller 814 observes 834 (e.g., monitors) a current state 832 of custom resource 818. In response to determining that a desired state 836 does not match current state 832, coordinator 816 can perform an action to adjust 838 the state of the custom resource so that current state 832 matches desired state 836. A request to change current state 832 of custom resource 818 to desired state 836 can be received by API server 300.

[0178] In the case where the API request is a create request for a custom resource, the coordinator 816 can act on a create event for instance data of the custom resource. The coordinator 816 can create instance data for custom resources that the requested custom resource depends on.

[0179] By default, the custom resource controller 302 operates in active-passive mode and uses master election for consistency. When the controller pod starts, it attempts to create a ConfigMap resource in Kubernetes using the specified key. If successful, the pod becomes the master and begins handling coordination requests; otherwise, it blocks and attempts to create the ConfigMap in an endless loop.

[0180] Configuration store 224 may be implemented as etcd. etcd is a consistent and highly available key-value store used as a backing store for Kubernetes cluster data.

[0181] Figure 4 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure. Figure 4 The computing device 500 can represent a real server or a virtual server, and can represent an exemplary instance of any server in the server 12, and can be referred to as a computing node, a master / slave node, or a host. In this example, the computing device 500 includes a bus 542 coupled to the hardware components of the computing device 500 hardware environment. The bus 542 couples a network interface card (NIC) 530, a storage disk 546, and one or more microprocessors 510 (hereinafter referred to as "microprocessors 510"). The NIC 530 can be SR-IOV capable. In some cases, the front side bus can be coupled to the microprocessor 510 and the memory device 524. In some examples, the bus 542 can be coupled to the memory device 524, the microprocessor 510, and the NIC 530. The bus 542 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to the bus 542. In some examples, components coupled to bus 542 control DMA transfers between components coupled to bus 542 .

[0182] The microprocessor 510 may include one or more processors, each of which includes an independent execution unit that executes instructions in accordance with an instruction set architecture and instructions stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0183] Disk 546 represents computer-readable storage media, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by microprocessor 510.

[0184] The main memory 524 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store desired program code and program data in the form of instructions or data structures and that can be accessed by a computer. The main memory 524 provides a physical address space consisting of addressable memory locations.

[0185] The network interface card (NIC) 530 includes one or more interfaces 532 configured to exchange packets using the links of the underlying physical network. The interfaces 532 may include a port interface card having one or more network ports. For example, the NIC 530 may also include on-card memory for storing packet data. Direct memory access transmissions between the NIC 530 and other devices coupled to the bus 542 may be read from the NIC memory or written to the NIC memory.

[0186] Memory 524, NIC 530, storage disk 546, and microprocessor 510 can provide an operating environment for a software stack including an operating system kernel 580 executed in kernel space. For example, kernel 580 can represent Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft. In some instances, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include kernel-based virtual machines (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can encompass virtual machine managers (VMMs). The operating system including kernel 580 provides an execution environment for one or more processes in user space 545.

[0187] The kernel 580 includes a physical driver 525 that uses the network interface card 530. The network interface card 530 may also implement SR-IOV to enable communication between one or more virtual execution elements (such as a container 529A or one or more virtual machines ( Figure 4 530)). A shared virtual device, such as a virtual function, can provide dedicated resources so that each virtual execution element can access the dedicated resources of the NIC 530, so that the NIC appears to each virtual execution element as a dedicated NIC. A virtual function can represent a lightweight PCIe function that shares physical resources with the physical function used by the physical driver 525 and with other virtual functions. For a NIC 530 that supports SR-IOV, the NIC 530 can have thousands of available virtual functions according to the SR-IOV standard, but for I / O-intensive applications, the number of configured virtual functions is typically much smaller.

[0188] Computing device 500 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from physical switches to software or "virtual" routers (including virtual router 506) on physical servers coupled to the switching fabric. A virtual router may be a server that is hosted by a physical server (e.g., Figure 1 A virtual router dynamically creates and manages one or more virtual networks that can be used for communications between virtual network endpoints by a process or thread, or a combination of processes and threads, executed by a server 12 of the server. In one example, the virtual routers each implement a virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executed. Each virtual network can use its own addressing and security scheme and can be considered orthogonal to the physical network and its addressing scheme. Various techniques can be used to transmit packets within and across virtual networks on top of a physical network. As used herein, the term "virtual router" can encompass Open vSwitch (OVS), OVS bridge, Linux bridge, Docker bridge, or other device and / or software that is located on a host device and performs switching, bridging, or routing packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more of the servers 12. In Figure 4 In the exemplary computing device 500 of FIG. 5 , the virtual router 506 executes in user space as a DPDK-based virtual router, but in various implementations, the virtual router 506 may execute within a hypervisor, a host operating system, a host application, or a virtual machine.

[0189] The virtual router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module, which is typically used in Kubernetes deployments of the container pool 502. The virtual router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 506 can perform networking services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0190] The virtual router 506 can be executed as a kernel module or as a user space DPDK process (here the virtual router 506 is shown in user space 545). The virtual router agent 514 can also be executed in user space. In the exemplary computing device 500, the virtual router 506 is executed in user space as a DPDK-based virtual router, but in various implementations, the virtual router 506 can be executed within a hypervisor, a host operating system, a host application, or a virtual machine. The virtual router agent 514 connects to the network controller 24 using a channel for downloading configuration and forwarding information. The virtual router agent 514 programs the forwarding state into the virtual router data (or "forwarding") plane represented by the virtual router 506. The virtual router 506 and the virtual router agent 514 can be processes. The virtual router 506 and the virtual router agent 514 can be containerized / cloud native, although they are not shown as being contained in a container pool.

[0191] The virtual router 506 can replace and include the virtual routing / bridging functionality of the Linux bridge / OVS module, which is typically used in Kubernetes deployments of the container pool 502. The virtual router 506 can perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. The virtual router 506 can perform networking services such as applying security policies, NAT, multicast, mirroring, and load balancing.

[0192] Virtual router 506 can be multi-threaded and executed on one or more processor cores. Virtual router 506 can include multiple queues. Virtual router 506 can implement a packet processing pipeline. Virtual router agent 514 can stitch the pipeline in a manner ranging from simplest to most complex, depending on the operations to be applied to the packet. Virtual router 506 can maintain multiple instances of forwarding base stations. Virtual router 506 can use RCU (read-copy-update) locks to access and update tables.

[0193] To send packets to other compute nodes or switches, the virtual router 506 uses one or more physical interfaces 532. Typically, the virtual router 506 exchanges overlay packets with workloads (such as VMs or container pools 502). The virtual router 506 has multiple virtual network interfaces (e.g., vifs). These interfaces may include the kernel interface vhost0 for exchanging packets with the host operating system and the interface pkt0 with the virtual router agent 514 to obtain forwarding state from the network controller and send exception packets upstream. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 532. The other virtual network interfaces of the virtual router 506 are used to exchange packets with workloads.

[0194] In general, each of container pools 502A through 502B can be assigned one or more virtual network addresses for use in a corresponding virtual network, where each of the virtual networks can be associated with a different virtual subnet provided by virtual router 506. Container pool 502B can be assigned its own virtual layer 3 (L3) IP address, e.g., for sending and receiving communications, but may not be aware of the IP address of computing device 500 on which container pool 502B executes. Thus, the virtual network address can be different from the logical address of the underlying physical computer system (e.g., computing device 500).

[0195] The computing device 500 includes a virtual router agent 514 that controls the overlay of the virtual network of the computing device 500 and coordinates the routing of data packets within the computing device 500. Generally, the virtual router agent 514 communicates with the network controller 24 for the virtualized infrastructure, which generates commands to create virtual networks and configure network virtualization endpoints, such as the computing device 500, and more specifically, the virtual router 506, and the virtual network interfaces 212. By configuring the virtual router 506 based on information received from the network controller 24, the virtual router agent 514 can support the configuration of network isolation, policy-based security, gateways, source network address translation (SNAT), load balancers, and service chaining capabilities for orchestration.

[0196] In one example, a network packet (e.g., a Layer 3 (L3) IP packet or a Layer 2 (L2) Ethernet packet generated or consumed by container 529A to container 529B within a virtual network domain) can be encapsulated in another packet (e.g., another IP or Ethernet packet) transmitted by a physical network. The packets transmitted in the virtual network may be referred to herein as "inner packets," while the physical network packets may be referred to herein as "outer packets" or "tunnel packets." The encapsulation and / or decapsulation of the virtual network packets within the physical network packets may be performed by the virtual router 506. This functionality is referred to herein as tunneling and can be used to create one or more overlay networks. In addition to IPinIP, other exemplary tunneling protocols that may be used include IP over Generic Routing Encapsulation (GRE) (IP / GRE), VxLAN, Multiprotocol Label Switching (MPLS) over GRE, MPLS over User Datagram Protocol (UDP), and the like. The virtual router 506 performs tunnel encapsulation on packets originating from any container in the container pool 502 , performs tunnel decapsulation on packets destined for any container in the container pool 502 , and exchanges packets with the container 502 via the bus 542 and / or the bridge of the NIC 530 .

[0197] As described above, the network controller 24 can provide a logically centralized controller that facilitates the operation of one or more virtual networks. For example, the network controller 24 can maintain a routing information base, such as one or more routing tables that store routing information for the physical network and one or more overlay networks. The virtual router 506 implements one or more virtual routing and forwarding instances (VRFs), such as VRF 422A, for the corresponding virtual network for which the virtual router 506 operates as a corresponding tunnel endpoint. Generally, each VRF stores forwarding information for the corresponding virtual network and identifies where a data packet is to be forwarded and whether the packet is to be encapsulated in a tunnel protocol, such as with a tunnel header, which may include one or more headers from different layers of the virtual network protocol stack. Each VRF may include a network forwarding table that stores routing and forwarding information for the virtual network.

[0198] NIC 530 may receive the tunnel packet. Virtual router 506 processes the tunnel packet to determine the virtual networks for the source and destination endpoints of the inner packet based on the tunnel encapsulation header. Virtual router 506 may strip off the Layer 2 header and the tunnel encapsulation header to forward only the inner packet internally. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN tag or an MPLS label, indicating a virtual network, for example, the virtual network corresponding to VRF 422A. VRF 422A may include forwarding information for the inner packet. For example, VRF 422A may map the destination Layer 3 address for the inner packet to virtual network interface 212. In response, VRF 422A forwards the inner packet to container pool 502A via virtual network interface 212.

[0199] Container 529A can also source internal packets as a virtual network endpoint. For example, container 529A can generate a Layer 3 internal packet destined for a target virtual network endpoint executed by another computing device (i.e., not computing device 500), or for another container. Container 529A can send the Layer 3 internal packet to virtual router 506 via a virtual network interface attached to VRF 422A.

[0200] Virtual router 506 receives the inner packet and the Layer 2 header and determines the virtual network for the inner packet. Virtual router 506 may use any of the aforementioned virtual network interface implementation technologies (e.g., macvlan, veth, etc.) to determine the virtual network. Virtual router 506 generates an outer header for the inner packet using VRF 422A corresponding to the virtual network for the inner packet. The outer header includes an outer IP header for the overlay tunnel and a tunnel encapsulation header identifying the virtual network. Virtual router 506 encapsulates the inner packet with the outer header. Virtual router 506 may encapsulate the tunnel packet with a new Layer 2 header that has a destination Layer 2 address associated with a device external to computing device 500 (e.g., one of TOR switches 16 or servers 12). If external to computing device 500, virtual router 506 outputs the tunnel packet with the new Layer 2 header to NIC 530 using physical function 221. NIC 530 outputs the packet on the outbound interface. If the destination is another virtual network endpoint executing on computing device 500 , virtual router 506 routes the packet to the appropriate one of virtual network interfaces 212 , 213 .

[0201] In some examples, a controller of computing device 500 (e.g., Figure 1The network controller 24 of the NIC 530 configures a default route in each container 502 so that the virtual machine 224 uses the virtual router 506 as the initial next hop for outbound packets. In some examples, the NIC 530 is configured with one or more forwarding rules so that all packets received from the virtual machine 224 are switched to the virtual router 506.

[0202] Container pool 502A includes one or more application containers 529A. Container pool 502B includes an instance of containerized routing protocol daemon (cRPD) 560. Container platform 588 includes container runtime 590, orchestration agent 592, service agent 593, and CNI 570.

[0203] The container engine 590 includes code that can be executed by the microprocessor 510. The container runtime 590 can be one or more computer processes. The container engine 590 runs containerized applications in the form of containers 529A to 529B. The container engine 590 can represent Docker, rkt, or other container engines for managing containers. Generally, the container engine 590 receives requests and manages objects such as images, containers, networks, and volumes. An image is a template with instructions for creating a container. A container is an executable instance of an image. Based on instructions from the controller agent 592, the container engine 590 can obtain an image and instantiate it as an executable container in the container pool 502A to 502B.

[0204] The service agent 593 includes code that can be executed by the microprocessor 510. The service agent 593 can be one or more computer processes. The service agent 593 monitors the addition and removal of services and endpoint objects, and it maintains the network configuration of the computing device 500 to ensure communication between the container pool and the containers, for example, using the service. The service agent 593 can also manage iptables to capture traffic to the virtual IP address and port of the service and redirect the traffic to the proxy port of the proxy backup container pool. The service agent 593 can represent a kube proxy for a slave node of a Kubernetes cluster. In some examples, the container platform 588 does not include the service agent 593, or the service agent 593s are disabled to support the configuration of the virtual router 506 and the container pool 502 through the CNI 570.

[0205] The orchestration agent 592 includes code that can be executed by the microprocessor 510. The orchestration agent 592 can be one or more computer processes. The orchestration agent 592 can represent a kubelet for a slave node of a Kubernetes cluster. The orchestration agent 592 is an orchestrator (e.g., Figure 1Orchestration agent 592 is an agent of orchestrator 23 that receives container specification data for a container and ensures that the container is executed by computing device 500. The container specification data can be in the form of a manifest file sent from orchestrator 23 to orchestration agent 592, or it can be received indirectly via a command line interface, HTTP endpoint, or HTTP server. The container specification data can be a container pool specification (e.g., a PodSpec—a YAML (Yet Another Markup Language) object or JSON object describing a container pool) for one of container pools 502 of the container. Based on the container specification data, orchestration agent 592 instructs container engine 590 to obtain and instantiate a container image for container 529 for computing device 500 to execute container 529.

[0206] Orchestration agent 592 instantiates or otherwise invokes CNI 570 to configure one or more virtual network interfaces for each container pool 502. For example, orchestration agent 592 receives container specification data for container pool 502A and instructs container engine 590 to create container pool 502A with container 529A based on the container specification data for container pool 502A. Orchestration agent 592 also invokes CNI 570 to configure a virtual network interface for the virtual network corresponding to VRF 422A for container pool 502A. In this example, container pool 502A is a virtual network endpoint for the virtual network corresponding to VRF 422A.

[0207] CNI 570 can obtain interface configuration data for configuring virtual network interfaces for container pool 502. Virtual router agent 514 operates as a virtual network control plane module, enabling network controller 24 to configure virtual router 506. Unlike the orchestration control plane (including container platform 588 for slave and master nodes, such as orchestrator 23), which manages the provisioning, scheduling, and management of virtual execution elements, the virtual network control plane (including network controller 24 for slave nodes and virtual router agent 514) partially manages the configuration of virtual networks implemented in the data plane by virtual router 506 of the slave nodes. Virtual router agent 514 transmits interface configuration data for virtual network interfaces to CNI 570, enabling the orchestration control plane element (i.e., CNI 570) to configure the virtual network interfaces based on the configuration state determined by network controller 24, thereby building a bridge between the orchestration control plane and the virtual network control plane. Furthermore, this may enable CNI 570 to obtain interface configuration data for multiple virtual network interfaces of a container pool and configure the multiple virtual network interfaces, which may reduce the communication and resource overhead inherent in invoking a separate CNI 570 for configuring each virtual network interface.

[0208] Figure 51300 is a block diagram of an exemplary computing device operating as a compute node of a computing system for one or more clusters of an SDN architecture system according to the techniques of this disclosure. Computing device 1300 can represent one or more real or virtual servers. Computing device 1300 can, in some cases, implement one or more master nodes for the corresponding cluster or clusters.

[0209] The scheduler 1322, API server 300A, custom API server 301A, controller 406A, ready controller 249, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration storage 1328 are components of the SDN architecture system and, although shown and described as being executed by a single computing device 1300, may be distributed among multiple computing devices constituting a computing system or hardware / server cluster. In other words, each of the multiple computing devices may provide a hardware operating environment for one or more instances of any one or more of the scheduler 1322, API server 300A, custom API server 301A, controller 406A, ready controller 249, controller manager 1326, SDN controller manager 1325, control node 232A, or configuration storage 1328.

[0210] In this example, computing device 1300 includes a bus 1342 that couples hardware components of the computing device 1300 hardware environment. Bus 1342 couples a network interface card (NIC) 1330, a storage disk 1346, and one or more microprocessors 1310 (hereinafter referred to as "microprocessors 1310"). In some cases, a front-side bus can couple microprocessors 1310 and a memory device 1344. In some examples, bus 1342 can couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 can represent a peripheral component interface (PCI) Express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers between components coupled to bus 1342. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

[0211] The microprocessor 1310 may include one or more processors, each of which includes an independent execution unit that executes instructions in accordance with an instruction set architecture and instructions stored in a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) each implemented using a single IC (i.e., a chip multiprocessor).

[0212] Disk 1346 represents computer-readable storage media, including volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by microprocessor 1310.

[0213] The main memory 1344 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store desired program code and program data in the form of instructions or data structures and that can be accessed by a computer. The main memory 1344 provides a physical address space consisting of addressable memory locations.

[0214] The network interface card (NIC) 1330 includes one or more interfaces 3132 configured to exchange packets using the links of the underlying physical network. The interfaces 3132 may include a port interface card having one or more network ports. For example, the NIC 1330 may also include on-card memory for storing packet data. Direct memory access transmissions between the NIC 1330 and other devices coupled to the bus 1342 may be read from the NIC memory or written to the NIC memory.

[0215] Memory 1344, NIC 1330, storage disk 1346, and microprocessor 1310 can provide an operating environment for a software stack including an operating system kernel 1314 executed in kernel space. For example, kernel 1314 can represent Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft. In some instances, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can encompass a virtual machine manager (VMM). The operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. Kernel 1314 includes a physical driver 1327 that uses the network interface card 1330.

[0216] Computing device 1300 may be coupled to a physical network switching fabric that includes an overlay network that extends the switching fabric from physical switches to software or "virtual" routers (such as virtual router 21) that are coupled to physical servers of the switching fabric. Computing device 1300 may configure slave nodes of a cluster using one or more dedicated virtual networks.

[0217] Scheduler 1322, API server 300A, custom API server 301A, controller 406A, readiness controller 249, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration store 1328 may implement a master node for a cluster and may alternatively be referred to as a “master component.” The cluster may be a Kubernetes cluster, and the master node may be a Kubernetes master node, in which case the master component is a Kubernetes master component.

[0218] Each of the scheduler 1322, API server 300A, custom API server 301A, controller 406A, readiness controller 249, controller manager 1326, SDN controller manager 1325, and control node 232A includes code executable by microprocessor 1310. Custom API server 301A validates and configures data for custom resources configured for the SDN architecture, as described in U.S. patent application Ser. No. 17 / 657,596, incorporated by reference above. A service can be an abstraction that defines a logical set of container pools and policies for accessing the container pools. A set of container pools that implement the service is selected based on the service definition. A service can be partially implemented as a load balancer or otherwise include a load balancer. API server 300A and custom API server 301A can implement a Representational State Transfer (REST) ​​interface to handle REST operations and, as part of the configuration plane for the SDN architecture, provide a front-end to the shared state of the corresponding cluster, which is stored in configuration store 1328. API server 300A can represent a Kubernetes API server.

[0219] Configuration store 1328 is the backing store for all cluster data. Cluster data can include cluster state and configuration data. Configuration data can also provide a backend for service discovery and / or provide locking services. Configuration store 1328 can be implemented as a key-value store. Configuration store 1328 can be a centralized database or a distributed database. Configuration store 1328 can represent etcd storage. Configuration store 1328 can also represent the Kubernetes configuration store.

[0220] Scheduler 1322 includes code executable by microprocessor 1310. Scheduler 1322 can be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., a container pool of containers) and selects slave nodes on which the virtual execution elements are to run. Scheduler 1322 can select slave nodes based on resource requirements, hardware constraints, software constraints, policy constraints, location, etc. Scheduler 1322 can represent a Kubernetes scheduler.

[0221] Typically, API server 1320 may call scheduler 1322 to schedule a container pool. Scheduler 1322 may select a slave node and return an identifier of the selected slave node to API server 1320, which may write the identifier to configuration storage 1328 associated with the container pool. API server 1320 may call orchestration agent 310 for the selected slave node, which may cause container engine 208 for the selected slave node to obtain the container pool from the storage server and create a virtual execution element on the slave node. Orchestration agent 310 for the selected slave node may update the state for the container pool to API server 1320, which may persist the new state to configuration storage 1328. In this manner, computing device 1300 instantiates a new container pool in computing system 8.

[0222] Controller manager 1326 includes code that can be executed by microprocessor 1310. Controller manager 1326 can be one or more computer processes. Controller manager 1326 can be embedded in the core control loop and monitor the shared state of the cluster by obtaining notifications from API server 1320. Controller manager 1326 can attempt to move the state of the cluster toward a desired state. Exemplary controller 406A and custom resource controller 302A can be managed by controller manager 1326. Other controllers can include replication controllers, endpoint controllers, namespace controllers, and service account controllers. Controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminated container garbage collection, cascade deletion garbage collection, node garbage collection, etc. Controller manager 1326 can represent a Kubernetes controller manager for a Kubernetes cluster.

[0223] The SDN controller manager 1325 can operate as an interface between Kubernetes core resources (services, namespaces, container pools, network policies, network attachment definitions) and extended SDN fabric resources (VirtualNetwork, RoutingInstance, etc.). The SDN controller manager 1325 monitors the Kubernetes API for changes to the Kubernetes core and custom resources used for SDN fabric configuration, and can therefore perform CRUD operations on related resources.

[0224] In some examples, the SDN controller manager 1325 is a collection of one or more Kubernetes custom controllers. In some examples, in a single cluster or multi-cluster deployment, the SDN controller manager 1325 can run on the Kubernetes cluster it manages.

[0225] The SDN controller manager 1325 listens for create, delete, and update events for the following Kubernetes objects:

[0226] ·Container pool

[0227] ·Serve

[0228] NodePort

[0229] ·Entrance

[0230] Endpoints

[0231] Namespace

[0232] ·deploy

[0233] Network strategy

[0234] When these events are generated, the SDN controller manager 1325 creates appropriate SDN architecture objects, which in turn are defined as custom resources for SDN architecture configuration. In response to detecting an event on an instance of a custom resource, whether instantiated by the SDN controller manager 1325 and / or through the custom API server 301, the control node 232A obtains configuration data for the instance of the custom resource and configures a corresponding instance of the configuration object in the SDN architecture 400.

[0235] For example, the SDN controller manager 1325 monitors container pool creation events and, in response, can create the following SDN architecture objects: VirtualMachine (workload / container pool), VirtualMachineInterface (virtual network interface), and InstanceIP (IP address). In this case, the control node 232A can then instantiate the SDN architecture objects in the selected compute node.

[0236] Figure 7A and Figure 7B700 is a flowchart illustrating an exemplary mode of operation for performing pre-deployment checks and post-deployment checks on a software-defined network architecture system according to the techniques of the present disclosure. The operations of method 700 can be performed in part by the readiness controller 249. The readiness controller 249 instantiates a pre-deployment test container pool on each of one or more of a plurality of servers based on a configuration, the pre-deployment test container pool including one or more containerized pre-deployment tests (702). The readiness controller 249 schedules the containerized pre-deployment tests for execution on each of one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a pre-deployment test log (704). The readiness controller 249 obtains a pre-deployment test log from each of the one or more of the plurality of servers (706). The pre-deployment test container pool can stream log data for the readiness test to a log system, and the readiness controller 249 can obtain the log from the log system.

[0237] The readiness controller 249 determines whether the resources of one or more of the plurality of servers are compatible with the configuration of the containerized network architecture to be deployed to the plurality of servers based on the pre-deployment test log (708). In response to the determination that the one or more servers are compatible with the configuration of the containerized network architecture (the "yes" branch of 710), the readiness controller 249 deploys the containerized network architecture to the one or more servers (712); based on the configuration data, instantiates a post-deployment test container pool on each of the one or more servers, the post-deployment test container pool including one or more containerized post-deployment tests (714); schedules the containerized post-deployment tests to be executed on each of the one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a post-deployment test log (716); obtains a post-deployment test log from each of the one or more of the plurality of servers (718); and determines the operational status of the containerized network architecture based on the post-deployment test log (720).

[0238] Aspects described in this disclosure include a testing scenario in which a custom resource (CR) is tasked with (1) first verifying that the cluster is particularly suitable for network controller / network data plane / CNI workloads (as opposed to other types of containerized applications) [pre-deployment checks of the pre-deployment test suite], and (2) is also responsible for subsequently verifying the network controller / network data plane / CNI deployment on the cluster to verify the readiness of the network controller / CNI and cluster for application workloads, including the network [post-deployment checks of the post-deployment test suite]. The SDN architecture system described herein is an example of a network controller / network data plane / CNI. Because the SDN architecture system components to be deployed operate as CNIs for the cluster to support additional applications, traditional methods for verifying deployed application workloads cannot account for and verify this early deployment of the network controller / network data plane / CNI. By performing pre-deployment checks and post-deployment checks as part of the SDN architecture system deployment, these techniques provide technical improvements that enable one or more practical applications in the field of application orchestration and management.

[0239] Figure 8 1380 。 1382A 。 1382B 。 1382C 。 1382D 。 1382A 。 1382A 。 1382B 。 1382A ... Thus, kernel 1380 may be a network endpoint for the underlying network including server 1600 as a network device, with the network endpoint having the IP address of vhost interface 1382A. The application layer endpoint may be cRPD 1324 or other process managed by kernel 1380.

[0240] The underlay network refers to the physical infrastructure that provides connectivity between nodes (usually servers) in the network. The underlay network is responsible for delivering packets within the infrastructure. The underlay network devices use routing protocols to determine IP connectivity. Typical routing protocols used on the underlay network devices for routing purposes are OSPF, IS-IS, and BGP. The overlay network refers to the virtual infrastructure that provides connectivity between virtual workloads (usually VMs / container pools). This connectivity is built on top of the underlay network and allows virtual networks to be built. Overlay traffic (i.e., virtual networks) is typically encapsulated in IP / MPLS tunnels or other tunnels that are routed by the underlay network. Overlay networks can run across all or part of the underlay network devices and enable multi-tenancy through virtualization.

[0241] Control traffic 1700 may represent routing protocol traffic for one or more routing protocols executed by cRPD 1324. In server 1600, control traffic 1700 may be received via physical interface 1322 owned by vRouter 1206A. vRouter 1206A is programmed with a route for the host IP address of vhost0 interface 1382A and the received next-hop address. This causes vRouter 1206A to send traffic received at physical interface 1322 and destined for the host IP address of vhost0 interface 1382A to kernel 1380 via vhost0 interface 1382A. From the perspective of cRPD 1324 and kernel 1380, all such control traffic 1700 appears to originate from vhost0 interface 1382A. Therefore, cRPD 1324 routes will specify vhost0 interface 1382A as the forwarding next hop for the routes. cRPD 1324 selectively installs some of the routes to vRouter agent 1314 and installs the same (or other) routes to kernel 1380, as described in further detail below. vRouter agent 1314 will receive forwarding information base (FIB) updates corresponding to some of the routes received by cRPD 1324. These routes will point to vHost0 interface 1382A, and vRouter 1206A can automatically translate or map vHost0 interface 1382A to physical interface 1322.

[0242] The routing information programmed by cRPD 1324 can be divided into underlay and overlay. cRPD 1324 will install underlay routes into kernel 1380, as cRPD 1324 may need this reachability to establish additional protocol adjacencies / sessions with external routers, such as BGP multi-hop sessions on top of the reachability provided by IGP. cRPD 1324 supports selective filtering of FIB updates for specific data planes (e.g., kernel 1380 or vRouter 1206A) using routing policy constructs that allow matching against RIB, routing instances, prefixes, or other attributes.

[0243] Control traffic 1700 sent by cRPD 1324 to vRouter 1206A through vhost0 interface 1382A may be sent by vRouter 1206A out the corresponding physical interface 1322 for vhost0 interface 1382A.

[0244] As shown, cRPD-based CNI 1312, upon notification by orchestrator 50 via orchestration agent 1310, creates a virtual network (herein referred to as a "container pool") interface for each of application container pools 1422A and 1422L. One end of the container pool interface terminates at a container included in the container pool. CNI 1312 can request vRouter 1206A to begin monitoring the other end of the container pool interface, and cRPD 1324 facilitates traffic received from physical interface 1322 destined for application containers in DPDK-based container pools 1422A and 1422L to be forwarded using only DPDK, without involving kernel 1380. The reverse process applies to traffic generated by container pools 1422A and 1422L.

[0245] However, because DPDK-based vRouter 1206A manages virtual network interfaces for container pools 1422A and 1422L, kernel 1380 is unaware of these virtual network interfaces. Server 1600 can use tunnels dedicated to the DPDK forwarding path to internally send and receive overlay data traffic 1800 between DPDK-based container pools 1422A and 1422L, vRouter 1206A, and NIC 1312B.

[0246] Thus, in server 1600, cRPD 1324 interfaces with two disjoint data planes: kernel 1380 and DPDK-based vRouter 1206A. cRPD 1324 utilizes the kernel 1380 network stack to set up routes specifically for the DPDK fast path. The routing information received by cRPD 1324 includes both underlay routing information and overlay routing information. cRPD 1324 runs a routing protocol on vHost interface 1382A visible from kernel 1380, and cRPD 1324 can install FIB updates corresponding to IGP learned routes (underlay routing information) in the kernel 1380 FIB. This can establish a multi-hop iBGP session to the destinations indicated in such IGP learned routes. Similarly, because kernel 1380 executes the network stack, cRPD 1324 routing protocol adjacencies involve kernel 1380 (and vHost interface 1382A).

[0247] The vRouter agent 1314 for vRouter 1206A notifies cRPD 1324A of the application container pool interfaces for container pool 1422A, container pool 1422L. These container pool interfaces are created by CNI 1312 and are managed exclusively by vRouter agent 1314 (i.e., without involving kernel 1380). Kernel 1380 is unaware of these container pool interfaces. cRPD 1324 can advertise the reachability of these container pool interfaces as L3VPN routes including network layer reachability information (NLRI) to the rest of the network. In a 5G mobile network environment, such L3VPN routes can be stored in the VRF of vRouter 1206A for different network slices. The corresponding MPLS routes can be programmed by cRPD 1324 only to vRouter 1206A via interface 340 with vRouter agent 1314, and not to kernel 1380. This is because the next hop for these MPLS labels is a container pool interface that is popped and forwarded to one of container pools 1422A and 1422L; these interfaces are only visible in vRouter 1206A and not in kernel 1380. Similarly, reachability information received via BGP L3VPN can be selectively programmed into vRouter 1206A by cRPD 1324, as such routes are only required to forward traffic generated by container pools 1422A and 1422L. Kernel 1380 has no applications requiring such reachability. The routes programmed into vRouter 1206A constitute the overlay routes for the overlay network.

[0248] The techniques described herein, with respect to verifying the environment in which a network controller and network data plane are deployed, and verifying the operation of the network controller and data plane after deployment, can be applied to a containerized network router (CNR) implemented using a cRPD 1324 control plane and a containerized virtual router 206A data plane. These techniques can be used to verify that the environment in which the cRPD 1324 and virtual router 206A are deployed, i.e., that the resources of an instance of server 1600 are capable of hosting the CNR. Once deployed, these techniques can be used to verify the operation of the CNR.

[0249] like Figure 8 As shown in , the pre-deployment test container pool 260 and the post-deployment test container pool 261 can be deployed by the orchestrator according to one or more application readiness specifications using custom resources (ready 62 and ready-test 63). Specific tests and test containers for instances of readiness-test 63 are deployed to instances of server 1600.

[0250] Figure 9 1 is a flowchart illustrating an exemplary mode of operation of a computing system for implementing an SDN architecture system according to the techniques of the present disclosure. The computing system 8 creates a ready custom resource in the container orchestrator 23. The ready custom resource 62 is configured to receive a specification specifying one or more tests for the software-defined network (SDN) architecture system 200. Each of the one or more tests has a corresponding container image that is configured to implement the test on a server and output a status for the test (900). The computing system 8 creates a ready test custom resource 63 in the container orchestrator 23 for each of the one or more tests (902). The computing system 8 deploys a corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers 12 (904). The computing system 8 sets a status for the ready custom resource based on the corresponding status output by the corresponding container image for the one or more tests (906). Based on the fact that the status of the ready custom resource 62 indicates success (the "yes" branch of 908), the computing system 8 deploys the workload to at least one of the plurality of servers, wherein the workload implements at least one of a component of the SDN architecture system or an application requiring network configuration of the workload by the SDN architecture system (910). If the status of the ready custom resource 62 does not indicate success (the "no" branch of 908), the computing system 8 outputs a failure indication (912).

[0251] This disclosure describes the following non-limiting enumeration of examples.

[0252] Example 1: A system comprising: multiple servers; and a container orchestrator that executes on the multiple servers and is configured to: create a ready custom resource in the container orchestrator, the ready custom resource configured to receive a specification specifying one or more tests for a containerized software-defined networking (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; create a ready test custom resource in the container orchestrator for each of the one or more tests; deploy a corresponding container image for each of the one or more tests to execute the test on at least one of the multiple servers; set a status for the ready custom resource based on the corresponding status output by the corresponding container image for the one or more tests; and deploy a workload to at least one of the multiple servers based on the status for the ready custom resource indicating success, wherein the workload implements: a component of the containerized SDN architecture system, or at least one of an application that requires network configuration through the containerized SDN architecture system.

[0253] Example 2: A containerized deployment system comprising: a plurality of servers, each of the plurality of servers comprising a memory and a processing circuit; a container platform executed on each of the plurality of servers; and a containerized-ready controller executed by the container platform of a first group of servers in the plurality of servers, the containerized-ready controller being configured to: instantiate a pre-deployment test container pool on each of one or more of the plurality of servers based on a configuration, the pre-deployment test container pool comprising one or more containerized pre-deployment tests; schedule the containerized pre-deployment tests to be executed on each of the one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a pre-deployment test log; obtain the pre-deployment test log; based on A pre-deployment test log determines whether resources of one or more of the plurality of servers are compatible with the configuration of the containerized network architecture, and in response to a determination that the one or more servers are compatible with the configuration of the containerized network architecture: deploying the containerized network architecture to the one or more servers; instantiating a post-deployment test container pool on each of the one or more servers based on the configuration data, the post-deployment test container pool including one or more containerized post-deployment tests; scheduling the containerized post-deployment tests to be executed on each of the one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a post-deployment test log; obtaining the post-deployment test log; and determining an operational status of the containerized network architecture based on the post-deployment test log.

[0254] Example 3: A method comprising: instantiating a pre-deployment test container pool on each of one or more of a plurality of servers based on a configuration, the pre-deployment test container pool including one or more containerized pre-deployment tests; scheduling the containerized pre-deployment tests to be executed on each of the one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a pre-deployment test log; obtaining the pre-deployment test log from each of the one or more of the plurality of servers; determining, based on the pre-deployment test log, whether resources of one or more of the plurality of servers are compatible with the configuration of a containerized network architecture to be deployed to the plurality of servers, and in response to a determination that the one or more servers are compatible with the configuration of the containerized network architecture: deploying the containerized network architecture to the one or more servers; instantiating a post-deployment test container pool on each of the one or more servers based on configuration data, the post-deployment test container pool including one or more containerized post-deployment tests; scheduling the containerized post-deployment tests to be executed on each of the one or more of the plurality of servers, wherein each of the containerized pre-deployment tests generates a post-deployment test log; obtaining the post-deployment test log from each of the one or more of the plurality of servers; and determining an operational status of the containerized network architecture based on the post-deployment test log.

[0255] The techniques described herein may be implemented in hardware, software, firmware, or any combination of hardware, software, and firmware. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuitry may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0256] If implemented in hardware, the present disclosure may relate to a device such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0257] Computer-readable media may form part of a computer program product, which may include packaging materials. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0258] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., in RAM or cache).

[0259] The code or instructions may be software and / or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor," as used herein, may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in this disclosure may be provided within software modules or hardware modules.

Claims

1. A system comprising: Multiple servers; as well as A container orchestrator is executed on at least one server of the plurality of servers and is configured to: creating a ready custom resource in the container orchestrator, the ready custom resource configured to receive a specification specifying one or more tests for a software defined network (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; creating, in the container orchestrator, a ready-test custom resource for each of the one or more tests; deploying the corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; setting a state for the ready custom resource based on a respective state output by the corresponding container image for the one or more tests; and Based on a success indication for the status of the ready custom resource, a workload is deployed to at least one of the multiple servers, wherein the workload implements: a component of the SDN architecture system, or at least one of an application that requires network configuration of the workload through the SDN architecture system.

2. The system according to claim 1, wherein: The SDN architecture system includes a network controller and a network data plane.

3. The system according to any one of claims 1 to 2, wherein: Each of the one or more tests specified by the specification includes a reference to the corresponding container image configured to implement the test.

4. The system according to any one of claims 1 to 3, wherein: The SDN architecture system includes a containerized network router, and the containerized network router includes a routing protocol process and a virtual router data plane.

5. The system according to any one of claims 1 to 4, wherein: The ready custom resources and the specifications implement a pre-deployment test suite to verify whether the system is suitable for the SDN architecture system.

6. The system according to any one of claims 1 to 5, wherein: The ready custom resources and the specifications enable a post-deployment test suite to validate the network controller operations and network data plane operations of the SDN architecture system on the system.

7. A system according to any one of claims 1 to 6, wherein: The container orchestrator is configured to: creating a ready controller to create one or more jobs on each of said plurality of servers as specified in said specification; and deploying a test container pool to each of the plurality of servers, the test container pool comprising the corresponding container image configured to implement the one or more tests, The one or more jobs cause the one or more tests to be performed by running the container image.

8. The system according to claim 7, wherein: The ReadyController is a custom controller for Kubernetes custom resources.

9. The system according to any one of claims 7 to 8, wherein: The readiness controller is configured to install the readiness custom resource and the readiness-test custom resource.

10. A method comprising: Creating, by a computing system, a ready custom resource in a container orchestrator executed on at least one server of a plurality of servers, the ready custom resource configured to receive a specification specifying one or more tests for a software defined network (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; creating, by the computing system, in the container orchestrator, a ready-to-test custom resource for each of the one or more tests; deploying, by the computing system, the corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; setting, by the computing system, a state for the ready custom resource based on a corresponding state output by the corresponding container image for the one or more tests; and Through the computing system, based on the success of the status indication for the ready custom resource, the workload is deployed to at least one of the multiple servers, wherein the workload implements: a component of the SDN architecture system, or at least one of an application that requires network configuration of the workload through the SDN architecture system.

11. The method according to claim 10, wherein: The SDN architecture system includes a network controller and a network data plane.

12. The method according to any one of claims 10 to 11, wherein: Each of the one or more tests specified by the specification includes a reference to the corresponding container image configured to implement the test.

13. The method according to any one of claims 10 to 12, wherein: The SDN architecture system includes a containerized network router, and the containerized network router includes a routing protocol process and a virtual router data plane.

14. The method according to any one of claims 10 to 13, further comprising: A pre-deployment test suite is implemented through the ready custom resources and the specifications to verify whether the computing system is suitable for the SDN architecture system.

15. The method according to any one of claims 10 to 14, further comprising: A post-deployment test suite is implemented through the ready custom resources and the specifications to verify the network controller operation and the network data plane operation of the SDN architecture system on the computing system.

16. The method according to any one of claims 10 to 15, further comprising: creating a ready controller to create one or more jobs on each of said plurality of servers as specified in said specification; deploying a test container pool to each of the plurality of servers, the test container pool comprising the corresponding container image configured to implement the one or more tests; and The container image is caused to be run through the one or more jobs to execute the one or more tests.

17. The method according to claim 16, wherein: The ReadyController is a custom controller for Kubernetes custom resources.

18. The method according to any one of claims 16 to 17, wherein: The readiness controller is configured to install the readiness custom resource and the readiness-test custom resource.

19. A non-transitory computer-readable medium comprising instructions that, when executed by a processing circuit, cause the processing circuit to: creating a ready custom resource in a container orchestrator executed on at least one of a plurality of servers, the ready custom resource configured to receive a specification specifying one or more tests for a software defined network (SDN) architecture system, each of the one or more tests having a corresponding container image, the container image configured to implement the test on the server and output a status for the test; creating, in the container orchestrator, a ready-test custom resource for each of the one or more tests; deploying the corresponding container image for each of the one or more tests to execute the test on at least one of the plurality of servers; setting a state for the ready custom resource based on a respective state output by the corresponding container image for the one or more tests; and Based on a success indication for the status of the ready custom resource, a workload is deployed to at least one of the multiple servers, wherein the workload implements: a component of the SDN architecture system, or at least one of an application that requires network configuration of the workload through the SDN architecture system.

20. The non-transitory computer readable medium of claim 19, wherein: The ready custom resource and the specification implementation: A pre-deployment test suite to verify that the cluster is suitable for the described SDN architecture system, and A post-deployment test suite is provided to verify the network controller operation and network data plane operation of the SDN architecture system on the cluster.

Images