Orchestration method, computing system and computer readable storage medium

Through network telemetry sensing scheduling technology, the performance degradation problem of containerized applications when network resources are insufficient is identified and solved, efficient scheduling of containerized applications and network resource optimization are achieved, and the performance and reliability of the overall system are improved.

CN119071301APending Publication Date: 2024-12-03HEWLETT PACKARD ENTERPRISE DEV LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311839507.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2023-12-28
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing containerized applications are difficult to effectively schedule and manage in network environments, especially when network resources are insufficient, resulting in application performance degradation, and the default scheduler cannot meet network QoS requirements, affecting overall application performance.

Method used

Through network telemetry sensing scheduling technology, the telemetry collector is used to obtain network data from worker nodes, analyze the system to establish an application performance baseline, and identify the cause of performance degradation based on the network telemetry data. The scheduler reschedules the workload to a node with sufficient network resources. Ensure that the application's minimum bandwidth or minimum latency requirements are met.

Benefits of technology

It realizes efficient scheduling of containerized applications and optimization of network resources, improves application performance, meets the application's network QoS requirements, and improves the reliability and efficiency of the overall system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071301A_ABST
    Figure CN119071301A_ABST
Patent Text Reader

Abstract

The invention relates to an orchestration method, a computing system and a computer readable storage medium. In general, techniques are described for deploying workloads of containerized services to worker nodes in a network using network telemetry aware scheduling. In an example, a method includes establishing, by a computing system, an application performance baseline for an application based on application performance data and network telemetry data; based on determining that the application performance has degraded from the application performance baseline, identifying, by the computing system, a correlation between an arrangement of a workload of the application to a first worker node of the plurality of worker nodes and the application performance data; and rescheduling, by the computing system, the workload to a second worker node of the plurality of worker nodes based on the identified correlation.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 325,760, filed on May 30, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to computing systems for executing and managing containerized applications. Background Art

[0004] Computer networks have become ubiquitous, and the number of network applications, network-connected devices, and types of network-connected devices are expanding rapidly. Such devices now include computers, smart phones, Internet of Things (IoT) devices, cars, medical devices, factory equipment, etc. End-user network-connected devices generally cannot directly access public networks such as the Internet. Instead, the end-user network device establishes a network connection with an access network, and the access network communicates with a core network connected to one or more packet data networks (PDNs) that provide services. There are currently several different types of access networks in use. Examples include radio access networks (RANs) as access networks of third generation partnership project (3GPP) networks, trusted and untrusted non-3GPP networks such as Wi-Fi or WiMAX networks, and fixed / wired networks such as digital subscriber lines (DSL), passive optical networks (PONs), and wired networks. The core network can be the core network of a mobile service provider network, such as a 3G, 4G / LTE, or 5G network.

[0005] In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and / or storage capacity to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment (such as networking and storage systems, redundant power supplies, and environmental controls). In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure throughout the world using user support equipment located in various physical hosting facilities.

[0006] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have a widely used virtualization environment in which virtual hosts such as virtual machines or containers (also referred to herein as virtual execution elements) are deployed and run on the underlying computing platform of physical computing devices.

[0007] Virtualization within a data center can provide several advantages. One advantage is that virtualization can provide significant improvements in efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical central processing unit (CPU), virtualization becomes easier and more efficient. A second advantage is that virtualization provides effective control over the computing infrastructure. As physical computing resources in computing environments such as cloud-based computing become replaceable resources, the provision and management of computing infrastructure becomes easier. As a result, enterprise information technology (IT) staff often prefer virtualized computing clusters in data centers due to their management advantages in addition to the efficiency and increased return on investment (ROI) provided by virtualization.

[0008] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable execution elements for applications that are isolated from each other and from the host. Since containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and run as a single lightweight package on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers provide a guarantee of continued operation from one computing environment to another virtual or physical environment.

[0009] Due to the inherent lightweight nature of containers, a single host can typically support more container instances than traditional virtual machines (VMs). These systems are characterized by being dynamic and ephemeral, as hosted services can quickly scale or adapt to new requirements. Typically, short-lived containers can be created and moved more efficiently than VMs, and they can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the requirements for container networking solutions: the network should be agile and scalable. In the same computing environment, VMs, containers, and bare metal servers may need to coexist and enable communication between multiple deployed applications. Container networks should also not be aware of working with multiple types of orchestration platforms for deploying containerized applications.

[0010] Managing the deployed computing infrastructure and the infrastructure for application execution may involve two main roles: (1) Orchestration - automating the deployment, scaling, and operation of applications across a cluster of hosts and providing computing infrastructure, which may include container-centric computing infrastructure; and (2) Network management - creating virtual networks in the network infrastructure to enable packetized communications among applications running on virtual computing instances such as containers or VMs and among applications running in traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the invention

[0011] In general, techniques are described for deploying workloads of containerized services to worker nodes in a network using network telemetry-aware scheduling. For example, a telemetry collector obtains network telemetry data from a network of worker nodes and / or from network nodes of the network. The telemetry collector also obtains application performance data of containerized services of applications executed on worker nodes, for example, using a service mesh layered across worker nodes. An analysis system processes the application performance data at runtime, and after establishing a baseline performance for the application, it can determine whether the application as a whole or the workload for the application specifically is experiencing performance degradation from the baseline. If the analysis system determines based on the network telemetry data that the application performance degradation is caused at least in part by insufficient network resources of the network or worker nodes of the network, the analysis system directs a scheduler for a container orchestration system (such as a Kubernetes platform) to modify the deployment of workloads for the application among the worker nodes.

[0012] For example, based on such an indication from the analysis system, a scheduler of the container orchestration system can process network telemetry data, node telemetry data, and / or application performance data to re-provision the workload to a different worker node. In some examples, the scheduler can determine that the current network telemetry data indicates that a minimum bandwidth or minimum latency requirement for the workload can be met by scheduling the workload to a specific node with sufficient network resource availability. The scheduler can therefore re-provision the workload to that specific node.

[0013] In one example, the present disclosure describes a method including: establishing, by a computing system, an application performance baseline for an application based on application performance data and network telemetry data; identifying, by the computing system, a correlation between placement of the application's workload to a first worker node among a plurality of worker nodes and the application performance data based on determining that application performance has degraded from the application performance baseline; and rescheduling, by the computing system, the workload to a second worker node among the plurality of worker nodes based on identifying the correlation.

[0014] In another example, the present disclosure describes a computing system that includes one or more storage devices and one or more programmable processors, the one or more programmable processors communicating with the one or more storage devices and configured to: establish an application performance baseline for an application based on application performance data and network telemetry data; identify a correlation between placement of an application workload to a first worker node among multiple worker nodes and the application performance data based on a determination that application performance has degraded from the application performance baseline; and reschedule the workload to a second worker node among the multiple worker nodes based on the identification of the correlation.

[0015] In another example, the present disclosure describes a non-transitory computer-readable storage medium comprising instructions that, when executed, configure one or more processors to: establish an application performance baseline for an application based on application performance data and network telemetry data; identify a correlation between placement of an application workload to a first worker node among multiple worker nodes and the application performance data based on a determination that application performance has degraded from the application performance baseline; and reschedule the workload to a second worker node among the multiple worker nodes based on the identification of the correlation.

[0016] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram illustrating an exemplary computing infrastructure in which examples of the techniques described herein may be implemented.

[0018] Figure 2 is a block diagram illustrating an example computing system in accordance with techniques described in this disclosure.

[0019] Figure 3 is a block diagram of an exemplary computing device including operations of an orchestrator for scheduling worker nodes.

[0020] Figure 4 is a flow diagram illustrating example operation of a computing infrastructure in accordance with one or more techniques of this disclosure.

[0021] Figure 5 is a block diagram illustrating an exemplary computing infrastructure in which examples of the techniques described herein may be implemented.

[0022] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0023] Figure 11 is a block diagram illustrating an exemplary computing infrastructure 100 in which examples of the techniques described herein may be implemented. In general, a data center 101 provides an operating environment for applications and services for a customer site 104 (illustrated as "customer 104") having one or more customer networks connected to the data center via a service provider network 106. For example, the data center 101 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 106 is connected to a public network 115, which may represent one or more networks managed by other providers, and thus may form part of a large-scale public network infrastructure (e.g., the Internet). For example, the public network 115 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 106, an enterprise IP network, or some combination thereof.

[0024] Although customer sites 104 and public network 115 are primarily shown and described as edge networks of service provider network 106, in some examples, one or more of customer sites 104 and public network 115 may be tenant networks within data center 10 or another data center. For example, data center 101 may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), where each VPN may implement one of customer sites 104.

[0025] The service provider network 106 provides packet-based connectivity to attached customer sites 104, data centers 101, and public networks 115. The service provider network 106 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 106 may implement multi-protocol label switching (MPLS) forwarding, and in such instances, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 106 represents multiple interconnected autonomous systems (such as the Internet) that provide services from one or more service providers.

[0026] In some examples, data center 101 may represent one of many geographically distributed data centers in which the techniques and systems described herein may be implemented. Figure 1As shown in the example of , data center 101 can be a facility that provides network services, cloud services, storage services and / or application services to customers. Data center 101 can represent a locally deployed data center, a private cloud, a public cloud, a hybrid cloud, or other types of deployments. Customers of a service provider can be collective entities such as enterprises and governments or individuals. For example, a data center can host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, science or supercomputing, etc. Although shown as a separate edge network of the service provider network 106, elements of the data center 101 such as one or more physical network functions (PNFs) or virtualized network functions (VNFs) can be included in the core of the service provider network 106.

[0027] The switch fabric 121 may include interconnected top-of-rack (TOR) (or other "leaf") switches coupled to a distribution layer of chassis (or "spine" or "core") switches. Although not shown, the data center 101 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cell phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 101 may also include one or more physical network functions (PNFs) (such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), evolved packet cores or other cellular network elements) and other PNFs.

[0028] The term "packet flow", "traffic flow", or simply "flow" refers to a collection of packets originating from a particular source device or endpoint and being sent to a particular destination device or endpoint. For example, a single flow of packets can be identified by the 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, the 5-tuple identifies the packet flow corresponding to the received packets. An n-tuple refers to any n items extracted from the 5-tuple. For example, a 2-tuple for a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for that packet.

[0029] Any server in the data center 101 may be configured with a virtual execution element by virtualizing the server's resources to provide isolation among one or more processes (applications) executing on the server. "Hypervisor-based", or "hardware-level", or "platform" virtualization refers to the creation of virtual machines that each include a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is visualized from the physical hardware of the host server, the executing application is isolated from both the host's hardware and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0030] A virtual network is a logical construct implemented on top of a physical network. A virtual network can be used to replace VLAN-based isolation and provide multi-tenancy in a virtualized data center (e.g., data center 101). Each tenant or application can have one or more virtual networks. Unless explicitly allowed by security policy, each virtual network can be isolated from all other virtual networks.

[0031] Virtual networks can use data center edge routers ( Figure 1 The virtual network is connected to a physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and an Ethernet virtual private network (EVPN) network and extends across the physical multi-protocol label switching (MPLS) layer 3 virtual private network (L3VPN) and the Ethernet virtual private network (EVPN) network. The virtual network can also be used to implement network function virtualization (NFV) and service linking.

[0032] Various mechanisms can be used to implement virtual networks. For example, each virtual network can be implemented as a virtual local area network (VLAN), a virtual private network (VPN), etc. A virtual network can also be implemented using two networks, a physical underlying network consisting of a switching structure 121 and a virtual overlay network. The role of the physical underlying network is to provide an "IP structure" that provides a unicast IP connection from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network can provide a unified low-latency, non-blocking, high-bandwidth connection from any point in the network to any other point in the network.

[0033] As further described below, a virtual router running in the kernel or hypervisor of a router in data center 101 can create a virtual overlay network on top of a physical underlying network using its own dynamic "tunnel" grid. For example, these overlay tunnels can be MPLS over GRE / UDP tunnels, or VXLAN tunnels, or NVGRE tunnels. For virtual machines or other virtual execution elements, the underlying physical routers and switches may not contain any state for each tenant, such as any media access control (MAC) addresses, IP addresses, or policies. For example, the forwarding tables of the underlying physical routers and switches may only contain the IP prefix or MAC address of the server. (The gateway router or switch that connects the virtual network to the physical network is an exception and may contain tenant MAC or IP addresses.)

[0034] The virtual routers of the servers of data center 101 typically contain state for each tenant. For example, a virtual router may contain a separate forwarding table (routing instance) for each virtual network. The forwarding table contains the IP prefixes (in the case of a layer 3 overlay) or MAC addresses (in the case of a layer 2 overlay) of the virtual machines or other virtual execution elements (e.g., a container pool of containers). A single virtual router does not need to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router only needs to contain those routing instances that exist locally on the server (i.e., it has at least one virtual execution element that exists on the server.)

[0035] The control plane protocol between the control plane nodes of the network controller or physical gateway router (or switch) can be BGP (and can be Netconf for management). This is the same control plane protocol that can also be used for MPLS L3VPN and MPLS EVPN. For example, the protocol between the network controller and the virtual router can be based on XMPP. The pattern of the messages exchanged over XMPP can conform to "BGP-Signaled End-System IP / VPNs", draft-ietf-l3vpn-end-system-06, Mackie et al., December 15, 2016, all of which is incorporated herein by reference.

[0036] "Container-based" or "operating system" virtualization refers to the virtualization of an operating system running multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers such as those provided by the open source DOCKER container application or by CoreOS Rkt ("Rocket"). Like virtual machines, each container is virtualized and can be kept isolated from the host and other containers. However, unlike virtual machines, each container can omit a single operating system and only provide a suite of applications and application-specific libraries. Typically, containers are run by a host as isolated user-space instances and the containers can share an operating system and common libraries with other containers executing on the host. As a result, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on a corresponding virtual network.

[0037] In some examples, the container is managed by its host kernel to allow the use of namespace isolation functions in some cases to limit and prioritize resources (CPU, memory, block I / O, network, etc.) without starting any virtual machines. The command space isolation function allows the application (e.g., a given container) view of the operating environment to be completely isolated, including process trees, networks, user identifiers, and mounted file systems. In some examples, the container can be deployed according to Linux containers (LXC), an operating system-level virtualization method that runs multiple isolated Linux systems (containers) on a control host using a single Linux kernel. LXC is an operating system-level virtualization method that runs multiple isolated Linux systems (containers) on a single control host (LXC host). LXC does not use virtual machines (although LXC can be hosted by virtual machines). Instead, LXC uses a virtual environment with its own CPU, memory, block I / O, network and / or other resource spaces. The LXC resource control mechanism is provided by namespaces and cgroups in the Linux kernel on the LXC host. Additional information about containers is found in "Docker Overview" Docker, Inc., available at docs.docker.com / engine / understanding-docker, last accessed on July 9, 2016. Additional examples of containerization approaches include OpenVZ, FreeBSD jails, AIX workload partitions, and Solaris containers. Thus, as used herein, the term "container" may encompass not only LXC-style containers, but also any one or more of a virtualization engine, a virtual private server, a silo, or a jail.

[0038] exist Figure 1In the example of , data center 101 includes storage and / or computing servers interconnected via a switching fabric 121 provided by one or more layers of physical network switches and routers, with worker nodes 110A to 110N ("worker nodes 110" herein) depicted as being connected to the switching fabric 121. Worker nodes 110 may be bare metal machines and / or virtual machines within data center 101, and may also be referred to herein as "hosts" or "host devices." Worker nodes 110 may represent computing devices, such as x86 processor-based servers, configured to operate according to the techniques described herein.

[0039] Worker nodes 110 host virtual network endpoints of one or more virtual networks operating on top of a physical network represented here by switch fabric 121. Although primarily described with respect to data center based switch networks, other physical networks such as service provider network 106 may underlie one or more virtual networks.

[0040] Each of the worker nodes 110 can host one or more virtual execution elements, such as each having at least one virtual network endpoint of one or more virtual networks configured in the physical network. A virtual network endpoint of a virtual network can represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint can be a virtual machine, a collection of one or more containers (e.g., a container pool), or another other virtual execution element such as a layer 3 endpoint of a virtual network. The term "virtual execution element" covers virtual machines, containers, and other virtualized computing resources that provide at least partially independent execution environments for applications. The term "virtual execution element" can also cover a container pool of one or more containers. Figure 1 As shown, worker node 110A hosts a virtual network endpoint with one or more containers. However, given the hardware resource limitations of worker node 110, worker node 110 may execute as many virtual execution elements as are practical. Each of the virtual network endpoints may use one or more virtual network interfaces to perform packet I / O or otherwise process packets. For example, a virtual network endpoint may use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by NIC 113A to perform packet I / O and receive / send packets on one or more communication links with switch fabric 121. Other examples of virtual network interfaces are described below.

[0041] The worker nodes 110 each include at least one network interface card (NIC) 113, which is composed of at least one interface for exchanging packets with the switching structure 121 through a communication link. For example, the worker node 110A includes a NIC 113A. Any of the NICs 113 can provide one or more virtual hardware components for virtualized input / output (I / O). The virtual hardware component for I / O can be a virtualization of a physical NIC 113 ("physical function"). For example, in the single root I / O virtualization (SR-IOV) described in the peripheral component interface special interest group SR-IOV specification, the PCIe physical function of the network interface card (or "network adapter") is virtualized to provide one or more virtual network interfaces as "virtual functions" used by the corresponding endpoints executed on the worker node 110. In this way, virtual network endpoints can share the same PCIe physical hardware resources, and virtual functions are examples of virtual hardware components.

[0042] NIC 113A may represent a library, plugin, module, runtime, or other executable code for worker node 110A. NIC 113A may be at least partially consistent with the container network interface (CNI) specification or rkt network proposal. NIC 113A may represent a Contrail or OpenContrail network plugin. Alternatively, NIC 113A may be referred to as a network plugin or a CNI plugin or a CNI instance. For the purposes of the container networking interface (CNI) specification, a container may be considered to be synchronized with a Linux network namespace. The corresponding unit depends on the specific container runtime implementation: for example, in an implementation of an application container specification such as rkt, each container pool runs in a unique network namespace. However, in Docker, there is typically a network namespace for each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that can be uniquely addressed and can communicate with each other. This can be a single container, a machine / server (real or virtual), or some other network device (e.g., a router). Containers may be conceptually added to or removed from one or more networks.

[0043] As another example, one or more of the servers executing worker node 110 may implement Virtio (a paravirtualization framework) available, for example, for a Linux operating system, which provides emulated NIC functionality as a type of virtual hardware component to provide a virtual network interface to a virtual network endpoint. As another example, one or more of the servers may implement Open vSwitch to perform distributed virtual multi-layer switching between one or more virtual NICs (vNICs) of a hosted virtual machine, where such a vNIC may also represent a type of virtual hardware component that provides a virtual network interface to a virtual network endpoint. In some instances, the virtual hardware component is a virtual I / O (e.g., NIC) component. In some instances, the virtual hardware component is an SR-IOV virtual function.

[0044] In some examples, any worker node 110 may implement a Linux bridge that emulates a hardware bridge and forwards packets among the server's virtual network interfaces or between the server's virtual network interfaces and the server's physical network interfaces. For Docker implementations of containers hosted by servers, a Linux bridge or other operating system bridge that executes on the server and switches packets among containers may be referred to as a "Docker bridge." As used herein, the term "virtual router" may encompass Open vSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and / or software that are located on a host device and that perform switching, bridging, or routing packets among virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more of the servers 12.

[0045] Any NIC 113 may include an internal device switch that switches data between virtual hardware components associated with the NIC. For example, for a NIC that supports SR-IOV, the internal device switch may be a virtual Ethernet bridge (VEB) that switches between SR-IOV virtual functions and, correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. Alternatively, the internal device switch may be referred to as a NIC switch, or, for an SR-IOV implementation, may be referred to as an SR-IOV NIC switch. The virtual hardware components associated with NIC 113A may be associated with a layer 2 destination address assigned by NIC 113A or a software process responsible for configuring NIC 113A. Physical hardware components (or "physical functions" for SR-IOV implementations) are also associated with a layer 2 destination address.

[0046] To switch data between virtual hardware components associated with NIC 113A, the internal device switch may perform layer 2 forwarding to switch or bridge layer 2 packets between the virtual hardware components and the physical hardware components for NIC 113 A. Each virtual hardware component may be located on a virtual local area network (VLAN) of a virtual network for a virtual network endpoint that uses the virtual hardware component for I / O.

[0047] One or more of the worker nodes 110 may each include an agent 135 that manages one or more workloads (such as workload 122A) on each worker node. The agent 135 may execute one or more routing instances for corresponding virtual networks within the data center 101 to provide virtual network interfaces and route packets among virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual routing and forwarding instance (VRF) of an Internet Protocol-Virtual Private Network (IP-VPN). For example, a packet received by the virtual router of the worker node 110A from the underlying physical network structure (i.e., the switching structure 12) of the data center 101 may include an outer header to allow the physical network structure to tunnel the payload or "inner packet" to the physical network address of the network interface card 113A of the server executing the virtual router. The outer header may include not only the physical network address of the network interface card 113A of the worker node 110A, but also a virtual network identifier that identifies one of the virtual networks and the corresponding routing instance executed by the virtual router, such as a VxLAN tag or a multi-protocol label switching (MPLS) label. The inner packet includes an inner header having a destination network address that conforms to the virtual network addressing space of the virtual network identified by the virtual network identifier.

[0048] The virtual router of worker node 110 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the tunnel encapsulation header for the packet, and forwards the packet to the appropriate destination virtual network endpoint for the packet. For example, for worker node 110A, for each of the packets outgoing from the virtual network endpoint (e.g., workload 122A) hosted by worker node 110A, the virtual router appends a tunnel encapsulation header indicating the virtual network to the packet to generate an encapsulated or "tunnel" packet, and the virtual router outputs the encapsulated packet to another node (such as another one of worker nodes 110) via the overlay tunnel for the virtual network. As used herein, a virtual router can perform operations of a tunnel endpoint to encapsulate internal packets originating from a virtual network endpoint to generate tunnel packets, and decapsulate tunnel packets to obtain internal packets for routing to other virtual network endpoints.

[0049] The computing infrastructure 100 implements an automation platform for automating the deployment, expansion, and operation of virtual execution elements across worker nodes 110 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration platform that provides a container-centric infrastructure to automate the deployment, expansion, and operation of containers, thereby providing a container-centric infrastructure. In the context of virtualized computing infrastructure, "orchestration" generally refers to the provision, scheduling, and management of virtual execution elements and / or applications and services executed on such virtual execution elements by the host server available to the orchestration platform. Specifically, for example, container orchestration permits container coordination and refers to the deployment, management, expansion, and configuration of containers to host servers by the container orchestration platform. Exemplary instances of orchestration platforms include Kubernetes, Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0050] The elements of the automation platform of the computing infrastructure 100 include at least worker nodes 110, an orchestrator 130, a telemetry collector 142, an analysis system 140, and a UI device 129. Virtual execution elements can be deployed to a virtualized environment using a cluster-based framework in which a cluster master node of a cluster manages the deployment and operation of containers to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" used herein cover different orchestration platform terms that distinguish between the main management element of the cluster and the analog device of the main virtual execution element hosting device of the cluster. For example, the Kubernetes platform uses the terms "cluster master" and "slave node", while the Docker Swarm platform refers to the cluster manager and cluster nodes.

[0051] Orchestrator 130 implements scheduler 148 for computing infrastructure 100. Orchestrator 130 can be a distributed or centralized application executed on one or more computing devices of a computing system. Orchestrator 130 can implement a respective master node for one or more clusters, each of which has one or more slave nodes implemented by one or more servers of computing infrastructure 100. Additional information about orchestrator 130 operating in conjunction with other devices of data center 101 or other software defined networks is found in International Application No. PCT / US2013 / 044378, filed on June 5, 2013, and entitled “PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET Flows,” U.S. Patent Application No. 14 / 226,509, filed on March 26, 2014, and entitled “Tunneled Packet Aggregation for Virtual Networks,” and U.S. Patent Application No. 17 / 305,110, filed on June 30, 2021, and entitled “Network Controller Horizontal Scaling for Network Device Configurations Session Management,” each of which is incorporated by reference as if fully set forth herein. U.S. Patent Application No. 14 / 226,509 also includes further description of virtual routers such as virtual router 21A.

[0052] Typically, the orchestrator 130 controls the deployment, expansion, and operation of virtual execution elements across a cluster of servers and provides a computing infrastructure, which may include a container-centric computing infrastructure. The orchestrator 130 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud can provide a virtualized infrastructure to the container management platform. The orchestrator 130 may represent any of the orchestration platforms listed above, such as Kubernetes.

[0053] Each of workloads 122A to 122N is an example of a virtual network endpoint and a virtual execution element. Workloads 122 may each represent or include one or more containers deployed by a container orchestration system. One or more of workloads 122 may together implement a distributed application. Each of workloads 122 may provide or implement one or more services, and in the case where workload 122 represents a container pool or other container deployment, one or more services are containerized services or "microservices."

[0054] In Kubernetes, a container pool is a group of one or more logically related containers ( Figure 1 ), shared storage devices for containers, and options on how to run containers. In the case of being instantiated for execution, the container pool may alternatively be referred to as a "container pool replica". Containers of a container pool are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of a container pool can be a collection of Linux namespaces, cgroups, and other isolation aspects. In the context of a container pool, individual applications may further apply sub-isolation. Typically, containers within a container pool have a common IP address and port space and are able to detect each other via the local host. Because they have a shared context, containers within a container pool also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. Typically, containers that are members of different container pools have different IP addresses, and containers cannot communicate via IPC in the absence of a configuration to enable this feature. Instead, containers that are members of different container pools typically communicate with each other via the container pool IP address.

[0055] In Kubernetes, by default, all container pools can communicate with all other container pools without using network address translation (NAT). In some cases, the orchestrator 130 creates a service virtual network and a container pool virtual network shared by all namespaces, and allocates service and container pool network addresses from the namespaces, respectively. In some cases, all container pools generated in the Kubernetes cluster in all namespaces can communicate with each other, and the network addresses of all container pools can be allocated from the container pool subnet specified by the orchestrator 130. When a user creates an isolated namespace for a container pool, the orchestrator 130 can create a new container pool virtual network and a new shared service virtual network for the new isolated namespace. The container pool generated in the Kubernetes cluster in the isolated namespace extracts the network address from the new container pool virtual network, and the corresponding service of such a container pool extracts the network address from the new service virtual network.

[0056] Further examples of technology involving virtual execution elements are described in U.S. application Ser. No. 16 / 118,107, filed on Aug. 20, 2018, for MULTIPLE VIRTUAL NETWORK INTERFACE SUPPORT FOR VIRTUAL EXECUTION ELEMENTS, and U.S. application Ser. No. 16 / 118,731, filed on Aug. 31, 2018, for UNIFIED CONTROL PLANE FOR NESTED CLUSTERS IN AVIRTUALIZED COMPUTING INFRASTRUCTURE, the entire contents of each of which are incorporated herein by reference.

[0057] The orchestrator 130 includes a scheduler 148 to schedule workloads 122A to 122N (collectively referred to as "workloads 122") to the worker nodes 110. In general, the scheduler 148 can manage the placement of each of the workloads 122 to the worker nodes 110 according to the scheduling policy, the amount of resources requested for the workload, and the available resources of the worker nodes 110. The worker node resources considered by the scheduler 148 when allocating workloads 122 to the worker nodes 110 include CPU-related resources (e.g., cores, CPU / core utilization), memory-related resources (available main memory, such as 2GB), temporary storage devices, and user-defined extension resources. In Kubernetes, the scheduler is called the kube-scheduler.

[0058] Application workloads 122 may have different performance requirements that need to be met in a highly dynamic application execution environment. In such an environment, application performance is a product of the dynamics of different resources, such as: worker node resources; network resources (e.g., bandwidth, latency, loss, jitter, firewall policies), network policies, and communication graphs among different services of a distributed application; and the performance of external services, such as authentication and external cloud services.

[0059] In the application execution environment, the performance of different applications and the underlying network is affected by the way different workloads of application services are scheduled among worker nodes. Applications may have different QoS and resource requirements that need to be met when they are scheduled across worker nodes 122, which may be distributed within a data center 101 or across WAN boundaries, such as the WAN boundaries of networks 106, 115. Considering an example in which two microservices communicate frequently, it may be desirable to collocate the workloads 122 corresponding to these services within the same worker node 110 or within a group of worker nodes 110 within the same availability zone to ensure acceptable network QoS (such as bandwidth, latency, loss, and / or jitter). Similarly, if the application microservices have specific requirements from a hardware perspective (e.g., SSD, GPU), these specific workloads may need to be allocated on specific nodes 110 with these capabilities.

[0060] In Kubernetes, the kube-scheduler is the main controller component responsible for supporting multiple such different scheduling requirements for different application microservices. Each application developer can specify detailed resource requirements based on their best understanding of the Kubernetes cluster resources and the expected application behavior. However, due to the highly dynamic nature of the Kubernetes environment and considering that multiple applications can co-exist in the same environment, there is no guarantee that the kube-scheduler can meet all application-specific requirements. For example, due to high CPU and memory requirements of one application, other co-existing applications may have problems on the cluster. The following are two key limitations of the Kubernetes default scheduler that may affect application and network performance:

[0061] 1. The default Kubernetes scheduler schedules container pools based only on their CPU and memory resource requirements. Therefore, network QoS requirements (such as bandwidth or latency requirements)

[0062] Remain agnostic while making scheduling decisions. This may lead to a situation where a worker node 110 can meet the CPU and memory requirements of all container pools, but network bandwidth may become a bottleneck causing performance degradation.

[0063] 2. In addition, the default Kubernetes scheduler makes decisions based on the static resource requirements of the container pool. The default Kubernetes scheduler will reserve the requested CPU and memory resources on the worker node 110, even when the actual usage of the CPU and memory resources is very low. There is no concept of using real-time usage telemetry of resources to make better scheduling decisions to improve resource efficiency and meet network QoS requirements.

[0064] According to the techniques of the present disclosure, scheduler 148 schedules workloads 122 to worker nodes 110 using network telemetry-aware scheduling. Scheduler 148 can reschedule one or more workloads 122 comprising an application to different worker nodes in worker nodes 110 based on an indication that the performance of the application has fallen below a baseline due to the placement of one or more of the workloads 122. Thus, scheduler 148 can address the limitations of the default scheduler described above and schedule workloads 122 in a manner that is sensitive to network QoS requirements so as to support not only a single workload 122 but, at least in some cases, the entire distributed application. In some exemplary instances, scheduler 148 can also or alternatively schedule (or reschedule) workloads 122 in a manner that takes into account dynamic resource usage of workloads 122 over time.

[0065] The computing infrastructure 100 may include a telemetry collector 142. The telemetry collector 142 may obtain network data from the worker nodes 110 via one or more sources, such as an agent 135 executing on the worker nodes 110. The telemetry collector 142 may include service telemetry 144, network telemetry 145, and node telemetry 146.

[0066] The service telemetry 144 may include network data, such as application-level telemetry, from a service grid of the computing infrastructure 100. The computing infrastructure 100 may include a service grid for managing worker nodes 110 and workloads 122. The telemetry collector 142 may obtain network data from the service grid for use in scheduling the workloads 122 to the worker nodes 110. In addition, the telemetry collector 142 may use the service telemetry 144 to obtain data about application performance of one or more applications, including one or more workloads scheduled to the worker nodes 110.

[0067] The telemetry collector 142 may obtain network telemetry data from the network of the computing infrastructure 100 as network telemetry 145. The network telemetry 145 may include network telemetry data, such as data about the state of the network (such as the switch fabric 121) that interconnects the worker nodes 110. The network telemetry 145 may also include data describing the interconnections between computing devices of the computing infrastructure 100, such as data describing the connections between the worker nodes in the worker nodes 110 and the intermediate connections between the worker nodes 110. The node telemetry 146 may include node telemetry data obtained from the worker nodes 110. For example, the node telemetry 146 may include data such as resource availability (e.g., processor and memory availability) of a single worker node in the worker nodes 110 and resource utilization of a single worker node and one or more clusters of worker nodes 110. The node telemetry 146 may include data such as network performance of the worker nodes 110 and the workloads 122 scheduled to the worker nodes 110. For example, the node telemetry 146 may include data about one or more network performance characteristics of the worker nodes and workloads, such as round trip time (RTT), latency, available bandwidth, network resource usage, etc. In addition, the telemetry collector 142 may receive quality of service (QoS) requirements for the workload of the application. The telemetry collector may receive QoS requirements such as minimum available bandwidth, RTT requirements, maximum latency requirements, and reliability requirements.

[0068] Telemetry collector 142 may provide collected network telemetry to analysis system 140 for analysis. Analysis system 140 may analyze the network telemetry and identify where application and network performance has degraded. As part of identifying degradation in application performance, analysis system 140 may use data such as service telemetry 144 to understand the current distribution of application workloads across worker nodes 110. Additionally, analysis system 140 may use node telemetry 146 to determine a baseline of network performance for each of worker nodes 110 for a given workload, such as an application's workload.

[0069] The analysis system 140 uses service telemetry 144 to identify baseline communication dependencies across workloads such as microservices. The analysis system 140 can use this determination to determine the latency and bandwidth profile of each link in the service mesh for the switch fabric 121. The analysis system 140 can use data from the service mesh acting as a data plane collected by the telemetry collector 142.

[0070] Analysis system 140 may establish a baseline of network performance for worker nodes 110. Analysis system 140 may determine a baseline of network performance for each worker node based on data obtained from telemetry collector 142, which may represent the network performance of the worker node when the worker node is healthy (e.g., when the network performance of the worker node does not affect the performance of a workload scheduled to the worker node).

[0071] The analysis system 140 can use data from the telemetry collector 142 to obtain application and network performance visibility in the Kubernetes environment using multiple data sources such as K8s cluster resource usage, service mesh for application performance visibility, and active detection for underlying network visibility. The analysis system 140 can determine the baseline performance of the scheduler 148 on application and network performance. The analysis system 140 can understand the current distribution of the application container pool across worker nodes. In addition, the analysis system 140 can use the service mesh telemetry to determine the baseline network performance of each worker node for a given workload and the baseline communication dependencies across multiple microservices. The analysis system 140 can determine the latency and bandwidth profile for each link in the service mesh. In the service mesh, the analysis system 140 can receive application service telemetry exported to the telemetry collector 142 from the data plane. The analysis system 140 can determine the correlation between the default scheduler arrangement and the corresponding impact on the application telemetry. The analysis system 140 can correlate each link in the service mesh with the underlying network path. This may require flow-level telemetry in the underlying layer to determine the path between two worker nodes between which two services communicate in the service mesh. Once the path information is available, the baseline state of each intermediate hop can be determined. In some examples, the analysis system 140 can use in-band telemetry (INT) when available on the worker nodes, which can then be used to determine the end-to-end network path between the worker nodes.

[0072] A worker node 110 or a subset of worker nodes 110 in a network path may support INT. This may provide end-to-end and per-hop latency between any two worker nodes for further root cause analysis.

[0073] The analysis system 140 may additionally establish baselines for application-level, node-level, and network-level telemetry. The analysis system 140 may include an analysis data pipeline that consumes real-time telemetry from one or more sources. The analysis system 140 may generate a trigger signal during any performance degradation to the scheduler 148, which may re-provision the workload to other worker nodes while taking into account the current health of existing worker nodes. In addition to the scheduler 148, there may be other consumers of the trigger signals generated by the data pipeline. Some possible consumers that may utilize these real-time performance feedback are:

[0074] a. Traffic management in a service mesh based on existing application performance. A service mesh controller can apply different traffic management policies, such as load balancing behavior can be changed from the default round-robin to location-aware, service load-aware load balancing. These decisions can be made based on real-time service performance visibility and continuous analysis of collected data. Service mesh traffic management policies can be triggered based on outlier detection policies, which are currently statically configured. Real-time analysis of application performance can enable fine-tuning of these outlier detection policies on a per-service basis, and can be dynamically applied to make adaptive traffic management decisions.

[0075] b. Cloud-native SD-WAN provision based on real-time application and network performance feedback.

[0076] Analysis system 140 may determine one or more correlations between placement of workloads to worker nodes in worker nodes 110 and data received from the service mesh consistent with degraded application performance based on data acquired by telemetry collector 142. For example, analysis system 140 may determine a correlation between placement of workload 122A to worker node 110A and service telemetry 144 that indicates degradation in performance of an application including workload 122A. In another example, analysis system 140 determines that network performance of a worker node has degraded from a baseline of network performance of the worker node. Analysis system 140 may identify a correlation based in part on a determination of degraded network performance of the worker node.

[0077] The analysis system 140 may obtain data about network performance of a plurality of worker nodes of the network and identify a worker node among the plurality of worker nodes that is executing any workload of the application. The analysis system 140 may determine, based on the data about network performance and the identified worker nodes, degraded network performance of the application caused by the network performance of the identified worker nodes, and cause the scheduler 148 to reschedule based on the determined degraded network performance of the application.

[0078] Analysis system 140 may establish a network performance baseline for the plurality of worker nodes and identify correlations based at least in part on the data regarding network performance and the network performance baseline.

[0079] Analysis system 140 may use data about the intermediate connections between worker nodes 110 to identify correlations. For example, analysis system 140 may obtain data about the intermediate connections from telemetry collector 142. Analysis system may determine, based on the intermediate connections and based on the data about the network performance of worker nodes 110, degraded network performance of the application caused by the network performance of the intermediate connections between worker nodes 110. Analysis system 140 may provide, to orchestrator 130, a determination for rescheduling one or more workloads of the application based on the determined degraded network performance of the application caused by the network performance of the intermediate connections.

[0080] Analysis system 140 may use data about application performance to determine whether the performance of one or more applications has degraded from an application performance baseline established when the application was first scheduled. Analysis system 140 may identify a correlation between the degradation of application performance and the placement of application workloads to worker nodes. Additionally, analysis system 140 may determine that one or more QoS requirements of the application workloads are not being met.

[0081] In an example, an application that schedules a workload to three worker nodes begins to respond to requests slowly and occasionally times out. Analysis system 140 may determine that the performance of the application has dropped below a baseline. Analysis system 140 analyzes data from telemetry collector 142 in response to the determination. Analysis system 140 may use data from telemetry collector to identify a correlation between application performance and the deployment of one or more of workloads 122 to worker nodes. For example, analysis system 140 may determine that one of worker nodes 122 has a network performance characteristic that affects the performance of the application. Analysis system 140 generates an indication of the correlation and provides the indication to orchestrator 130. Orchestrator 130 identifies a replacement worker node based on the indication and causes scheduler 148 to reschedule the workload executed on the problematic worker node to the replacement worker node. In another example, analysis system 140 generates an indication that one or more QoS requirements are not met. Orchestrator 130 may reschedule one or more workloads based on the QoS requirements and the indication based on the indication.

[0082] exist Figure 1In an example of , orchestrator 130 schedules workload 122 to worker node 110 using telemetry-aware scheduling. For example, telemetry collector 142 obtains network telemetry 145 from a network of worker node 110 and / or from a network node of a network (such as switch fabric 121). Telemetry collector 142 may, for example, obtain application performance data of containerized services of an application executed on worker node 110 using a service mesh layered across worker node 110. Analysis system 142 may process the application performance data at runtime and, after establishing a baseline performance for the application, determine whether the application as a whole or a workload for the application specifically (such as workload 122A) is experiencing performance degradation from the baseline. If analysis system 142 determines based on network telemetry 145 that the application performance degradation is caused at least in part by insufficient network resources of the network or worker node 110 of the network, analysis system 142 directs scheduler 148 of orchestrator 130 to modify the deployment of workloads for the application among worker nodes.

[0083] For example, scheduler 148 of orchestrator 130 may process network telemetry 145, node telemetry 146, and / or application performance data from service telemetry 144 based on such indications from analysis system 140 to re-provision the workload to a different worker node in worker nodes 110. In some examples, scheduler 148 may determine that the current network telemetry data indicates that a minimum bandwidth or minimum latency requirement for the workload can be met by scheduling the workload to a specific node with sufficient network resource availability. Scheduler 148 may therefore re-provision the workload to that specific node (such as worker node 110B in worker nodes 110).

[0084] Scheduler 148 can be an extension to the default Kubernetes scheduler to take into account latency and bandwidth requirements when scheduling a pool of containers across worker nodes, such as worker node 110. Scheduler 148 can use strategically placed round trip time (RTT) labels to determine whether a particular service is suitable for deployment based on the target location specified on the container pool configuration file. Node selection can be based on minimization of RTT, which depends on the target location for the service after the filtering step is completed by scheduler 148. Additionally, in terms of bandwidth, scheduler 148 can check whether the best candidate node has sufficient bandwidth to support a given service based on a bandwidth requirement label.

[0085] The scheduler 148 may schedule workloads based on one or more metrics of real-time performance telemetry across applications and underlying networks, such as application bandwidth requirements, application latency requirements, real-time and historical network telemetry from worker nodes 110 regarding current and historical use of network bandwidth, and real-time and historical application-level telemetry from the service mesh (e.g., service telemetry 144). The scheduler 148 may schedule based on active probing between nodes to determine node-to-node latency and jitter and use that to schedule new container pools or reschedule existing container pools. The telemetry collector 542 may obtain the probing information from the NIC 113 and provide it to the scheduler 148 via the analysis system 140 for use, wherein the NIC 113 has additional intelligence (e.g., smart NIC) that enables the NIC 113 to probe connections between worker nodes 110. The scheduler 148 may correlate network-aware scheduling with corresponding data from the telemetry collector 142 to verify improvements in application performance.

[0086] UI device 129 can enable administrator 128 to request to reschedule workload 122 among worker nodes 110. UI device 129 can be one or more computing devices, such as a server, laptop, desktop computer, tablet computer, or other computing device. UI device 129 can receive data from orchestrator 130, such as a list of current workloads 122 and their status on worker nodes 110. UI device 129 can display data to administrator 128 via one or more components such as a touch-sensitive display. UI device can receive input from administrator 128 and provide data about the input to orchestrator 130.

[0087] Figure 2 202 is a block diagram illustrating an exemplary computing system according to the techniques described in the present disclosure. Computing system 202 may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, appliances, cloud computing systems, and / or other computing systems that may be capable of performing operations and / or functions described according to one or more aspects of the present disclosure. In some examples, computing system 202 represents a cloud computing system, a server farm, and / or a server cluster (or a portion thereof) that provides services to other devices or systems. In other examples, computing system 202 may represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, a server farm, a data center, and / or a server cluster.

[0088] exist Figure 2In the example of , computing system 202 may include one or more processors 213, communication unit 215, one or more input devices 217, one or more output devices 218, and one or more storage devices of storage system 205. Storage system 205 includes analysis system 140, telemetry collector 142, and orchestrator 130. One or more of the devices, modules, storage areas, or other components of computing system 202 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connections may be provided through a communication channel, which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for transferring data.

[0089] One or more of the processors 213 may implement functions associated with the computing system 202 or associated with one or more modules shown herein and / or described below and / or execute instructions associated with them. One or more of the processors 213 may be a processing circuit system that performs operations according to one or more aspects of the present disclosure, may be a part of the processing circuit system, and / or may include the processing circuit system. Examples of processors 213 include microprocessors, application processors, display controllers, auxiliary processors, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. The computing system 202 may use one or more processors 213 to perform operations according to one or more aspects of the present disclosure using software, hardware, firmware, or a combination of hardware, software, and firmware resident in and / or executed at the computing system 202.

[0090] One or more communication units 215 of computing system 202 may communicate with devices external to computing system 202 by sending and / or receiving data, and in some aspects may operate as both an input device and an output device. In some examples, communication unit 215 may communicate with other devices over a network. In other examples, communication unit 215 may send and / or receive radio signals over a radio network (such as a cellular radio network). In other examples, communication unit 215 of computing system 202 may send and / or receive satellite signals over a satellite network. Examples of communication unit 215 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of communication unit 215 may include a device that is capable of transmitting information over a network. GPS, NFC, Devices for communicating with cellular networks (e.g., 3G, 4G, 5G), and devices found in mobile devices radio and Universal Serial Bus (USB) controllers, etc. Such communications may comply with, implement or adhere to appropriate protocols, including Transmission Control Protocol / Internet Protocol (TCP / IP), Ethernet, NFC or other technologies or protocols.

[0091] The one or more input devices 217 may represent any input device of the computing system 202 that is not described separately herein. The input device 217 may generate, receive, and / or process input. For example, the one or more input devices 217 may generate input or receive input from a network, a user input device, or any other type of device for detecting input from a person or a machine.

[0092] One or more output devices 218 may represent any output device of computing system 202 that is not described separately herein. Output device 218 may generate, present, and / or process output. For example, one or more output devices 218 may generate, present, and / or process output in any form. Output device 218 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical, or other output. Some devices may be used as both input devices and output devices. For example, a communication device may send data to other systems or devices over a network and receive data from other systems or devices.

[0093] One or more storage devices of the storage system 205 within the computing system 202 may store information for processing during operation of the computing system 202. The storage system 205 may store program instructions and / or data associated with one or more of the modules described according to one or more aspects of the present disclosure. One or more processors 213 and one or more storage devices may provide an operating environment or platform for such modules, which may be implemented as software, but in some examples may include any combination of hardware, firmware, and software. One or more processors 213 may execute instructions, and one or more storage devices of the storage system 105 may store instructions and / or data of one or more modules. The combination of the processor 213 and the storage system 205 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The processor 213 and / or the storage device of the storage system 205 may also be operably connected to one or more other software and / or hardware components, including but not limited to one or more of the components of the computing system 202 and / or one or more devices or systems shown as being connected to the computing system 202.

[0094] The telemetry collector 142 may be obtained from the Figure 1The telemetry collector 142 can provide the network and application performance data to the analysis system 140 for processing and comparing with the baseline of network and application performance. The telemetry collector 140 can obtain network telemetry data such as available bandwidth, network link utilization, per-hop latency among multiple worker nodes, historical network telemetry data, node latency, node packet loss, and node jitter.

[0095] Analysis system 140 may generate a baseline of application performance. Analysis system 140 may generate a baseline of application performance when an application is first deployed to one or more of worker nodes 110. Analysis system 140 may store the baseline of application performance for use in determining whether the performance of the application has degraded from when the application was initially deployed.

[0096] Analysis system 140 may determine that the performance of the application has degraded from a baseline of application performance. Analysis system 140 may compare data about the performance of the application with a stored performance baseline to determine whether the performance of the application has degraded (eg, the application is experiencing higher latency).

[0097] Analysis system 140 may compare data regarding degradation of application performance with network performance data acquired by telemetry collector 142 to identify one or more correlations between network performance and application performance. Analysis system 140 may associate the network performance data with a plurality of worker nodes, such as worker node 110. Analysis system 140 may determine a network performance indicator for the plurality of worker nodes based on the association, wherein the network performance indicator indicates degraded network performance of the plurality of worker nodes. Analysis system 140 may identify a correlation between placement of an application's workload to a first worker node and the application performance data. For example, analysis system 140 may identify a correlation between a first worker node and a workload of an application. Figure 1 The correlation between the placement of workload 122A to worker node 110A and the performance of the application incorporating workload 122A is shown in FIG.

[0098] Analysis system 140 may identify a correlation between the performance degradation of the application and the identification of the fault in the network. Analysis system 140 may obtain an indicator of dependencies between multiple resources in the network and event dependencies between multiple network events and one or more of the multiple resources, and receive an indication of the fault in the network. Analysis system 140 may identify a second correlation between the fault, the placement of the second workload to the third worker node, and the application performance data based on identifying that the application performance has degraded from the application performance baseline. Analysis system 140 may provide an indication to reschedule the second workload to orchestrator 130 to reschedule the second workload based on the correlation.

[0099] Analysis system 140 may identify the correlation based on one or more factors. Analysis system 140 may identify the correlation based in part on one or more of: communication dependencies between multiple worker nodes, network data from service mesh telemetry, end-to-end network paths among multiple worker nodes, network data from multiple worker nodes, network data from probes of connections utilized by multiple worker nodes, and network data from network elements between multiple worker nodes. Analysis system 140 may additionally establish a baseline of network performance and identify the correlation based in part on the baseline of network performance.

[0100] Analysis system 140 may utilize a map of connections between worker nodes to determine a correlation between degraded application performance and placement of workloads to worker nodes. Analysis system 140 may receive data about intermediate connections between worker nodes (e.g., virtual routers, network paths, etc.) and generate a map of connections between worker nodes. Analysis system 140 may identify a correlation between placement of workloads and application performance data based on an association between the map of intermediate connections and network performance indicators.

[0101] Analysis system 140 can provide an indicator of the dependency to orchestrator 130 based on the identification of the dependency. Orchestrator 130 can use the indicator to determine that the application needs to be reallocated. Orchestrator 130 can utilize data about the configuration of computing infrastructure 100 (e.g., which nodes have capacity for additional workloads) to determine to reallocate the application workload to a node different from the node currently executing the application workload.

[0102] The analysis system 140 can identify critical and non-critical workloads of the application. The analysis system 140 can identify a workload as critical or non-critical based on one or more factors, such as whether a particular workload affects the functionality and performance of the application. In addition, the application executing on the worker node 110 can provide data to the analysis system 140, which classifies the workload of the application as critical or non-critical. The analysis system 140 can use the classification of critical and non-critical to determine whether the workload needs to be rescheduled to a different worker node. In an example, the analysis system 140 reschedules the workload based on determining that the workload of the application is a critical workload.

[0103] Analysis system 140 may maintain scheduling of non-critical workloads to worker nodes that are experiencing degraded network performance. In an example, analysis system 140 may determine that a non-critical workload of an application is deployed to a worker node with degraded network performance. Analysis system 140 may determine that there is no correlation between the placement of the non-critical workload and the application performance data (i.e., for example, there is not enough correlation to justify rescheduling in view of a threshold), and maintain the non-critical workload scheduled to the worker node with degraded application performance. In an example, analysis system 140 determines that a non-critical workload of an application is deployed to a worker node (such as worker node 110) among a plurality of worker nodes with degraded network performance. Analysis system 140 determines that there is no correlation between the placement of the non-critical workload and the application performance data, and maintains the non-critical workload as scheduled to the worker node with degraded network performance.

[0104] Analysis system 140 may obtain an indicator of dependencies between multiple resources in a network, such as switch fabric 121, and event dependencies between multiple network events and one or more of the multiple resources. Analysis system 140 may receive an indication of a fault in the network from telemetry collector 142, and based on identifying that application performance has degraded from an application performance baseline, identify a second correlation between the fault, placement of the second workload to a third worker node, and application performance data. Analysis system 140 may provide an indication to orchestrator 130 to reschedule based on the second correlation.

[0105] The orchestrator 130 may represent a container orchestration platform. The orchestrator 130 may utilize the scheduler 148 to reschedule workloads among the worker nodes. The scheduler 148 may use data about the worker nodes to identify one or more candidate worker nodes that, if the workload of the application is assigned to the candidate worker nodes, will enable the application to meet the baseline of the application performance. The scheduler 148 may identify the candidate worker nodes based on one or more criteria to ensure that once the workload is assigned to the candidate worker nodes, the application performance baseline is met. In addition, the scheduler 148 may reschedule the workload based on the QoS requirements of the workload.

[0106] Scheduler 148 may reschedule the workload of the application to the candidate worker node based on the identification of the candidate worker node. Scheduler 148 may reschedule the workload based on one or more factors, such as whether the candidate worker node has sufficient resources to support the scheduling of the workload. In addition, scheduler 148 may keep non-critical workloads on the worker nodes instead of reallocating non-critical workloads in order to optimize network resources for critical workloads.

[0107] In the example, telemetry collector 142 receives data about network performance associated with workload 122A from worker node 110A. Analysis system 140 determines, based on the received data about workload 122A, that a degradation in performance of the application is associated with network performance of workload 122A. Analysis system 140 further determines that workload 122A is a critical workload for the application and provides an indication to scheduler 148 to reschedule workload 122A to a different worker node in worker nodes 110.

[0108] In another example, the telemetry collector 142 receives network performance data consistent with the workload 122A experiencing degraded network performance. The analysis system 140 obtains the network data from the telemetry collector 142 and determines that the workload 122A is a non-critical workload for the associated application. The analysis system 140 avoids providing an indication to reschedule the workload 122A because the workload is a non-critical workload whose degraded network performance does not affect the performance of the application.

[0109] The container including the implementation simulator 125 and the test agent 140 can be deployed to a virtualized environment using a cluster-based framework in which a cluster master node of the cluster manages the deployment and operation of the container to one or more cluster slave nodes of the cluster. The server 112 or a virtual machine thereon can represent a cluster node.

[0110] Figure 3 is a block diagram of an exemplary computing device including operations of an orchestrator for scheduling worker nodes.

[0111] exist Figure 3 In the example of , computing device 400 includes a bus 442 that connects hardware components of the computing device 400 hardware environment. Bus 442 connects a network interface card (NIC) 430, a storage disk 446, and one or more processors 410 (hereinafter referred to as "processor 410"). In some cases, the front-side bus can connect the processor 410 and the memory device 444. In some examples, bus 442 can connect the memory device 444, the processor 410, and the NIC 430. Bus 442 can represent a high-speed peripheral component interface (PCI) (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers among components connected to bus 442. In some examples, components connected to bus 442 control DMA transfers among components connected to bus 442.

[0112] The microprocessor 410 may include one or more processors, each of which includes an independent execution unit that executes instructions consistent with an instruction set architecture, instructions stored to a storage medium. The execution units may be implemented as separate integrated circuits (ICs) or may be combined within one or more multi-core processors (or "multi-core" processors) that are each implemented using a single IC (i.e., a chip multiprocessor).

[0113] Disk 446 represents a computer-readable storage medium, which includes volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by processor 410.

[0114] The main memory 444 includes one or more computer-readable storage media, which may include random access memory (RAM) such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store the required program code and program data in the form of instructions or data structures and accessed by the computer. The main memory 444 provides a physical address space consisting of addressable memory locations.

[0115] The network interface card (NIC) 430 includes one or more interfaces 432 configured to exchange packets using the links of the underlying physical network. The interface 432 may include a port interface card having one or more network ports. The NIC 430 may also include, for example, on-card memory for storing packet data. Direct memory access transfers between the NIC 430 and other devices coupled to the bus 442 may be read from the NIC memory, or may be written to the NIC memory.

[0116] Memory 444, NIC 430, storage disk 446, and processor 410 can provide an operating environment for a software stack including an operating system kernel 414 executed in kernel space. For example, kernel 414 can represent Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft. In some instances, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Exemplary hypervisors include a kernel-based virtual machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor can cover a virtual machine manager (VMM). An operating system including kernel 414 provides an execution environment for one or more processes in user space 445. Kernel 414 includes a physical driver 425, which provides an interface for use by components of computing device 400 using network interface card 430.

[0117] The computing device 400 may be coupled to a physical network switch fabric (e.g., via the NIC 430) that includes an overlay network that extends the switch fabric from physical switches to software or "virtual" routers of physical servers coupled to the switch fabric. The computing device 400 may configure the worker nodes 110 of the cluster using one or more dedicated virtual networks. Figure 1 ).

[0118] The computing device 400 may include an API server 214, a default scheduler 215, a custom scheduler 216, and a controller manager 210, each of which includes code that can be executed by a processor as one or more computer processes. Typically, the API server 214 may call the scheduler 215 or the custom scheduler 216 to schedule the worker node 110 ( Figure 1 ) on a virtual execution element on a container pool. If the container pool specification of the container pool specifies that a custom scheduler 216 is to be used to schedule containers, the custom scheduler 216 may select a worker node to schedule the virtual execution element according to the techniques described herein. For example, the custom scheduler 216 may select a worker node based on the best match between a node profile and a tuning parameter with a virtual element characteristic (such as a container pool characteristic in the container pool specification). The API server 214 may call the orchestrator 130 ( Figure 1 ), which can be used for the scheduler 148 ( Figure 1) obtains the virtual execution element from the storage server and creates the virtual execution element on the worker node. The orchestrator 130 for the selected worker node can update the state of the virtual execution element to the API server 214, which persists the new state to the database 218. In this way, the computing device 400 instantiates a new virtual execution element in the computing infrastructure 100.

[0119] Computing device 400 may include instances of orchestrator 130, analysis system 140, and telemetry collector 142. Orchestrator 130, analysis system 140, and telemetry collector 142 may provide similar functionality to that of previous figures (such as Figure 1 ) as shown in the accompanying drawings.

[0120] Figure 4 is a flow diagram illustrating example operation of a computing infrastructure in accordance with one or more techniques of this disclosure.

[0121] Arrangers (such as Figure 1 The orchestrator 130 (shown) deploys the application to a worker node (such as worker node 110A). The orchestrator 130 establishes a baseline of application performance based on the application performance data and the network telemetry data (450). The orchestrator 130 may use network data (such as node telemetry data, service mesh data, and network telemetry data) to establish a baseline of application performance.

[0122] The analysis system 140 may determine that the application performance has degraded from a baseline of the application performance. The analysis system 140 identifies one or more workloads of the application to a first worker node (such as a worker node) of the plurality of worker nodes based on determining that the application performance has degraded from the baseline. Figure 1 10A) and the application performance data. Analysis system 140 may identify the correlation based on one or more factors. For example, analysis system 140 may identify the correlation based on factors such as degraded network performance between the first worker node and other worker nodes.

[0123] Analysis system 140 reschedules the workload to a second worker node among the plurality of worker nodes based on the identification of the correlation (454). Analysis system 140 may select the second worker node based on one or more criteria, such as a level of available resources of the second worker node, performance statistics of the second worker node, and other metrics.

[0124] Figure 5 is a block diagram illustrating an exemplary computing infrastructure in which examples of the techniques described herein may be implemented.

[0125] Figure 5A computing infrastructure 500 is shown having Figure 1 Elements similar to the elements of the computing infrastructure 100 shown in , but in the context of dynamically providing software defined networking (SD-WAN) resources in a wide area network among worker nodes 510A to 510N (hereinafter referred to as "worker nodes 510") that communicate via WAN 521.

[0126] Network providers and enterprises can use SD-WAN to manage network connections among distributed locations, such as remote branch offices or central offices or data centers. SD-WAN extends software-defined networking (SDN) to enable organizations to quickly and efficiently create connections over a WAN (such as WAN 521), which can include the Internet or other transport networks that provide various WAN connection types, such as Multiprotocol Label Switching (MPLS)-based connections, mobile network connections (e.g., 3G, Long Term Evolution (LTE), 5G), Asymmetric Digital Subscriber Line (ADSL), cable broadband, etc. Such connections are often referred to as "WAN links." SD-WAN is considered a connectivity solution that utilizes the above or other WAN connection types through a WAN link as an overlay layer on top of traditional WAN access.

[0127] WAN 521 provides a network for geographically distributed worker nodes 510 to communicate. In some examples, data center 521 can represent a network that interconnects many geographically distributed network data centers. WAN 521 can include one or more types of network switches and non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices.

[0128] exist Figure 5 In the example of , worker node 510 can be a bare metal machine and / or a virtual machine in various data centers interconnected via WAN 521. Worker node 510 can include one or more components, such as, in the example of worker node 510A, NIC 513A, agent 535A, and workload 522A. NIC 513A-513N (hereinafter referred to as "NIC 513") can include one or more internal devices for switching data between virtual hardware components associated with NIC 513. NIC 513A can enable worker node 510A to communicate with other worker nodes of worker node 510 via WAN 521.

[0129] Telemetry collector 542 may obtain network data from one or more sources, such as worker node 510. Telemetry collector 542 may obtain network data, such as service telemetry 544, network telemetry 545, and node telemetry 546. Telemetry collector 542 may provide network data to analysis system 540 for analysis. Analysis system 540 may determine that one or more applications executing on a worker node, such as worker node 510, have fallen below a performance threshold. Analysis system 540 may provide an indication of degraded application performance to orchestrator 530 via interface 532 for orchestrator 530 to process.

[0130] exist Figure 5 In the example of FIG. 5 , computing infrastructure 500 includes SD-WAN client 550 and SD-WAN controller 570. SD-WAN client 550 may enable orchestrator 530 to provide a request to modify a WAN link to SD-WAN controller 570. SD-WAN client 550 may include one or more components, such as dedicated network hardware (e.g., a network card in a server) or software components of orchestrator 530 (e.g., an application plug-in, an application that interfaces with orchestrator 530, etc.).

[0131] The SD-WAN controller 570 may include or represent a network service controller (NSC). The SD-WAN controller 570 may manage the SD-WAN edge to create and operate end-to-end SD-WAN managed services. In addition, the SD-WAN controller 570 may provide topology lifecycle management capabilities. In some examples, the SD-WAN controller 570 is a cloud-native SD-WAN controller implemented using workloads deployed by the orchestrator 530. In some examples, the SD-WAN controller 570 may enable the worker nodes 510 to communicate via a WAN link of the WAN 521 as an overlay layer on top of traditional WAN access utilizing the above or other WAN connection types. Additional information of an exemplary SD-WAN controller 570, virtual routers, and virtual router agents is found in U.S. Patent No. 11,381,474, issued on July 5, 2022 and entitled “WAN LINK SELECTION FOR SD-WAN SERVICES,” which is incorporated by reference as if fully set forth herein.

[0132] Different enterprise applications in Kubernetes may require connectivity from WAN 521 for reliable operation, with strict QoS requirements for communications among different worker nodes 510 interconnected via WAN 521. SD-WAN controller 570 may enable cloud-native applications to provide dedicated traffic policies to the SD-WAN to meet their QoS requirements.

[0133] Based on the data collected by telemetry collector 542 and the QoS requirements of one or more applications with workloads (such as workload 522) scheduled to worker node 510, analysis system 540 can determine that one or more QoS requirements of the one or more applications are not met. Analysis system 540 can receive QoS requirements for the one or more applications, the QoS requirements defining one or more performance metrics that the applications need to meet. Analysis system 540 can provide an indication to orchestrator 530 to reschedule the workload in workload 522 based on the determination. Orchestrator 530 can cause scheduler 548 to reschedule the workload based on the QoS requirements based on the indication.

[0134] The analysis system 540 identifies a correlation between the placement of the workload of the application to the worker node 510A, the provisioned WAN link of WAN 521 to the first worker node in the SD-WAN, and the determination that the one or more quality of service (QoS) requirements are not met based on the determination that the workload of the application placed to the first worker node, such as worker node 510A, is not met. The analysis system 540 can provide an indication to the orchestrator 530 to determine that the orchestrator 530 re-provisions the WAN link to the worker node 510A based on the correlation and the one or more QoS requirements. The orchestrator 530 can cause the SD-WAN controller 570 to re-provision the WAN link of WAN 521 to the worker node 510A.

[0135] The orchestrator 530 may communicate with the SD-WAN controller 570 via the SD-WAN client 550. The orchestrator 530 may use the application performance visibility in the Kubernetes environment to continuously monitor the QoS received by the application and use it as input to the SD-WAN controller 570 to initiate existing network policies in its SD-WAN environment. Due to the highly dynamic nature of container pool allocation in a Kubernetes cluster, the SD-WAN controller 570 may be required to dynamically provision WAN links for the SD-WAN as needed. Whenever a container pool (such as one of the workloads 522) is reassigned to a different worker node (such as one of the worker nodes 510), the orchestrator 530 using the SD-WAN client 550 may instruct the SD-WAN controller 570 to update / reprovision the WAN links within the WAN 521 to meet the QoS. The SD-WAN controller 570 includes an interface through which the SD-WAN client 550 may request to configure or reprovision WAN links across the WAN 521 to interconnect the workloads 510 via the WAN 521.

[0136] The SD-WAN controller 570 may use the SD-WAN client 550 to create and manage links between worker nodes 510 across the WAN 521 based on requests from the orchestrator 530. For example, the SD-WAN controller may provision a WAN link to a workload based on a request from the orchestrator 530 to create a link. In some examples, the orchestrator 530 may request the SD-WAN controller 570 to re-provision a WAN link to improve the reliability of the link between two of the worker nodes 510 based on a determination that the workload has been rescheduled to geographically distributed worker nodes, and / or based on an identification of a correlation between placement of an application workload to two worker nodes and degradation of application performance. In some examples, the SD-WAN controller 570 may dynamically provision SD-WAN resources based on one or more considerations (such as a request from the orchestrator 530 to re-provision SD-WAN resources) and in response to scheduling of workloads to worker nodes 510, depending on the location of such worker nodes 510. The SD-WAN controller 570 may provide links between worker nodes 510 to enable worker nodes in geographically distributed worker nodes 510 to communicate among themselves.

[0137] The orchestrator 530 may re-provision the SD-WAN resources to the worker nodes using the SD-WAN controller 570. The orchestrator 530 may request that the SD-WAN controller 570 re-provision the SD-WAN resources (e.g., WAN links or update the SD-WAN policy to modify the WAN links for traffic to / from the worker nodes) to the worker nodes based on the indication of the correlation between the application performance and / or placement of the workload on the worker nodes. The orchestrator 530 may provide an indication to the SD-WAN client 550 of the orchestrator 530 to output the request to the SD-WAN controller 570.

[0138] The SD-WAN controller 570 may allocate WAN links across the WAN 521 to the worker nodes 510 based on one or more requirements such as QoS or minimum quality of experience (QoE) metrics based on a request for provisioning WAN links from the orchestrator 530 via the SD-WAN client 550. The QoS and / or QoE metrics may include metrics such as maximum latency, maximum RTT, minimum bandwidth, and other metrics. Users may expect their applications to experience connections with acceptable levels of quality, commonly referred to as quality of experience (QoE) and QoS. QoE / QoS may be measured based on various performance metrics of the link, including latency, delay (inter-frame gap), jitter, packet loss, and / or throughput. A user may define one or more metrics of QoE / QoS that the user expects in a service contract (e.g., a service level agreement (SLA)) with a service provider. SLA metrics are typically user-configurable values ​​and are derived through a trial-and-error approach or a benchmarking environment relative to the user experience or real-world best application metrics.

[0139] The UI device 529 may enable the administrator 528 to request that one or more workloads 522 be rescheduled among the worker nodes 510. Additionally, the UI device 529 may enable the administrator 528 to request that the WAN link among the worker nodes 510 be re-provisioned by the SD-WAN controller 570.

[0140] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit system may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0141] If implemented in hardware, the present disclosure may be directed to a device such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause one or more processors to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by one or more processors.

[0142] The computer readable medium may form part of a computer program product that may include packaging materials. The computer readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer readable storage media. The computer readable storage media may be distributed among multiple packages, devices, or other components capable of being configured with computer instructions.

[0143] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., stored in RAM or cache).

[0144] The code or instructions may be software and / or firmware executed by a processing circuit system including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A compilation method, comprising: Establishing an application performance baseline for the application based on the application performance data and the network telemetry data by the computing system; identifying, by the computing system, a correlation between placement of a workload of the application to a first worker node of a plurality of worker nodes and the application performance data based on determining that application performance has degraded from the application performance baseline; as well as Based on identifying the dependency, the workload is rescheduled, by the computing system, to a second worker node among the plurality of worker nodes.

2. The method according to claim 1, further comprising: acquiring, by the computing system, data regarding network performance of the plurality of worker nodes of a network; identifying, by the computing system, a worker node among the plurality of worker nodes that is executing any workload of the application; as well as determining, by the computing system and based on the data regarding network performance and the identified worker nodes, degraded network performance of the application caused by network performance of the identified worker nodes, Wherein the rescheduling is based on a determined degraded network performance of the application.

3. The method according to claim 2, further comprising: establishing, by the computing system, a network performance baseline for the plurality of worker nodes, Wherein identifying the correlation is based at least in part on the data regarding network performance and the network performance baseline.

4. The method according to claim 2, further comprising: acquiring, by the computing system, data regarding intermediate connections between the plurality of worker nodes; as well as determining, by the computing system and based on the intermediate connection and based on the data regarding network performance of the plurality of worker nodes, degraded network performance of the application caused by network performance of the intermediate connection between the plurality of worker nodes, and Wherein the rescheduling is based on a determined degraded network performance of the application caused by the network performance of the intermediate connection.

5. The method according to claim 1, further comprising: The computing system obtains a quality of service (QoS) requirement for the workload of the application; as well as determining, by the computing system, that a QoS requirement of the workload is not met, The rescheduling includes: rescheduling the workload based on the QoS requirement.

6. The method according to any one of claims 1 to 5, wherein: The network telemetry data includes at least one of the following: Available bandwidth; Network link utilization; a per-hop latency among the plurality of worker nodes; Historical network telemetry data; Node delay; Node packet loss; or Node jitter.

7. The method according to any one of claims 1 to 5, wherein: The computing system identifies the correlation based on at least one of: communication dependencies between the plurality of worker nodes; Network data from service mesh telemetry; an end-to-end network path among the plurality of worker nodes; network data from the plurality of worker nodes; network data from probes of connections used by the plurality of worker nodes; or Network data from a network element of a network interconnecting the plurality of worker nodes.

8. The method according to any one of claims 1 to 5, wherein: Rescheduling the workload includes rescheduling the workload based on determining that the workload of the application is a critical workload.

9. The method according to any one of claims 1 to 5, further comprising: Determining, by the computing system, to deploy a non-critical workload of the application to a worker node having degraded network performance among the plurality of worker nodes; determining, by the computing system, that there is no correlation between the placement of the non-critical workload and the application performance data; as well as The non-critical workload is maintained by the computing system to be scheduled to the worker node with degraded network performance.

10. The method according to any one of claims 1 to 5, wherein: The correlation is a first correlation and comprises: obtaining, by the computing system, dependencies between a plurality of resources in the network and indicators of event dependencies between a plurality of network events and one or more of the plurality of resources; receiving, by the computing system, an indication of a fault in the network; identifying, by the computing system, a second correlation between the fault, placement of a second workload to a third worker node, and the application performance data based on identifying that application performance has degraded from the application performance baseline; and The rescheduling includes: rescheduling based on the second correlation.

11. The method according to any one of claims 1 to 5, further comprising: Based on the rescheduling and determining that one or more quality of service (QoS) requirements for the application are not met by a wide area network (WAN) link, which will include at least two interconnections among the multiple worker nodes of the first worker node, the computing system outputs a request to a software defined networking (SD-WAN) controller in the WAN to configure the SD-WAN to use a WAN link that meets the QoS requirements for the application to transmit traffic associated with the first worker node.

12. A computing system comprising one or more storage devices and one or more programmable processors, the one or more programmable processors communicating with the one or more storage devices and configured to: Establish application performance baselines for applications based on application performance data and network telemetry data; identifying a correlation between placement of a workload of the application to a first worker node of a plurality of worker nodes and the application performance data based on a determination that application performance has degraded from the application performance baseline; as well as Based on the identification of the dependency, the workload is rescheduled to a second worker node among the plurality of worker nodes.

13. The computing system of claim 12, wherein: The one or more programmable processors are configured to: obtaining data regarding network performance of the plurality of worker nodes of the network; identifying a worker node among the plurality of worker nodes that is executing any workload of the application; as well as determining, based on the data regarding network performance and the identified worker nodes, degraded network performance of the application caused by network performance of the identified worker nodes, Wherein, to reschedule the workload, the one or more programmable processors are configured to reschedule the workload based on the determined degraded network performance of the application.

14. The computing system of claim 13, wherein: The one or more programmable processors are configured to: establishing a network performance baseline for the plurality of worker nodes, Wherein, to identify the correlation, the one or more programmable processors are configured to identify the correlation based at least in part on the data regarding network performance and the network performance baseline.

15. The computing system of claim 13, wherein: The one or more programmable processors are configured to: obtaining data regarding intermediate connections between the plurality of worker nodes; and determining, based on the intermediate connection and based on the data regarding the network performance of the plurality of worker nodes, degraded network performance of the application caused by network performance of the intermediate connection between the plurality of worker nodes, and Wherein, to reschedule the workload, the one or more programmable processors are configured to reschedule the workload based on a determined degraded network performance of the application caused by the network performance of the intermediate connection.

16. The computing system of claim 12, wherein: The one or more programmable processors are configured to: Obtaining a quality of service (QoS) requirement for the workload of the application; and determining that the QoS requirement of the workload is not met, Wherein, in order to reschedule the workload, the one or more programmable processors are configured to reschedule the workload based on the QoS requirement.

17. A computing system according to any one of claims 12 to 16, wherein: Network telemetry data includes at least one of the following: Available bandwidth; Network link utilization; a per-hop latency among the plurality of worker nodes; Historical network telemetry data; Node delay; Node packet loss; or Node jitter.

18. The computing system of any one of claims 12 to 16, wherein: The computing system is configured to identify the correlation based on at least one of: communication dependencies between the plurality of worker nodes; Network data from service mesh telemetry; an end-to-end network path among the plurality of worker nodes; network data from the plurality of worker nodes; network data from probes of connections used by the plurality of worker nodes; or Network data from a network element of a network interconnecting the plurality of worker nodes.

19. The computing system of any one of claims 12 to 16, wherein: To reschedule the workload, the one or more programmable processors are configured to reschedule the workload based on a determination that the workload of the application is a critical workload.

20. The computing system of any one of claims 12 to 16, wherein: The one or more programmable processors are configured to: determining to deploy a non-critical workload of the application to a worker node with degraded network performance among the plurality of worker nodes; determining that there is no correlation between the placement of the non-critical workload and the application performance data; as well as The non-critical workload is kept scheduled to the worker node with degraded network performance.

21. A computing system according to any one of claims 12 to 16, in, The correlation is a first correlation, and Wherein, the one or more programmable processors are configured to: obtaining dependencies between a plurality of resources in the network and indicators of event dependencies between a plurality of network events and one or more of the plurality of resources; receiving an indication of a fault in the network; and identifying a second correlation between the fault, placement of a second workload to a third worker node, and the application performance data based on a determination that application performance has degraded from the application performance baseline; Wherein, to reschedule the workload, the one or more programmable processors are configured to reschedule the workload based on the second dependency.

22. The computing system of any one of claims 12 to 16, wherein: The one or more programmable processors are further configured to: Based on the rescheduling and determining that one or more quality of service (QoS) requirements for the application are not met by a wide area network (WAN) link, the wide area network (WAN) link will include at least two interconnections among the multiple worker nodes of the first worker node, outputting a request to a software defined networking (SD-WAN) controller in the WAN to configure the SD-WAN to use a WAN link that meets the QoS requirements for the application to transmit traffic associated with the first worker node.

23. The computing system of any one of claims 12 to 16, wherein: The one or more programmable processors are configured to: Based on a determination that the workload of the application deployed to the first worker node does not meet one or more quality of service (QoS) requirements, identifying a correlation between deployment of the workload of the application to the first worker node, a provided wide area network (WAN) link to the first worker node in a software defined WAN (SD-WAN), and the determination that one or more QoS requirements are not met; as well as Based on the correlation and the one or more QoS requirements, the WAN link to the first worker node is re-provisioned.

24. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to be configured to execute the arrangement method of any one of claims 1 to 11 or to be configured as a computing system of any one of claims 12 to 23.