Distributed application call path performance analysis

Abstract

In general, distributed application call path performance analysis is provided. Specifically, techniques for managing a distributed application based on a call path across an underlying network infrastructure between a plurality of services of the distributed application. In an example, a method includes determining, by a computing system, for a distributed application implemented using a plurality of services, a call path from an ingress endpoint service of the plurality of services to a termination endpoint service of the plurality of services; determining, by the computing system, a corresponding network path for each pair of neighboring services of the plurality of pairs of services communicating for the invoking path; and performing, by the computing system, one or more of: reconfiguring the network, or redeploying one of the plurality of services to a different one of the computing nodes, based on the performance metric of the network device for the corresponding network path reaching the threshold.

Description

[0001] This application claims the benefit of U.S. Patent Application No. 18 / 478,260, filed on September 29, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to computing systems and, more particularly, to managing distributed applications operating over a network. Background Art

[0003] Computer networks have become ubiquitous, and the number of network applications, network-connected devices, and types of network-connected devices are expanding rapidly. Such devices now include computers, smart phones, Internet of Things (IoT) devices, cars, medical devices, factory equipment, etc. End-user network-connected devices generally cannot directly access public networks such as the Internet. Instead, the end-user network device establishes a network connection with an access network, and the access network communicates with a core network connected to one or more packet data networks (PDNs) that provide services. There are currently several different types of access networks in use. Examples include radio access networks (RANs) as access networks of third-generation partnership project (3GPP) networks, trusted and untrusted non-3GPP networks such as Wi-Fi or WiMAX networks, and fixed / wired networks such as digital subscriber lines (DSL), passive optical networks (PONs), and wired networks. The core network can be the core network of a mobile service provider network, such as a 3G, 4G / LTE, or 5G network.

[0004] In a typical cloud data center environment, there is a large collection of interconnected servers that provide computing and / or storage capacity to run various applications. For example, a data center may include facilities that host applications and services for users (i.e., customers of the data center). For example, a data center may host all infrastructure equipment, such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, storage system clusters and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide infrastructure throughout the world using user support equipment located in various physical hosting facilities.

[0005] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have a widely used virtualized environment in which virtual hosts such as virtual machines or containers (also referred to herein as virtual execution elements or workloads) are deployed and run on the underlying computing platform of physical computing devices. Workloads may also include bare metal processes.

[0006] Virtualization within a data center can provide several advantages. One advantage is that virtualization can provide significant improvements in efficiency. As the underlying physical computing devices (i.e., servers) become increasingly powerful with the advent of multi-core microprocessor architectures with a large number of cores per physical central processing unit (CPU), virtualization becomes easier and more efficient. A second advantage is that virtualization provides effective control over the computing infrastructure. As physical computing resources in computing environments such as cloud-based computing become fungible resources, the provision and management of computing infrastructure becomes easier. As a result, enterprise information technology (IT) staff often prefer virtualized computing clusters in data centers due to their management advantages, in addition to the efficiency and increased return on investment (ROI) provided by virtualization.

[0007] Containerization is a virtualization solution based on operating system-level virtualization. Containers are lightweight and portable workloads for applications that are isolated from each other and from the host. Since containers are not tightly coupled to the host hardware computing environment, applications can be bound to container images and run as a single lightweight grouping on any host or virtual host that supports the underlying container architecture. Therefore, containers solve the problem of how to make software work in different computing environments. Containers provide a guarantee of continued operation from one computing environment to another virtual or physical environment.

[0008] Due to the inherent lightweight nature of containers, a single host can typically support more container instances than traditional virtual machines (VMs). These systems are characterized by being dynamic and ephemeral, as hosted services can quickly scale or adapt to new requirements. Typically, short-lived containers can be created and moved more efficiently than VMs, and they can also be managed as groups of logically related elements (sometimes called "pods" for some orchestration platforms, such as Kubernetes). These container characteristics affect the requirements for container networking solutions: the network should be agile and scalable. In the same computing environment, VMs, containers, and bare metal servers may need to coexist and enable communication between multiple deployed applications. Container networks should also be aware of working with multiple types of orchestration platforms used to deploy containerized applications.

[0009] Managing the deployed computing infrastructure and the infrastructure for application execution may involve two main roles: (1) Orchestration - used to automate the deployment, scaling, and operation of applications across a cluster of hosts and to provide computing infrastructure, which may include container-centric computing infrastructure; and (2) Network management - used to create virtual networks in the network infrastructure to enable packetized communications between applications running on virtual computing instances such as containers or VMs and between applications running in traditional (e.g., physical) environments. Software-defined networking facilitates network management. Summary of the invention

[0010] In general, techniques for managing distributed applications based on call paths between multiple services of the distributed application through an underlying network infrastructure are described. The services of the distributed application call other services and form a call chain from an entry endpoint service to a termination endpoint service. This end-to-end call chain is called a call path, and the call path that contributes the most latency to the end-to-end behavior is the critical path of the distributed application. For each pair of services along the call path, there is a corresponding network path for the call path between the pair of services.

[0011] In an example of the described technology, an analysis system processes telemetry data obtained from a computing node executing a service and network flow data obtained from a network device. The analysis system processes the telemetry data and the network flow data to identify a network path for each pair of adjacent services in a service that communicates for a call path. A network device that processes service traffic (including service calls) for a call path may exhibit degraded performance, such as increased latency, reduced transmission speeds, reduced data throughput, dropped packets, and other effects on the performance and functionality of the network device because it affects service traffic. Network devices that exhibit degraded performance may affect service traffic between service pairs and, by extension, the end-to-end performance of distributed applications along the call path. If the analysis system determines that a network device along a network path is experiencing degraded performance, the analysis system may reconfigure the network or instruct a scheduler (such as a Kubernetes scheduler) to modify the service deployment of a distributed application.

[0012] The techniques of the present invention may provide one or more technical advantages that may be implemented in one or more practical applications. Existing solutions for analyzing the impact of network performance degradation on distributed applications attempt to track the impact of degraded network devices on distributed application services. In contrast, an analysis system that applies the techniques described herein identifies end-to-end call paths and associates these paths with underlying network paths. The critical paths of distributed applications can themselves be indicators of network performance issues due to their contribution to application latency. By focusing the analysis on certain call paths in an application-aware manner, the analysis system can focus network analysis on components of the network that contribute to application latency, and can more quickly and reliably identify and improve negative impacts on application performance.

[0013] In one example, the present disclosure describes a method, including: determining, by a computing system, a call path from an entry endpoint service among the multiple services to a termination endpoint service among the multiple services for a distributed application implemented using multiple services executed on computing nodes interconnected by a network of network devices; determining, by the computing system, a corresponding network path for each pair of adjacent services among multiple pairs of services that communicate with respect to the call path; and based on a performance indicator of a network device corresponding to the network path reaching a threshold, performing, by the computing system, one or more of the following operations: reconfiguring the network, or redeploying one of the multiple services to a different computing node among the computing nodes.

[0014] In one example, the present disclosure describes a computing system comprising: a memory and a processing circuit that communicates with the memory and is configured to: determine, for a distributed application implemented using multiple services executed on computing nodes interconnected by a network of network devices, a call path from an entry endpoint service among the multiple services to a termination endpoint service among the multiple services; determine a corresponding network path for each pair of adjacent services among multiple pairs of services that communicate with respect to the call path; and based on a performance indicator of the network device of the corresponding network path reaching a threshold, perform one or more of the following operations: reconfigure the network, or redeploy one of the multiple services to a different computing node among the computing nodes.

[0015] In one example, the present disclosure describes a non-transitory computer-readable storage medium comprising instructions that, when executed, cause one or more processors to: determine, for a distributed application implemented using multiple services executed on computing nodes interconnected by a network of network devices, a call path from an entry endpoint service among the multiple services to a termination endpoint service among the multiple services; determine a corresponding network path for each pair of adjacent services among multiple pairs of services that communicate with respect to the call path; and, based on a performance indicator of the network device corresponding to the network path reaching a threshold, perform one or more of the following operations: reconfigure the network, or redeploy one of the multiple services to a different computing node among the computing nodes.

[0016] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram illustrating an example computing infrastructure in which examples of the techniques described herein may be implemented.

[0018] Figure 2 is a block diagram illustrating an example computing system in accordance with techniques described in this disclosure.

[0019] FIG. 3A to FIG. 3C is a block diagram illustrating an example diagram in accordance with one or more techniques of this disclosure.

[0020] Figure 4 is a flow diagram illustrating example operation of a computing system in accordance with one or more techniques of this disclosure.

[0021] Figure 5 is a flow diagram illustrating example operation of a computing system in accordance with one or more techniques of this disclosure.

[0022] Like reference numerals refer to like elements throughout the specification and drawings. DETAILED DESCRIPTION

[0023] Figure 1 1 is a block diagram of an example computing infrastructure 100 illustrating an example in which the techniques described herein may be implemented. In general, a data center 101 provides an operating environment for applications and services for a customer site 104 (illustrated as "customer 104") having one or more customer networks coupled to the data center via a service provider network 106. For example, the data center 101 may host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. The service provider network 106 is coupled to a public network 115, which may represent one or more networks managed by other providers, and thus may form part of a large-scale public network infrastructure (e.g., the Internet). For example, the public network 115 may represent a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the service provider network 106, an enterprise IP network, or some combination thereof.

[0024] Although customer sites 104 and public network 115 are primarily shown and described as edge networks of service provider network 106, in some examples, one or more of customer sites 104 and public network 115 may be tenant networks within data center 10 or another data center. For example, data center 101 may host multiple tenants (customers) each associated with one or more virtual private networks (VPNs), where each tenant may implement one of customer sites 104.

[0025] The service provider network 106 provides packet-based connectivity to attached customer sites 104, data centers 101, and public networks 115. The service provider network 106 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 106 may implement multi-protocol label switching (MPLS) forwarding, and in such instances, may be referred to as an MPLS network or MPLS backbone. In some instances, the service provider network 106 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers.

[0026] In some examples, data center 101 may represent one of many geographically distributed data centers in which the techniques and systems described herein may be implemented. Figure 1 As shown in the example of , data center 101 can be a facility that provides network services, cloud services, storage services and / or application services to customers. Data center 101 can represent a locally deployed data center, a private cloud, a public cloud, a hybrid cloud, or other types of deployments. Customers of a service provider can be collective entities such as enterprises and governments or individuals. For example, a data center can host network services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, science or supercomputing, etc. Although shown as a separate edge network of the service provider network 106, elements of the data center 101 such as one or more physical network functions (PNFs) or virtualized network functions (VNFs) can be included in the core of the service provider network 106.

[0027] The switch fabric 121 may include interconnected top-of-rack (TOR) (or other "leaf") switches 16A-16N coupled to a distribution layer of chassis (or "spine" or "core") switches 18A-18N (hereinafter "chassis switches 18"). Although not shown, the data center 101 may also include, for example, one or more non-edge switches, routers, hubs, gateways, security devices (such as firewalls, intrusion detection, and / or intrusion prevention devices), servers, computer terminals, laptops, printers, databases, wireless mobile devices such as cellular phones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. The data center 101 may also include one or more physical network functions (PNFs), such as physical firewalls, load balancers, routers, route reflectors, broadband network gateways (BNGs), evolved packet cores or other cellular network elements, and other PNFs.

[0028] The term "packet flow", "traffic flow", or simply "flow" refers to a collection of packets originating from a particular source device or endpoint and being sent to a particular destination device or endpoint. For example, a single flow of packets can be identified by a 5-tuple: <source network address, destination network address, source port, destination port, protocol>. Typically, the 5-tuple identifies the packet flow corresponding to the received packets. An n-tuple refers to any n items extracted from a 5-tuple. For example, a 2-tuple of a packet can refer to a combination of <source network address, destination network address> or <source network address, source port> for that packet.

[0029] Any server in the data center 101 may be configured with a workload by virtualizing the server's resources to provide isolation between one or more processes (applications) executing on the server. "Hypervisor-based," or "hardware-level," or "platform" virtualization refers to the creation of virtual machines that each include a guest operating system for executing one or more processes. Typically, a virtual machine provides a virtualized / guest operating system for executing applications in an isolated virtual environment. Because the virtual machine is visualized from the physical hardware of the host server, the executing application is isolated from both the host's hardware and other virtual machines. Each virtual machine may be configured with one or more virtual network interfaces for communicating on a corresponding virtual network.

[0030] "Container-based" or "operating system" virtualization refers to the virtualization of an operating system running multiple isolated systems on a single machine (virtual or physical). Such isolated systems represent containers such as those provided by the open source DOCKER container application or by CoreOS Rkt ("Rocket"). Like virtual machines, each container is virtualized and can be kept isolated from the host and other containers. However, unlike virtual machines, each container can omit a single operating system and only provide a suite of applications and application-specific libraries. Typically, containers are run by a host as isolated user-space instances and the containers can share an operating system and common libraries with other containers executing on the host. As a result, containers may require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on a corresponding virtual network.

[0031] In some examples, the container is managed by its host kernel to allow the use of namespace isolation functions in some cases to limit and prioritize resources (CPU, memory, block I / O, network, etc.) without starting any virtual machines. The command space isolation function allows the application (e.g., a given container) view of the operating environment to be completely isolated, including process trees, networks, user identifiers, and mounted file systems. In some examples, the container can be deployed according to Linux containers (LXC), an operating system-level virtualization method that runs multiple isolated Linux systems (containers) on a control host using a single Linux kernel. LXC is an operating system-level virtualization method that runs multiple isolated Linux systems (containers) on a single control host (LXC host). LXC does not use virtual machines (although LXC can be hosted by virtual machines). Instead, LXC uses a virtual environment with its own CPU, memory, block I / O, network and / or other resource space. The LXC resource control mechanism is provided by namespaces and cgroups in the Linux kernel on the LXC host. Additional information about containers is found in "Docker Overview" Docker, Inc., available at docs.docker.com / engine / understanding-docker, last accessed on July 9, 2016. Additional examples of containerization approaches include OpenVZ, FreeBSD jails, AIX workload partitions, and Solaris containers. Thus, as used herein, the term "container" may encompass not only LXC-style containers, but also any one or more of a virtualization engine, a virtual private server, a silo, or a jail.

[0032] exist Figure 1 In the example of , data center 101 includes storage and / or computing servers interconnected by one or more layers of physical network switches and routers, where computing nodes 110A-110N ("computing nodes 110" herein) are depicted as being interconnected via TOR switches 16 and chassis switches 18. Computing nodes 110 may be bare metal machines and / or virtual machines within data center 101, and may also be referred to herein as "hosts" or "host devices." Computing nodes 110 may represent computing devices configured to operate according to the techniques described herein, such as servers based on x86 processors.

[0033] Compute nodes 110 may host virtual network endpoints of one or more virtual networks that operate over the physical network provided by TOR switch 16 and chassis switch 18. Although primarily described with respect to a data center-based switched network, other physical networks such as service provider network 7 may underlie one or more virtual networks.

[0034] Each of the compute nodes 110 may host one or more workloads. The term “workload” encompasses virtual machines, containers, Kubernetes Pods, and / or other virtualized computing resources that provide at least partially independent execution environments for applications. Figure 1 As shown, compute node 110A hosts a workload that implements service 122 A. However, given the hardware resource limitations of compute node 110, compute node 110 may execute as many workloads as practical.

[0035] The computing infrastructure 100 implements an automation platform for automating the deployment, expansion, and operation of workloads across computing nodes 110 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform can be a container orchestration platform that provides a container-centric infrastructure to automate the deployment, expansion, and operation of containers, thereby providing a container-centric infrastructure. In the context of a virtualized computing infrastructure, "orchestration" generally refers to the provision, scheduling, and management of workloads and / or applications and services executed on such workloads by a host server available to an orchestration platform. Specifically, for example, container orchestration permits container coordination and refers to the deployment, management, expansion, and configuration of containers to host servers by a container orchestration platform. Example instances of orchestration platforms include Kubernetes, Docker swarm, Mesos / Marathon, OpenShift, OpenStack, VMware, and Amazon ECS.

[0036] Elements of the automation platform of computing infrastructure 100 include at least computing nodes 110, orchestrator 130, telemetry collector 142, analysis system 140, and UI device 129. Workloads can be deployed to a virtualized environment using a cluster-based framework in which a cluster master node of a cluster manages the deployment and operation of containers to one or more cluster slave nodes of the cluster. The terms "master node" and "slave node" used herein cover different orchestration platform terms that distinguish between primary management elements of a cluster and analog devices of primary workload hosting devices of a cluster. For example, the Kubernetes platform uses the terms "cluster master" and "minion node", while the Docker Swarm platform refers to a cluster manager and a cluster node.

[0037] Orchestrator 130 implements scheduler 148 for computing infrastructure 100. Orchestrator 130 can be a distributed or centralized application executed on one or more computing devices of a computing system. Orchestrator 130 can implement a respective master node for one or more clusters, each of which has one or more slave nodes implemented by one or more servers of computing infrastructure 100.

[0038] Typically, orchestrator 130 controls the deployment, scaling, and operation of workloads across a cluster of servers and provides computing infrastructure, which may include container-centric computing infrastructure. Orchestrator 130 may implement a corresponding cluster master for one or more Kubernetes clusters. As an example, Kubernetes is a container management platform that provides portability across public and private clouds, where each cloud may provide a virtualized infrastructure to the container management platform. Orchestrator 130 may represent any of the orchestration platforms listed above, such as Kubernetes.

[0039] Each of services 122A-122N (collectively referred to as "services 122") is deployed using a workload. Each service 122 may represent or include one or more containers deployed by a container orchestration system. One or more services 122 may together implement a distributed application that includes a collection of one or more services 122. For example, a distributed application may include services 122A-122N. Each of services 122 may provide or implement one or more services, and in the case where service 122 represents a container pool or other container deployment, one or more services are containerized services or "microservices." Compute node 110 may host services for multiple different distributed applications. In some examples, the services of a distributed application are distributed across compute nodes managed by any combination of service providers, enterprises, or other entities. Such compute nodes may be located in multiple different data centers, local deployments, or in private, public, or hybrid clouds.

[0040] The orchestrator 130 includes a scheduler 148 to schedule services 122 to the computing nodes 110. In general, the scheduler 148 can manage the placement of each of the services 122 to the computing nodes 110 according to the scheduling policy, the amount of resources requested for the service, and the available resources of the computing nodes 110. The computing node resources that the scheduler 148 considers when allocating services 122 to the computing nodes 110 include CPU-related resources (e.g., cores, CPU / core utilization), memory-related resources (available main memory, such as 2GB), temporary storage devices, and user-defined extension resources. In Kubernetes, the scheduler is called the kube-scheduler.

[0041] Services 122 may have different performance requirements that need to be met in a highly dynamic application execution environment. In such an environment, application performance is an artifact of the dynamics of different resources, such as: worker node resources; network resources (e.g., bandwidth, latency, loss, jitter, firewall policies), network policies, and the communication graph between different services of a distributed application; and the performance of external services, such as authentication and external cloud services.

[0042] Services 122 may communicate with each other as part of providing functionality for a distributed application. Each service of services 122 may provide functionality for one or more components of a distributed application. For example, service 122A may provide functionality for a portion of a distributed application, while service 122N may provide functionality for a different portion of a distributed application.

[0043] Services 122 may communicate with each other using calls such as remote procedure calls (RPCs). Services 122 may communicate with each other along an RPC chain to provide the functionality of a distributed application. For example, as part of providing the functionality of a distributed application, service 122A may communicate with service 122N and send an RPC to service 122N.

[0044] Services 122 may call each other in the path of service calls. For example, service 122B may call service 122C, which then calls service 122F. As part of providing distributed application functionality, a series of services may call each other in sequence. In some cases, service 122A is an entry endpoint service, service 122N is a termination endpoint service, and one or more other services are called between service 122A and service 122N for an end-to-end call path for a distributed application.

[0045] In a distributed system, a service request that arrives at an entry point (also known as an endpoint) goes through multiple "hops" through many microservice operations before being fully serviced. The lifecycle of a request results in complex microservice interactions. These interactions are deeply nested, asynchronous, and invoke many other downstream operations. Due to this complexity, it can be difficult to identify which underlying services contribute to the overall end-to-end latency experienced by a top-level request.

[0046] The analysis system 140 may be executed as an application on one or more devices of the data center 101. The analysis system 140 may consume network information, such as network telemetry obtained by the telemetry collector 142, and network flow information obtained from the chassis switch 18 and the TOR switch 16.

[0047] The flow database 143 may be a database maintained by the analysis system 140 and stored to a storage medium. The analysis system 140 may store data about network flows, which in some cases represent RPC call flows between the service pairs 122. In addition, the analysis system 140 may store one or more mappings of service dependencies and network configurations in the flow database 143. The flow database 143 may include sFlow or other flow records provided by the switches 16, 18, and the flow records indicate, for example, flows processed by each switch over a period of time. Each flow record may include a switch identifier, source IP information, destination IP information, number of packets, number of bytes, actions taken, or other flow description data.

[0048] The analysis system 140 can use an application tracing tool to determine the call paths between the services 122. For example, the analysis system 140 can use an application tracing toolkit such as OpenTelemetry and an application tracing tool such as Jaegar to obtain tracing data of calls between the services 122. The application tracing toolkit can use tools such as APIs integrated into the services 122 and software development kits (SDKs) of the application tracing toolkit that instrument or build the services 122. The application tracer can obtain data from the instrumented services of the services 122. The analysis system 140 can use an application tracing tool such as Jaegar to analyze the tracing data generated by the tracing toolkit and determine the call paths between the services 122.

[0049] The analysis system 140 can use an application tracking tool to determine the critical path in one or more call paths of the service 122 supporting the distributed application. For example, the distributed application is composed of a large number of communication services. The distributed application includes an entry endpoint service, which causes the execution of multiple dependent call paths (dependentcall path) between the services 122 that end at the termination endpoint service. Each of the intermediate steps involves an RPC call, which can be a synchronous call or an asynchronous call. Each step has a delay cost associated with it, which may include delays due to calculations and due to network delays. There are certain relatively most expensive call paths in the computational directed acyclic graph (DAG), and therefore may act as a bottleneck path that determines the end-to-end application performance. The analysis system 140 can use an application tracking tool to determine the DAG, which determines the set of nodes on the critical call path in the call path. The analysis system 140 can determine that the critical call path is the call path with the highest end-to-end delay in multiple call paths among multiple call paths. However, the analysis system 140 can determine multiple call paths of interest.

[0050] The analysis system 140 can determine the network path traversed by the calls in the critical call path. The analysis system 140 can use the flow data maintained in the flow database 143 to determine each network path corresponding to a pair of adjacent services in the critical call path. For example, the analysis system 140 can identify the IP addresses of a pair of services along the critical call path, and determine which network devices (such as the TOR switch 16 and the chassis switch 18) have processed flows originating from / going to the IP addresses identifying the pair of services based on the flow data in the flow database 143. Each service 122 can have one or more service instances executing one or more computing nodes 110.

[0051] Each service instance can be identified in the flow data using the following: the physical IP address of the computing node to which the service is deployed, the virtual IP address of the workload of the service instance, or a combination of the physical IP address and the virtual IP address. Both the physical IP address and the virtual IP address are IP information for the service. The analysis system 140 can determine a set of source and destination IP information for each pair of services 122 in the critical path, wherein the source IP information is used for the caller service in each pair of adjacent services (e.g., sending an RPC call) and the destination IP information is used for the callee service in each pair of adjacent servers (e.g., receiving an RPC call and providing an RPC response). The analysis system 140 can associate the call path with the corresponding network path based on the source IP information and the destination IP information included in the flow database 143. The analysis system 140 can determine the underlying network device of the critical call path using information about which network devices have processed the flow corresponding to the critical call path phase. For example, analysis system 140 can determine that the network path from service 122A to service 122N passes through TOR switch 16A, chassis switch 18M, and TOR switch 16N because each of these switches has provided flow data to flow database 143, which flow data is based on the flow data indicating source IP information of service 122A and destination IP information of service 122N, indicating that the switch has processed the RPC flow from service 122A to service 122N.

[0052] Analysis system 140 may obtain IP information of service 122 from orchestrator 130. Server 148 of orchestrator 130 enables orchestrator 130 to deploy service 122 to computing node 110. Orchestrator 130 implements interface 132, through which analysis system 140 may obtain IP information of any service 122. The IP information of a service may include IP information of each instance of the service.

[0053] The analysis system 140 may obtain and analyze performance metrics of network devices under critical call paths. The analysis system 140 may obtain data (such as bandwidth, latency, transmission time, resource utilization) and other data from network devices (such as chassis switches 18 and TOR switches 16). In addition, the analysis system 140 may determine performance metrics by correlating the performance of a network path between a pair of services with RPC calls between the pair of services. For example, the analysis system 140 may determine the latency of a network path between a pair of services based on the latency of RPC calls between the pair of services.

[0054] The analysis system 140 can analyze the performance metrics and determine whether one or more of the performance metrics reaches one or more of the thresholds of the performance metrics. For example, the analysis system 140 can determine that the delay of the TOR switch 16A reaches the performance threshold of the network device delay. In some examples, the analysis system 140 can determine that all network devices in the first network path associated with the first pair of adjacent services in the critical call path are operating normally. The analysis system 140 analyzes the network path between the next pair of adjacent services within the critical call path based on the determination of the normal operation of the network device. In another example, the analysis system 140 determines that all network devices of the call path between the service pairs for the critical call path are operating normally. The analysis system 140 identifies the second critical call path based on the determination and analyzes the network device of the second critical call path. In this way, the analysis system 140 can traverse and process the critical call path of the distributed application to try to identify abnormal behavior in the network that has a significant impact on the performance of the distributed application.

[0055] Analysis system 140 provides an indication of abnormal behavior of the network device to orchestrator 130 via interface 132 based on the performance indicator of the network device reaching a threshold. Analysis system 140 may provide an indication to orchestrator 130 for orchestrator 130 to perform remedial actions. Analysis system 140 may provide an indication including information identifying the network device that is experiencing the network path, calling the path, and the network device that has reached a threshold / is experiencing abnormal behavior.

[0056] As described above, orchestrator 130 orchestrates the placement of workloads to compute nodes 110, the workloads including services 122. For example, orchestrator 130 may orchestrate the placement of services 122 of a distributed application to one or more compute nodes 110. In response to receiving an indication from analytics system 140, orchestrator 130 may cause scheduler 148 to adjust the scheduling of one or more of services 122 to redeploy the services to compute nodes different from the compute nodes on which the services are currently executing.

[0057] For example, orchestrator 130, based on receiving an indication from analysis system 140, can cause scheduler 148 to reschedule a service in a service pair whose network path passes through a network device that is experiencing abnormal behavior. For example, orchestrator 130 receives an indication that a network device under a network path between a pair of services 122 is experiencing abnormal behavior. Orchestrator 130 can identify one of computing nodes 110 to reallocate one of the services to it so that the network path between the service pair does not pass through the network device that is experiencing abnormal behavior. In another example, orchestrator 130 identifies another instance of one of the services in service pair 122 that is executed on a different computing node 110 based on receiving the indication. In yet another example, orchestrator 130 can reconfigure one or more network devices to reroute the network path around the network device that is experiencing abnormal behavior.

[0058] Distributed applications may consist of multiple services that communicate over a network to complete the required tasks. This will result in a complex set of dependent RPC calls during application execution. Therefore, the end-to-end performance of the application is determined by the performance of these service call chains. Only a few RPC call chains will cause the maximum latency in end-to-end behavior and can be considered as the critical path of the application. Considering that in a distributed system, many of these call chains need to traverse the underlying network, network latency may be one of the key contributing factors to the latency of these call chains. The technology disclosed in this disclosure provides a framework for quickly associating application critical path performance with the underlying network. The framework is a highly scalable solution that focuses on the impact of the network on the application critical path of large-scale, highly distributed applications.

[0059] The techniques of the present disclosure may provide one or more technical advantages. For example, the techniques of the present disclosure enable a computing system to identify bottlenecks in the performance of a distributed application caused by the affected performance of service calls between services of the distributed application. The techniques of the present disclosure also enable a computing system to use call paths and network path information between service pairs to identify network devices that are experiencing abnormal behavior and affecting the performance of service calls between service pairs. By utilizing information about call paths and corresponding network paths, these techniques can enable faster diagnosis of distributed application performance.

[0060] Figure 2202 is a block diagram illustrating an example computing system according to the techniques described in the present disclosure. Computing system 202 may be implemented as any suitable computing system, such as one or more server computers, workstations, mainframes, appliances, cloud computing systems, and / or other computing systems that may be capable of performing operations and / or functions described according to one or more aspects of the present disclosure. In some examples, computing system 202 represents a cloud computing system, a server farm, and / or a server cluster (or a portion thereof) that provides services to other devices or systems. In other examples, computing system 202 may represent or be implemented by one or more virtualized computing instances (e.g., virtual machines, containers) of a cloud computing system, a server farm, a data center, and / or a server cluster.

[0061] exist Figure 2 In the example of , computing system 202 may include one or more processors 213, communication unit 215 (illustrated as COMM. 215), one or more input devices 217, one or more output devices 218, and one or more storage devices of storage system 205. Storage system 205 includes analysis system 240 and orchestrator 130. One or more of the devices, modules, storage areas, or other components of computing system 202 may be interconnected to enable inter-component communication (physically, communicatively, and / or operationally). In some examples, such connection may be provided by a communication channel, which may represent one or more of a system bus, a network connection, an inter-process communication data structure, or any other method for transmitting data.

[0062] One or more of the processors 213 may implement functions associated with the computing system 202 or associated with one or more modules shown herein and / or described below and / or execute instructions associated with them. One or more of the processors 213 may be a processing circuit that performs operations according to one or more aspects of the present disclosure, may be a part of the processing circuit, and / or may include the processing circuit. Examples of processors 213 include a microprocessor, an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, a processing unit, or a processing device. The computing system 202 may use one or more processors 213 to perform operations according to one or more aspects of the present disclosure using software, hardware, firmware, or a combination of hardware, software, and firmware resident in and / or executed at the computing system 202.

[0063] One or more communication units 215 of computing system 202 can communicate with devices external to computing system 202 by sending and / or receiving data, and in some aspects can operate as both input devices and output devices. In some examples, communication unit 215 can communicate with other devices through a network. In other examples, communication unit 215 can send and / or receive radio signals on a radio network (such as a cellular radio network). In other examples, communication unit 215 of computing system 202 can send and / or receive satellite signals on a satellite network. Examples of communication unit 215 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device that can send and / or receive information. Such communications can comply with, implement, or comply with appropriate protocols, including transmission control protocol / Internet protocol (TCP / IP), Ethernet, or other technologies or protocols.

[0064] The one or more input devices 217 may represent any input device of the computing system 202 that is not described separately herein. The input device 217 may generate, receive, and / or process input. For example, the one or more input devices 217 may generate input or receive input from a network, a user input device, or any other type of device for detecting input from a person or a machine.

[0065] One or more output devices 218 may represent any output device of computing system 202 that is not described separately herein. Output device 218 may generate, present, and / or process output. For example, one or more output devices 218 may generate, present, and / or process output in any form. Output device 218 may include one or more USB interfaces, video and / or audio output interfaces, or any other type of device capable of generating tactile, audio, visual, video, electrical, or other output. Some devices may be used as both input devices and output devices. For example, a communication device may send data to other systems or devices over a network and receive data from other systems or devices.

[0066] One or more storage devices of the storage system 205 within the computing system 202 may store information for processing during operation of the computing system 202. The storage system 205 may store program instructions and / or data associated with one or more of the modules described according to one or more aspects of the present disclosure. One or more processors 213 and one or more storage devices may provide an operating environment or platform for such modules, which may be implemented as software, but in some examples may include any combination of hardware, firmware, and software. One or more processors 213 may execute instructions, and one or more storage devices of the storage system 105 may store instructions and / or data of one or more modules. The combination of the processor 213 and the storage system 205 may retrieve, store, and / or execute instructions and / or data of one or more applications, modules, or software. The processor 213 and / or the storage device of the storage system 205 may also be operably coupled to one or more other software and / or hardware components, including but not limited to one or more of the components of the computing system 202 and / or one or more devices or systems shown as being connected to the computing system 202.

[0067] Processor 213 may execute analysis system 240. Analysis system 240 may be an application, platform, or other type of process configured to analyze call paths between services of a distributed application executed across multiple computing nodes. For example, analysis system 240 may monitor various distributed applications and underlying services (such as Figure 1 The performance of the service 122) is shown.

[0068] As part of monitoring the services of a distributed application, analysis system 240 may identify call paths between services 122. Analysis system 240 may use application tracing tools, such as dependency map generator service (DMGS) 244. DMGS 244 may include one or more tracing tools and toolkits, such as Jaegar or OpenTelemetry. DMGS 244 may instrument a distributed application including services 122 and obtain call path information for a given time window. DMGS 244 may use the call path information to determine the call paths between services 122.

[0069] Analysis system 240 executes application critical path analysis engine (ACPAE) 246. ACPAE may be an application module, plug-in, standalone service, or other component or service of analysis system 240 invoked by analysis system 240. ACPAE 246 may determine a critical call path from among multiple call paths identified by analysis system 240.

[0070] ACPAE 246 may determine a critical call path based on a dependency graph of multiple call paths of a distributed application. For example, ACPAE 246 may determine a critical call path based on a DAG including multiple call paths for a particular distributed application. ACPAE 246 may determine a critical call path based on an end-to-end delay of the call path. In another example, ACPAE 246 may identify a critical call path that acts as a bottleneck for the performance of a distributed application. ACPAE 246 may determine a critical call path based on a delay of a call path from an entry endpoint service to a termination endpoint service.

[0071] Analysis system 240 may execute network flow analysis service 248 (hereinafter referred to as "flow analysis 248"). Flow analysis 248 may be a module, plug-in, standalone service, or other service or application. Flow analysis 248 may analyze the traffic received by analysis system 240 from one or more network devices (such as Figure 1 TOR switch 16 and chassis switch 18 shown). Flow analysis 248 can analyze the network flow data of flow database 143 to identify a network path corresponding to a call path between a pair of services. In an example, flow analysis 248 receives network data from one or more network devices. Flow analysis 248 processes the data and identifies which network devices are located under the call path between a pair of services. In some examples, flow analysis 248 can use the IP address of the computing node executing the service to determine the network path by determining which network devices have processed the IP address of the service. Flow analysis 248 can use physical IP addresses and / or virtual IP addresses to determine whether one or more network devices have processed the network flow of the call path between a pair of services that is part of a critical call path.

[0072] The analysis system 240 may perform a critical application path to network path mapping service (CAP) 250. The analysis system 240 may use CAP 250 to determine a network path for a call path between services corresponding to a critical call path. CAP 250 may use flow analysis 248 to determine the network path between services for the critical call path. In an example, CAP 250 receives an indication of the critical call path from a component of the analysis system 240, such as ACPAE 246. CAP 250 obtains network path information for one or more call paths between pairs of services including the critical call path from flow analysis 248. CAP 250 uses the network path information to identify the network path for the call path between the service pairs, and by extension, to identify network devices and infrastructure underlying the call path between the service pairs.

[0073] The analysis system 240 executes a root cause recommendation system (RCRS) 252. Similar to other components of the analysis system 240, the RCRS 252 can be a plug-in, a standalone service, or another type of process. Based on the identification of the underlying network devices of the critical path, the RCRS 252 can determine whether one or more of the underlying network devices are experiencing abnormal behavior. The RCRS 252 can determine whether the network device is experiencing abnormal behavior based on the network device achieving / achieving one or more performance metrics.

[0074] RCRS 252 determines one or more performance metrics of the network device based on one or more factors. RCRS 252 can determine the performance metrics of the network device by mapping the performance of the call path between services of the critical call path to the corresponding network path. RCRS 252 can determine whether the network device in the network path is experiencing abnormal behavior based on mapping the call path performance to the network path performance. In an example, RCRS 252 determines that the TOR switches within the network path are experiencing considerable delays and slowing down the call path between services based on mapping the performance of the call path to the network path.

[0075] RCRS 252 determines whether one or more network devices under the call path between the services of the critical call path have reached one or more performance thresholds that may indicate abnormal behavior. For example, RCRS 252 may determine one or more performance metrics of each network device, such as latency, transmission time, bandwidth, resource utilization of the network device, and other performance metrics. RCRS 252 may determine whether one or more of the performance metrics have reached one or more performance thresholds of the network device. In the example, RCRS 252 determines that a specific TOR switch with a maximum allowable delay of 30ms is experiencing an average delay of 55ms. RCRS 252 determines that when the delay of a specific TOR switch reaches a delay threshold, the specific TOR switch is experiencing abnormal behavior. Based on determining that the network device has reached the performance threshold, RCRS 252 may provide an indication to the orchestrator 230 via interface 232 to redeploy service 122 for a service pair with a critical call path for communication passing through the TOR switch.

[0076] One or more processors 213 or computing systems 202 execute orchestrator 230. Orchestrator 230 may be a distributed or centralized application that implements a master node of one or more clusters having a plurality of nodes such as Figure 1One or more slave nodes of a server implementation of a computing infrastructure of the computing infrastructure 100 shown. Orchestrator 230 can orchestrate the placement of services to computing nodes within computing infrastructure 100. In response to receiving an indication from analysis system 240, orchestrator 230 can perform one or more remedial actions. Orchestrator 230 can take remedial actions, such as rescheduling a workload from one computing node to a different computing node so that the network path between the two computing nodes does not pass through the network device that is experiencing abnormal behavior. Orchestrator 230 can take remedial actions, such as redirecting the call path to end at different service instances executed on different computing nodes. In the example, service A calls service B on a network path including a network device that is experiencing abnormal behavior. In response to an indication from analysis system 240, orchestrator 230 causes service A to call different instances of service B executed on different computing nodes, and avoids passing through the network device that is experiencing abnormal behavior via the call path. Orchestrator 230 can be executed on a computing system separate from analysis system 240.

[0077] Analysis system 240 may execute UI 254. In addition to providing indications to orchestrator 230, analysis system 240 may also provide alerts to a user via UI 254. UI 254 may generate a user interface including one or more visual elements. For example, UI 254 may generate a user interface including one or more visual elements, each of which is associated with one or more services and network devices. In another example, UI 254 may generate a user interface including a visual representation of a DAG of a network infrastructure. In yet another example, UI 254 may generate a user interface including a visual element associated with an alert indicating that a network device under a critical path is experiencing abnormal behavior.

[0078] Analysis system 240 may output a user interface via output device 218. Analysis system 240 may output a user interface generated by UI 254 for display to a user. For example, UI 254 may generate a user interface that causes output device 218 to display a user interface that includes a visual indicator of an alert that a device is experiencing abnormal behavior. In another example, UI 254 may generate a user interface that includes a visual representation of a DAG of calls between a service and an underlying network infrastructure.

[0079] FIG. 3A to FIG. 3C is a block diagram illustrating an example diagram of one or more techniques in accordance with this disclosure. Figure 3A An example application dependency graph 300 is shown. For clarity, Figure 2 To describe Figure 3A .

[0080] exist Figure 3AIn the example of FIG. 3 , application dependency graph 300 includes services 302. Services 302 may be services of a distributed application that call each other as part of providing functionality of the distributed application.

[0081] Services 302 can call each other in a series of calls that form a call path. The call path is generated by the service call chain. Figure 3A As shown, service 302 calls other services of service 302 via non-critical calls 310 and critical calls 312 .

[0082] The analysis system 240 may use tracing tools and / or tracing toolkits to identify the call paths of the distributed application. The analysis system 240 may use tracing tools and toolkits to instrument the distributed application and determine the calls between services 302. The analysis system 240 may analyze the calls between services to derive a dependency graph (such as graph 300) of the calls of the distributed application. The analysis system 240 may analyze the calls for a given time window and determine the call paths for the call window.

[0083] For each time window, the analysis system 240 can use the application communication call trace to determine the bottleneck critical path. A critical path (such as a critical call path) is a chain of RPC calls that determines the end-to-end performance (such as response time) of an application. Figure 3A In the example of , analysis system 240 determines a call path including non-critical call 310 and critical call 312. For example, analysis system can determine one or more call paths that are not critical call paths including non-critical call 312 between services 302. In another example, analysis system 240 determines a critical call path including critical call 310.

[0084] Analysis system 240 can generate application dependency graph 300 as a graph of calls between services of a distributed application, including non-critical and critical call paths. Analysis system 240 can generate application dependency graph 300 as a graph of calls between services of a distributed application, including non-critical and critical call paths. Figure 3A An indicator of a service 302) that is visually hashed in FIG.

[0085] Figure 3B shows an example diagram of services and the underlying network infrastructure. Figure 2 Let's discuss Figure 3B .exist Figure 3BIn the example, an analysis system such as analysis system 240 determines an application dependency graph including services 302A-302G (hereinafter referred to as "services 302"), compute nodes 304A-304H (hereinafter referred to as "compute nodes 304"), TOR switches 306A-306D (hereinafter referred to as "TOR switches 306"), and chassis switches 308A and 308B (hereinafter referred to as "chassis switches 308").

[0086] Analysis system 240 may identify the call paths of services 302 at the application layer across the network infrastructure. Analysis system 240 may use application tracing tools and toolkits such as DMGS 244 to determine the calls between services 302 of the distributed application. For example, DMGS 244 may determine, for example, Figure 3B As shown, service 302A calls service 302B as part of providing the functionality of the distributed application. In another example, DMGS 244 may determine that service 302C calls service 302D, which then calls service 302E.

[0087] Analysis system 240 can identify a critical call path from a plurality of call paths between services 302. Analysis system 240 can use ACPAE 246 to determine the critical call path from the call paths between services 302. ACPAE 246 can determine the critical call path by determining the call path based on the end-to-end delay of the critical path. For example, ACPAE 246 can determine a call path that proceeds from 302A to 302B, to 302C, to 302D, to 302E, to 302F and terminates at 302G. In some examples, the critical path can change in different monitoring time windows.

[0088] Based on determining the critical call path, analysis system 240 uses information provided from orchestrator 230 to determine the underlying compute nodes of the compute layer of the network infrastructure that execute the service for the critical call path. Once the critical path is known, it must be determined whether the latency bottleneck is due to the underlying network. This requires determining the path of the RPC call in the network and then correlating the performance of the network path with the performance of the RPC calls in the critical path. Analysis system 240 can obtain data including configuration data of service 302 from orchestrator 230 of service 302. For example, analysis system 240 can obtain configuration data of service 302, which includes data about scheduling service 302 to compute nodes 304. Analysis system 240 uses data about the scheduling of service 302 to determine which of compute nodes 304 execute the service for the critical call path. For example, analysis system 240 determines that compute nodes 304A and 304C each execute an instance of service 302A (e.g., Figure 3BIn another example, analysis system 240 determines that computing node 304C executes instances of services 302A, 302C, and 302F (e.g., Figure 3B ).

[0089] The analysis system 240 determines the network path between the computing nodes hosting the services of the critical call path based on the computing nodes determined under the critical call path. Each service in the critical call path can have multiple instances hosted on different computing nodes (hereinafter referred to as "computing nodes 304") in the computing layer indicated by computing nodes 304A-304H. Any RPC call between two adjacent services can be made between different sender and receiver instances. The analysis system 240 can use flow analysis 248 to determine the network path between the computing nodes 304, and by extension, determine the network path between the computing nodes 302 of the critical call path. The flow analysis 248 can use the IP addresses of the computing nodes hosting the services of the critical call path. The flow analysis 248 can use the IP addresses of the computing nodes and determine which TOR switches 306 and chassis switches 308 have processed the IP addresses of the computing nodes.

[0090] The analysis system 240 can determine the network path for each call between adjacent services corresponding to the critical call path. The analysis system 240 can determine the network devices in the underlying structure through which these call flows pass. This can be achieved by collecting flow data from the network devices in the structure layer, and then querying the flow database 143 to determine the set of TOR switches 306 and chassis switches 308 that have observed the flow corresponding to the RPC call between two adjacent services in the critical path. Repeat this operation for each span in the critical path. The analysis system 240 can determine which network devices have observed the remote procedure call (RPC) flow of the call path based on which network devices have observed the source IP address and destination IP address of a pair of services. The analysis system 240 can use CAP 250 to determine each network path that is the basis of each call between the services of the critical call path. In the example, service 302B calls service 302C as part of the critical call path. CAP 250 determines that the network path of the call between services 302B and 302C passes through TOR switch 306A, chassis switch 308A, and TOR switch 306B. In another example, CAP 250 determines that the network path between service 302C and the instance of service 302D executing on node 304D passes through ToR switch 306B. ToR switch 306 and chassis enable switch 308 to be similar to Figure 1 ToR switch 16 and chassis switch 18.

[0091] Once the critical application path is mapped to the underlying or upper layer network path as shown above, it is possible to determine which specific path or sub-path in the network has the greatest impact on the application RPC calls in the critical path. In order to determine the router that affects the largest number of service spans (i.e., RPC calls). For example, the performance of the chassis switch 308A may affect 3 spans: [302A-302B], [302B-302C], [302F-302G]. If the critical KPI of the chassis switch 308A (such as interface packet loss rate or error rate) shows abnormal behavior in the monitoring window T, there is a high probability that the performance of the critical path spans (302A-302B, 302B-302C, 302F-302G) is affected by the network during this time window. In the event that no problems are observed based on the chassis switch 308A KPI, the same correlation analysis is performed with the next set of network devices in descending order of the number of spans affected. If there is no correlation to determine the causal relationship between network path performance and critical path performance, this can help operators eliminate the network as a bottleneck for application performance degradation. The analysis system 240 can use RCRS252 to determine whether one or more network devices that are the basis of the network path are experiencing abnormal behavior. RCRS 252 can associate the performance of calls between services with the underlying network path. For example, RCRS 252 can associate the performance of calls from service 302B to service 302C with the network path between service 302B and service 302C to determine the performance of the network path. RCRS 252 determines the performance metrics of the network devices of the network path based on the correlation of call performance with the underlying network path. RCRS 252 can determine metrics such as delay, transmission time, bandwidth, resource utilization and other performance metrics. RCRS 252 determines whether one or more performance metrics reach one or more performance thresholds based on determining performance metrics. For example, RCRS 252 can determine that the delay of TOR switch 306A reaches the performance threshold of delay.

[0092] In some cases, in response to RCRS 252 determining that a performance threshold has been reached for at least one network device at the bottom of the network path, analysis system 240 can instruct orchestrator 230 to remedy the network path by redeploying services. In response, orchestrator 230 determines one or more actions to remedy the performance of the critical call path. In the example, analysis system 240 determines that TOR switch 306C is experiencing abnormal behavior and is affecting the performance of the critical call path by slowing down the call between service 302C executed on computing node 304C and service 302D executed on computing node 304E. Analysis system 240 provides instructions to orchestrator 230 to remedy the call between service 302C and service 202D. In response to the indication, orchestrator 230 modifies the call between services 302C and 302D so that service 302C calls an instance of service 302D executed on computing node 304D instead of an instance of service 302D executed on computing node 304E to avoid traversing TOR switch 306C. In another example, orchestrator 230 can identify a replacement service for any of the services in the pair of adjacent services, wherein the replacement service is another instance of any of the services in the pair of adjacent services that is executed on a different computing node. Orchestrator 230 can reconfigure the distributed application to execute using the replacement service instead of one of the services in the pair of adjacent services.

[0093] Figure 3C An example diagram of application dependencies is shown. For clarity, Figure 2 To describe Figure 3C . Figure 3C yes Figure 3B , and illustrates an example call path between service 302A and service 302B along the underlying network path between instances of those services. Figure 3C In the example of FIG. 3 , service 302A calls service 302B as part of providing the functionality of a distributed application. Service 302A uses service call 310 to call service 302B.

[0094] Analysis system 240 may determine the network path corresponding to service call 310. Analysis system may use data from orchestrator 230 to identify the computing nodes executing services 302A and 302B. Figure 3C In the example of FIG. 3 , analysis system 240 can determine that computing nodes 304A and 304C execute an instance of service 302A and computing node 304B executes an instance of service 302B.

[0095] Analysis system 240 determines a network path for transporting service call 310. Analysis system 240 may use source IP information and destination IP information for services 302A and 302B and determine which network devices handle flows that include source IP information and destination IP information for instances of services 302A and 302B, respectively.

[0096] The analysis system 240 may determine that there are more than one network path for a particular service path. The analysis system 240 may determine that there are multiple network paths due to multiple instances of one or more services executed on different computing nodes. For example, the analysis system determines that in the case where service 302A is executed on computing node 304A, the network path of service call 310 starts at computing node 304A, passes through TOR switch 306A, and is received by service 302B hosted at computing node 304B. In another example, service 302A is executed on computing node 304C. Service call 310 starts at computing node 304C, passes through TOR switch 306B, chassis switch 308A, and TOR switch 306A, and is received by service 302B hosted at computing node 304B.

[0097] Analysis system 240 can provide instructions to orchestrator 230 to remediate the critical call path by remediating service call 310. Analysis system 240 can provide instructions to orchestrator 230 that include information about the network path of service call 310.

[0098] The orchestrator 230 can remedy the network path of the service call 310 to avoid passing through the network device that is experiencing abnormal behavior. The orchestrator 230 can remedy the network path of the service call 310 by rescheduling the service of the service path 310 to a different computing node, and avoid passing through the network device that is experiencing abnormal behavior. The orchestrator 230 can also make the service call of the service call 310 different instances of the service executed on different computing nodes to avoid passing through the network device that is experiencing abnormal behavior. The orchestrator 230 can also make the distributed application use different entry services to avoid passing through the network device. For example, the orchestrator 230 can make the distributed application use an instance of the service 302A executed on the computing node 304C instead of the instance of the service 302A executed on the computing node 304A. In another example, the analysis system 240 determines that the chassis switch 308A that the network path 314 passes through from the computing node 304C to the computing node 304B is experiencing abnormal behavior. In response to this determination, the analysis system 240 provides an indication of the remediation service call 310 to the orchestrator 230. Orchestrator 230 causes distributed application to use the instance of service 302A executed on computing node 304A instead of the instance of service 302A executed on computing node 304C as the entry service. As a result, service call 310 traverses network path 316 instead of network path 314.

[0099] Figure 4 is a flowchart illustrating an example operation of a computing system according to one or more techniques of the present disclosure. Figure 1 Let’s discuss this in the context of Figure 4 .

[0100] A computing system (such as analysis system 140) determines a call path from an entry endpoint service in the plurality of services to a termination endpoint service in service 122 for a distributed application implemented using a plurality of services (such as service 100) executed on computing nodes (such as computing node 110) interconnected by a network of network devices (402). Analysis system 140 may determine a call path for the distributed application, which includes a plurality of services forming a call chain as the call path. To determine the call path, analysis system 140 uses a tracing tool and / or a tracing toolkit. For example, analysis system 140 may determine the call path using a tracing tool that instrumented services 122 and traced calls between services 122.

[0101] Analysis system 140 determines a corresponding network path for each pair of adjacent services of services 122 in plurality of service pairs 122 that communicate for the call path (404). Analysis system 140 may determine the corresponding network paths using flow data obtained from network devices such as TOR switches 16 and chassis switches 18. For example, analysis system 140 may obtain virtual and physical IP addresses of compute nodes executing services 122 and determine which network devices have seen the IP addresses of those compute nodes.

[0102] Based on the performance indicator of the network device corresponding to the network path reaching the threshold, the analysis system 140 performs one or more of the following: reconfiguring the network or redeploying one of the services 122 to a different computing node in the computing nodes 110 (406). The analysis system 140 can map or associate the performance of the service call between a pair of nodes to the performance of the network path. For example, the analysis system 140 can determine that the network device is experiencing increased latency or abnormal behavior by mapping the sluggish performance of the service call between a pair of services to the network device supporting or supporting the connection between those services. The analysis system 140 can analyze the performance metrics of the network device, such as latency, transmission time, bandwidth, resource utilization, availability of labels (such as MPLS labels), and other performance metrics. The analysis system 140 can compare the performance metrics with one or more thresholds, which are predetermined or dynamically adjusted based on network and performance requirements. Based on determining that the performance indicator achieves / reaches the performance threshold, the analysis system 140 can perform one or more remedial actions. The analysis system 140 can reconfigure the network and remediate the network device that reaches the threshold and is experiencing abnormal behavior. Analysis system 140 can cause an orchestrator, such as orchestrator 130, to reschedule or redeploy one or more services to different computing nodes to avoid traversing the network device. In some examples, analysis system 140 can cause a distributed application to execute using different instances of services executed on different computing nodes to avoid traversing the network device.

[0103] Figure 5 is a flowchart illustrating an example operation of a computing system according to one or more techniques of the present disclosure. Figure 1 Let's discuss Figure 5 .

[0104] An analysis system, such as analysis system 140, determines an application call trace for a given time window (502). Analysis system 140 may use an application tracing tool and / or a toolkit such as Jaegar and OpenTelemetry to obtain the application call trace. Analysis system 140 may use the application call trace to determine call paths between services of the application. Analysis system 140 may determine call paths between services of the application, where the application may be a distributed application. In the case of a distributed application, due to the distributed nature of the application and the myriad of potential connections and network paths between the services of the application, an administrator may find it challenging to determine how the performance of individual services and the connections between the services affect the entire distributed application.

[0105] The analysis system 140 determines a critical path for the application (504). The analysis system 140 may determine the critical call path from the plurality of call paths between the services 122. The analysis system 140 may determine the critical call path by selecting the call path with the greatest delay. The analysis system 140 may determine the critical call path based on the total delay of the call path from the entry endpoint service to the termination endpoint service.

[0106] Analysis system 140 determines a set of remote procedure call (RPC) source IP and destination IP pairs (506). Analysis system 140 determines virtual and / or physical IP addresses for source and destination pairs of services within the critical call path. Analysis system 140 may determine the IP addresses for each source and destination pair of services within the critical call path. For example, analysis system 140 may determine the IP addresses for four pairs of services that include services within the critical call path.

[0107] Analysis system 140 determines a set of network devices that have processed (e.g., switched or forwarded) the RPC flow for the source and destination service pair (508). Analysis system 140 may use flow data obtained from network devices within data center 101 to determine the set of network devices that processed the flow of RPC calls. For example, analysis system 140 may determine that TOR switch 16A, chassis switch 18M, and TOR switch 16N have all processed the RPC flow for a pair of services.

[0108] The analysis system 140 uses the dependency graph information to determine a set of possible paths for the RPC calls. The analysis system 140 can generate a dependency graph of the critical call paths, wherein the dependency graph shows the computing nodes executing the services and the underlying network infrastructure including network devices such as TOR switches 16 and chassis switches 18. The analysis system 140 can use the dependency graph to associate with the flows and thereby identify network devices that potentially affect the performance of the RPC calls between services and, by extension, the performance of the distributed application.

[0109] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0110] If implemented in hardware, the present disclosure may be directed to a device such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause one or more processors to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by one or more processors.

[0111] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media. The computer-readable storage media may be distributed among multiple groups, devices, or other components capable of being configured with computer instructions.

[0112] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not incorporated into a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that changes over time (e.g., stored in RAM or cache).

[0113] The code or instructions may be software and / or firmware executed by a processing circuit including one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in the present disclosure may be provided within a software module or a hardware module.

Claims

1. A computing system comprising: Memory; as well as a processing circuit in communication with the memory and configured to: For a distributed application implemented using a plurality of services executed on computing nodes interconnected by a network of network devices, determining a call path from an entry endpoint service in the plurality of services to a termination endpoint service in the plurality of services; determining a corresponding network path for each pair of adjacent services in a plurality of pairs of services communicating with respect to the call path; and Based on the performance indicator of the network device of the corresponding network path reaching a threshold, performing one or more of the following operations: reconfiguring the network; and One of the plurality of services is redeployed to a different one of the computing nodes.

2. The computing system of claim 1, wherein: In order to determine the corresponding network path, the processing circuit is configured to: determining a source IP address and a destination IP address for a pair of neighboring services communicating for the call path, wherein the source IP address is an IP address of a caller service in the pair of neighboring services, and wherein the destination IP address is an IP address of a callee service in the pair of neighboring services; and Based on the source IP address and the destination IP address, a flow is associated to the corresponding network path.

3. The computing system of claim 2, wherein: To associate the call path with the corresponding network path, the processing circuit is configured to determine, based on the flow data, a network device that has processed a flow including the source IP address and the destination IP address.

4. The computing system of claim 2, wherein: The source IP address is one of a physical IP address of a computing node hosting the caller service and a virtual IP address of a workload of the caller service.

5. The computing system according to any one of claims 1 to 4, wherein: The network device includes at least one top-of-rack switch and at least one chassis switch.

6. The computing system according to any one of claims 1 to 4, wherein: To reconfigure the network, the processing circuit is further configured to: identifying a replacement service for either service in a pair of adjacent services, wherein the replacement service is another instance of a first service in the pair of adjacent services executing on a different computing node; and The distributed application is reconfigured to execute using the replacement service.

7. The computing system according to any one of claims 1 to 4, wherein: The call path is a first call path, wherein the entry endpoint is a first entry endpoint, wherein the termination endpoint is a first termination endpoint, and wherein the processing circuit is further configured to: determining, based on determining that no performance indicator of the network device reaches a threshold, a second call path from a second entry endpoint service in the plurality of services to a second termination endpoint service in the plurality of services for the distributed application implemented using the plurality of services; determining a second corresponding network path for each pair of adjacent services in the plurality of pairs of services communicating with respect to the call path; identifying a second network device on one of the second corresponding network paths to the adjacent service; and Based on the performance indicator of the second network device reaching the threshold, perform one or more of the following operations: reconfiguring the network; and One of the plurality of services is redeployed to a different one of the computing nodes.

8. The computing system according to any one of claims 1 to 4, wherein: The performance indicators are one or more of the following: Packet loss rate, Transmission time, Resource utilization, and Delay.

9. The computing system according to any one of claims 1 to 4, wherein: The call path is a call path among a plurality of call paths for the distributed application that has a highest end-to-end delay among the plurality of call paths.

10. A computer networking method, comprising: For a distributed application implemented using a plurality of services executed on computing nodes interconnected by a network of network devices, determining, by a computing system, a call path from an entry endpoint service in the plurality of services to a termination endpoint service in the plurality of services; determining, by the computing system, a corresponding network path for each pair of adjacent services in a plurality of pairs of services communicating with respect to the call path; and Based on the performance indicator of the network device of the corresponding network path reaching a threshold, the computing system performs one or more of the following operations: reconfiguring the network; and One of the plurality of services is redeployed to a different one of the computing nodes.

11. The computer networking method according to claim 10, wherein: Determining the corresponding network path further includes: determining a source IP address and a destination IP address for a pair of neighboring services communicating for the call path, wherein the source IP address is an IP address of a caller service in the pair of neighboring services, and wherein the destination IP address is an IP address of a callee service in the pair of neighboring services; and Based on the source IP address and the destination IP address, a flow is associated to the corresponding network path.

12. The computer networking method according to claim 11, wherein: Associating the call path with the corresponding network path further includes determining which network devices have observed the remote procedure call flow of the call path based on which devices have observed the source IP address and the destination IP address.

13. The computer networking method according to claim 12, further comprising: A network device that has processed a flow including the source IP address and the destination IP address based on the flow data.

14. The computer networking method according to any one of claims 10 to 13, wherein: The network device includes at least one top-of-rack switch and at least one chassis switch.

15. The computer networking method according to any one of claims 10 to 13, wherein: Reconfiguring the network includes: identifying a replacement service for either service in a pair of adjacent services, wherein the replacement service is another instance of a first service in the pair of adjacent services executing on a different computing node; and The distributed application is reconfigured to execute using the replacement service.

16. The computer networking method according to any one of claims 10 to 13, wherein: The call path is a first call path, wherein the entry endpoint is a first entry endpoint, wherein the termination endpoint is a first termination endpoint, and the computer networking method further comprises: Based on determining that no performance indicator of the network device reaches the threshold, determining, by the computing system, a second call path from a second entry endpoint service among the plurality of services to a second termination endpoint service among the plurality of services for the distributed application implemented using the plurality of services; determining, by the computing system, a second corresponding network path for each pair of adjacent services in the plurality of pairs of services communicating with respect to the call path; identifying, by the computing system, a second network device on one of the second corresponding network paths for the adjacent services; and Based on the performance indicator of the second network device reaching a threshold, perform one or more of the following operations: reconfiguring the network; and One of the plurality of services is redeployed to a different one of the computing nodes.

17. The computer networking method according to any one of claims 10 to 13, wherein: The performance indicators are one or more of the following: Packet loss rate, Transmission time, Resource utilization, and Delay.

18. The computer networking method according to any one of claims 10 to 13, wherein: The call path is a call path among a plurality of call paths for the distributed application that has a highest end-to-end delay among the plurality of call paths.

19. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to: For a distributed application implemented using a plurality of services executed on computing nodes interconnected by a network of network devices, determining a call path from an entry endpoint service in the plurality of services to a termination endpoint service in the plurality of services; determining a corresponding network path for each pair of adjacent services in a plurality of pairs of services communicating with respect to the call path; and Based on the performance indicator of the network device of the corresponding network path reaching a threshold, performing one or more of the following operations: reconfiguring the network; and One of the plurality of services is redeployed to a different one of the computing nodes.

20. The non-transitory computer-readable storage medium of claim 19, wherein: The instructions further configure one or more processors to form a network computing system according to any one of claims 2 to 9 , or to execute a computer networking method according to any one of claims 11 to 18 .

Images