Locating network faults through differential analysis of TCP telemetry

Abstract

Embodiments of the present disclosure relate to locating network faults through differential analysis of transmission control protocol (TCP) telemetry. A server includes a processor and a memory. An operating system is executed by the processor and the memory. A network interface is run by the operating system and uses TCP to send and receive flows. An agent application is run by the operating system and is configured to a) retrieve and store TCP telemetry data of flows in a flow table; b) when a flow is closed, move the selected flow from the flow table to a closed connection table; c) periodically send the flow table and the closed connection table to a remote server via the network interface.

Description

[0001] Related patent applications

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2017 / 045997, international application date August 9, 2017, priority date August 12, 2016, Chinese national phase entry date February 2, 2019, and Chinese application number 201780048969.8. Technical Field

[0003] The present disclosure relates to computer networks, and more particularly to systems and methods for locating network faults. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. To the extent that the work described in this background section may not otherwise qualify as prior art at the time of filing, the work of the present inventors is neither explicitly nor implicitly admitted as prior art to the present disclosure.

[0005] Network engineers often rely on network packet captures to locate network connectivity issues. While effective, packet captures are often not running when connectivity issues occur. Network engineers often launch long-running packet captures and / or attempt to reproduce connectivity issues. The former approach is expensive in terms of processing power. The latter approach is difficult and complex to perform, as data center networks include many interacting components. As a result, replicating the network state when connectivity issues occur is often cumbersome. Summary of the invention

[0006] A server includes a processor and a memory. An operating system is executed by the processor and the memory. A network interface is run by the operating system and uses a transmission control protocol (TCP) to send and receive flows. An agent application is run by the operating system and is configured to a) retrieve and store TCP telemetry data of flows in a flow table; b) move selected flows from the flow table to a closed connection table when the flow is closed; c) periodically send the flow table and the closed connection table to a remote server via the network interface.

[0007] In other features, the proxy application is configured to clear the flow entry in the closed connection table after c). The proxy application is configured to set the TCP telemetry data of the flow entry in the flow table to 0 after c). The proxy application is also configured to aggregate the TCP telemetry data before sending the TCP telemetry data in c). The proxy application is also configured to filter the TCP telemetry data before sending the TCP telemetry data in c).

[0008] In other features, the proxy application is further configured to monitor the activity status of the flow entries in the flow table and selectively change the status of the flow entries in the flow table. The proxy application is further configured to move one of the flow entries with an inactive status to the closed connection table when a new flow is reported.

[0009] A server includes a processor and a memory. An operating system is executed by the processor and the memory. A network interface is executed by the operating system. A flow processing application is executed by the operating system and is configured to a) receive a flow table from a plurality of remote servers via the network interface, the flow table including aggregated transmission control protocol (TCP) telemetry data for active flow entries; b) receive a closed connection table from the plurality of remote servers via the network interface, the closed connection table including aggregated TCP telemetry data for inactive flow entries; c) geo-tag the flow entries in the flow table and the closed connection table based on the locations of the plurality of remote servers; and d) forward the TCP telemetry data with the geo-tags to a data repository.

[0010] In other features, the stream processing application is configured to aggregate at least one of a plurality of flows, failed flows, new flows, closed flows, and terminated flows between an Internet Protocol (IP) source and an IP destination during an aggregation interval. The stream processing application is configured to aggregate at least one of a plurality of sent and received bytes, published bytes, and read bytes between the Internet Protocol (IP) source and the IP destination during an aggregation interval.

[0011] In other features, the stream processing application is configured to calculate at least one of an average round trip time and a maximum round trip time between an Internet Protocol (IP) source and an IP destination during the aggregation interval.

[0012] In other features, the stream processing application is configured to calculate an average size of a congestion window and a number of congestion window reductions between an Internet Protocol (IP) source and an IP destination during an aggregation interval.

[0013] In other features, the stream processing application is further configured to identify and authenticate the flow table and the closed connection table using a unique identifier corresponding to a remote server in the plurality of remote servers. The stream processing application is further configured to filter the TCP telemetry data.

[0014] A server includes a processor and a memory. An operating system is executed by the processor and the memory. A network interface is run by the operating system. A differential data analysis application run by the operating system is configured to a) receive TCP telemetry data with geotags from a remote server and store it in a database; b) in response to a network error at a source server, retrieve the TCP telemetry data with geotags for the source server during a first period before the error starts and during a second period after the error starts or ends; c) in response to the network error at the source server, identify a comparison server in communication with the source server, and retrieve the TCP telemetry data with geotags for the comparison server during a first period before the error starts and during a second period after the error starts or ends; and d) perform at least one function on the TCP telemetry data with geotags for at least one of the source server and the comparison server.

[0015] In other features, at least one function includes aggregating geo-tagged TCP telemetry data for a comparison server. The at least one function includes normalizing geo-tagged TCP telemetry data for the source server and the comparison server.

[0016] In other features, the differential data analysis application generates a table including the normalized metrics of the source server arranged adjacent to the normalized metrics of the comparison server.

[0017] In other features, the trend analysis application is configured to compare the normalized metrics of the source server and the comparison server to predetermined values ​​and generate a proposed diagnosis based on the comparison.

[0018] In other features, the network configuration application configures the plurality of remote servers with a push-based configuration or a pull-based configuration.

[0019] Other areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A is a functional block diagram of an example of a network including multiple data centers according to the present disclosure.

[0021] Figure 1B is a functional block diagram of another example of a network including multiple data centers according to the present disclosure.

[0022] Figure 2 is a functional block diagram of an example of a server including a proxy application in a data center according to the present disclosure.

[0023] Figure 3is a functional block diagram of an example of a server including a stream processor application according to the present disclosure.

[0024] Figure 4 is a flowchart of an example of a method performed by a proxy application for generating and updating a flow table and a closed connection table according to the present disclosure.

[0025] Figure 5 is a flow chart of an example of a method performed by a proxy application for determining an active or inactive state of a flow in a flow table according to the present disclosure.

[0026] Figure 6 is a flowchart of an example of a method performed by a proxy application for managing a flow table and a closed connection table according to the present disclosure.

[0027] Figure 7 is a flowchart of an example of a method performed by a stream processing application for processing a stream table and closing a connection table according to the present disclosure.

[0028] Figure 8 is a flow chart of an example of a method performed by a differential data analysis application according to the present disclosure.

[0029] Fig. 9 is a flow chart of an example of a method performed by a trend analysis application according to the present disclosure.

[0030] Among the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0031] The system and method according to the present disclosure uses TCP telemetry data collected from the Transmission Control Protocol (TCP) stack at each server to locate connection problems, rather than using computationally expensive, long-running packet capture. To ensure reliable delivery, the TCP stack already maintains TCP telemetry data related to the state of the network and the state of the remote server with which the server is communicating.

[0032] Most modern operating systems (OS) (such as Windows 7 and later) provide user-level programs with an application protocol interface (API) that allows access to TCP telemetry data. Systems and methods according to the present disclosure include an agent application running at each server (or node) in a data center or other network location. Nodes or servers can be arranged in a cluster including multiple nodes. A data center can include multiple clusters.

[0033] The proxy application collects TCP telemetry data from the TCP stack. Since the TCP telemetry data is already collected and maintained by the TCP stack, this process has minimal computational impact on the node. The proxy application aggregates the TCP telemetry data and sends the TCP telemetry data to one or more other servers for analysis at predetermined intervals. At the analysis server, the TCP telemetry data is enriched with geography. In other words, location information (such as cluster name or ID, domain name of the node, etc.) is added to the TCP telemetry data. The geo-enriched telemetry data is stored (or forwarded to another server) for further analysis.

[0034] When a connection problem occurs from a first node (e.g., node 1) to a second node (e.g., node 2), an analysis server storing TCP telemetry data identifies the top N talkers to node 2. The top N talkers are nodes that are geographically close to node 1 and transmit a large amount of data to node 2. The analysis server performs one or more functions on the TCP telemetry data from the top N talkers when the connection problem occurs. In some examples, the functions include one or more of aggregation, averaging, normalization, and / or sorting. This generates a common view of the network from the top N talkers.

[0035] TCP telemetry data from node 1 to node 2 is selected during a period starting before the connection problem starts and ending after the connection problem starts. The TCP telemetry data of the top N talkers to node 2 is compared with the TCP telemetry data from node 1 to node 2. The TCP telemetry data that stands out during the comparison provides an indication of the location of the problem, as will be further described below.

[0036] For example, during a connection problem, both Node 1 and the top N talkers report a connection failure to Node 2. The problem is likely to involve Node 2. Whereas if only Node 1 reports a connection failure, the problem is likely to be a problem with Node 1.

[0037] Reference now Figure 1A and 1B , shows an example network configuration for locating network faults according to the present disclosure. However, it will be appreciated that other network configurations may be used without departing from the scope of the present disclosure. Figure 1A , the network 10 includes one or more data centers 12-1, 12-2, ..., and 12-D (collectively referred to as data centers 12), where D is an integer greater than or equal to 1. The data center 12 sends and receives data through a distributed communication system (DCS) 14. The DCS 14 may include a wide area network (WAN) such as the Internet, a local area network (LAN), and / or other types of DCSs. The data repository 16 also sends data to and receives data from the data center 12 through the DCS 14.

[0038] The data center 12-1 includes one or more servers 18-1, 18-2, ..., and 18-S (collectively referred to as servers 18), where S is an integer greater than or equal to 1. Each of the servers 18-1, 18-2, ..., and 18-S includes an agent application 20-1, 20-2, ..., and 20-S (collectively referred to as agent applications 20). The group of servers 18 can be arranged into a cluster.

[0039] The data center 12-1 may also include a server 22, which includes a stream processing application 23. The stream processing application 23 periodically receives TCP telemetry data from the proxy application 20 in the server 18. The stream processing application 23 processes the data and forwards the processed TCP telemetry data to the data repository 16. The data repository 16 includes one or more servers 24 and a database (DB) 30 for storing TCP telemetry data. At least one of the servers 24 includes a data analysis application 26 for performing difference and trend analysis on the TCP telemetry data from the stream processing application 23. The server 24 may also include a configuration network service 28 for configuring the server 18 using a push-based or pull-based technique.

[0040] exist Figure 1B , another example network configuration is shown. In this example, the stream processing application 42 is associated with the data repository 40. In other words, the stream processing application is implemented in the data repository 40 instead of in the data centers 36-1, 36-2, ..., and 36-D (collectively referred to as the data center 36).

[0041] Reference now Figure 2 , shows an example of a server 50 used in a data center 12 or a data center 36. The server 50 includes one or more processors 104 and a memory 112. An operating system 114, one or more node applications 116, and an agent application 118 are executed by the processor 104 and the memory 112 during operation. As will be further described below, the agent application 118 maintains a flow table 120 and a closed connection table 122. The agent application 118 periodically sends the flow table 120 and the closed connection table 122 to the stream processing application. The server 50 also includes a display subsystem 124, which includes a display 126. The server 50 communicates through the DCS 14 using a network interface 130. The network interface 130 uses a transmission control protocol (TCP) 132 to send and receive data.

[0042] Reference now Figure 3, shows an example of a server 150 used in the data repository 16. The server 150 includes one or more processors 154 and a memory 160. An operating system 164 and one or more applications 165 are executed by the processor 104 and the memory 160 during operation. The applications 165 may include a stream processing application 166, a data analysis application 168, a trend analysis application 170, and / or a configuration network service application 172. Although Figure 3 A single server is shown in FIG, but application 165 may run on multiple local or remote servers.

[0043] The server 150 also includes a display subsystem 173, which includes a display 174. The server 150 communicates through the DCS 14 using a network interface 171. The network interface 171 uses the transmission control protocol (TCP) to send and receive data. The server 150 may include a mass storage 180 for locally storing a database 182. Alternatively, the database 182 may be associated with another local or remote server (not shown).

[0044] Reference now Figure 2 and Figure 4 , the proxy application 118 collects, aggregates, and sends TCP telemetry data for the flow. As used herein, a flow refers to a TCP connection 4-tuple that includes the source Internet Protocol (IP) address, source port, destination IP address, and destination port from one of the servers 18 in the data center 12. A fifth tuple defining the direction of the flow may also be included. To ensure reliable packet delivery, TCP collects information about how the server application sends / receives data and the state of the network during the life cycle of the flow. The operating system 114 exposes this information to the proxy application 118.

[0045] In addition to the quad-tuple, the proxy application 118 uses the mechanisms provided by the OS 114 to collect one or more types of TCP telemetry data about the flow from the TCP stack, as described below in

[0046] As described in Table 1:

[0047]

[0048]

[0049] exist Figure 4, a method 200 for operating the proxy application 118 is shown. At 210, a timer corresponding to the aggregation interval is reset. At 214, the proxy application 118 determines whether a new flow has been received or sent by the server 50. If 214 is true, then at 218, a new entry is created in the flow table 120. At 222, a unique flow identification (ID) is generated. At 226, the TCP telemetry value of the new entry is set equal to zero.

[0050] The method continues from 214 (if no) or 226 with 230, where the proxy application 118 updates the TCP telemetry data for the flow in the flow table 120. At 234, the proxy application 118 determines whether the flow is closed. If 234 is true, then at 238, the proxy application 118 moves the flow to the closed connection table 122.

[0051] The method continues from 234 (if no) or 238 with 242. At 242, the proxy application 118 determines whether the timer has expired. If 242 is false, the method returns to 214. If 242 is true, at 246, the proxy application 118 serializes the flow table 120, the closed connection table 122, and the flow ID and sends the flow table 120, the closed connection table 122, and the flow ID to the server associated with the stream processing application. At 250, the proxy application 118 clears the flow entry in the closed connection table 122. At 252, the proxy application 118 sets the data of the flow entry in the flow table 120 to 0.

[0052] In some examples, the proxy application 118 aggregates TCP telemetry data about the flow at predetermined time intervals to reduce the amount of TCP telemetry data to be sent from the node. In some examples, the predetermined time interval is in the range of 5 seconds to 5 minutes, but other aggregation intervals can be used. In some examples, the flow ID is a unique hash calculated based on a TCP 4-tuple or 5-tuple. In some examples, when TCP reports a new flow, a flow entry is created in the flow table 120 with all associated telemetry values ​​set to 0.

[0053] While the connection persists, the proxy application 118 collects TCP telemetry data from the TCP stack and performs aggregation or other functions on the TCP flow data, including one or more of the following Table 2:

[0054]

[0055] When the TCP stack reports that a flow is closed (terminated, failed, or successfully closed), the proxy application 118 removes the flow from the flow table 120 and adds this information to the closed connection table 122. At the end of each aggregation interval, the proxy application 118 serializes the entries in the flow table 120 and the closed connection table 122 and sends these tables to the server using the stream processing application.

[0056] After sending these tables, the proxy application 118 clears the closed connection table 122 and sets the associated telemetry of the flow in the flow table 120 to 0. In some examples, standard line protocols for TCP telemetry are used, such as Internet Protocol Flow Information Export (IPFIX) and ) for serialization.

[0057] Typically, a server 18 at a data center 12 has many incoming and outgoing connections. Thus, maintaining a flow table 120 for each connection may be memory intensive. In some examples, an upper limit is set for the number of flows that the proxy application 118 maintains to reduce the required memory. With this limit, capturing active flows becomes more important. In some examples, a flow is considered inactive if the TCP stack does not report a TCP telemetry change during a TCP session timeout or if TCP only reports a "keep alive" event during a TCP session timeout.

[0058] Reference now Figure 5 , a method 300 for changing the activity state of a flow is shown. At 310, the proxy application 118 monitors TCP telemetry data for each flow or a sample set of flows during a predetermined period. At 314, the proxy application 118 determines whether the TCP telemetry data of the flow changes during the period at 318. If 314 is false, the flow is marked as inactive. If 314 is true, at 322, the flow is marked as active. The method continues from 318 or 322 with 324. At 324, the proxy application 118 determines whether only "keep alive" events occurred during the predetermined period of the flow. If 324 is true, at 328, the flow is marked as inactive. If 324 is false, at 332, the flow is marked as active.

[0059] When the flow table 120 is full and TCP reports a new flow, the proxy application 118 attempts to open space in the flow table 120 by moving inactive flows to the closed connection table. If there are no inactive flows, the proxy application 118 drops the flow and ignores the reported TCP telemetry data. At the end of the aggregation interval, the proxy application 118 reports the number of ignored flows. The number of ignored flows can be used to configure the size of the flow table 120 of the server 50.

[0060] Reference now Figure 6, which shows a method 350 for moving an inactive flow from the flow table 120 to the closed connection table 122. At 360, the method determines whether the flow table 120 is full. If 360 is true, the method determines at 364 whether to report a new flow. If 364 is true, the method determines at 366 whether there is an inactive flow in the flow table 120. If 366 is true, then at 370, the inactive flow is moved to the closed connection table 122. At 374, a new flow is added to the flow table 120. If 366 is false, then at 378 the new flow is ignored. At 382, the flow counter (corresponding to the count of ignored flows) is incremented.

[0061] For many active flows at the server, collecting TCP telemetry data can be processor-intensive. Instead of polling or processing each TCP event, the proxy application 118 can be programmed to sample flow events. Sampling strategies such as sampling N out of M events (where N and M are integers and N < M), sampling every other flow event, sampling 1 / 4 of the events, etc. can be employed. In some examples, TCP events related to flow state changes are captured (regardless of the sampling mechanism) to help the proxy application 118 capture new active flows.

[0062] In some examples, the proxy application 118 provides a mechanism for remote control and configuration. For example, an operator needs to be able to remotely turn on / off the tracking, storage, and / or forwarding of TCP telemetry data. In some examples, a push-based or pull-based method is used to implement remote control. In the push-based method, updates are sent directly to the proxy application 118. In this model, the proxy application 118 needs to accept incoming connections.

[0063] In the pull-based method, the proxy application 118 actively polls for configuration updates. In some examples, standard REST web services can be utilized to supply configurations to the proxy application 118s. The proxy application 118 connects to the configuration web service 172 at a configurable interval to check for updates to its configuration. In some examples, HTTP or SSL connections are used for configuration updates to enhance security. The pull-based method for configuration updates also serves as a heartbeat from the proxy application 118. The heartbeat can be used to identify node failures due to software or hardware errors and / or security issues such as distributed denial of service (DDOS) attacks or node compromises.

[0064] As described above, at the end of each aggregation interval, the proxy application 118 sends the aggregated TCP telemetry data for each flow (stored in the flow table 120 and the closed connection table) to the flow processing application. The proxy application 118 sends the flow ID with the TCP telemetry data. In some examples, the flow ID is sent using the IPFIX domain ID field. The flow ID allows the flow processing application 166 to identify and authenticate the proxy application 118 that sends the TCP telemetry data. In some examples, when the proxy application 118 requests a configuration update, the flow ID is set remotely at configuration time (i.e., when the proxy application 118 is deployed to the server) or at runtime.

[0065] Reference now Figure 7 , a method 400 for processing a stream at a stream processing application 166 is shown. At 410, the method determines whether TCP telemetry data is received. At 414, the stream processing application 166 deserializes the TCP telemetry data. At 418, the stream processing application 166 uses the unique stream ID to identify and authenticate the proxy application 118. At 422, the TCP telemetry data is geotagged. At 426 and 428, optional aggregation and / or filtering are performed. At 432, the TCP telemetry data with the geotagged is sent to a data repository.

[0066] In other words, the stream processing application 166 deserializes the TCP telemetry data and then tags the TCP telemetry data with location (or geographic) information ("geo-tags"). Geo-tags enrich the TCP telemetry data with information about the server location. In some examples, this information includes, but is not limited to, data center name / location, cluster name / cluster, and / or rack number / name / location of a node. Geo-tagging can be accomplished using the source IP or flow ID of the TCP telemetry data.

[0067] After geotagging, the TCP telemetry data is sent to the data repository. In some examples, in order to reduce the amount of TCP telemetry data, auxiliary aggregation and / or filtering is applied by the stream processing application 166. Auxiliary aggregation can group the TCP telemetry data based on IP-IP pairs (i.e., all flows from one source IP to one destination IP) or source IP destination IP port pairs. In some examples, aggregation is performed without losing the semantics of the TCP telemetry data. Since nodes at a data center may run different applications, the source / destination port field can be used to mark and distinguish different applications. In some examples, the proxy application 118 sends the name of the process that established the flow.

[0068] Table 3, set forth below, includes one or more examples of aggregate calculations or functions that may be generated:

[0069]

[0070]

[0071] In some examples, the stream processing application 166 performs filtering and selectively discards some TCP telemetry data associated with flows that are not of interest. For example, filtering based on traffic volume (e.g., discarding flows that are less than a predetermined number of bytes) and / or specific destination / source pairs. Flows can also be filtered by the proxy application 118. For example, a remote configuration update can be used to configure filtering parameters to instruct the proxy application 118 not to track flows to a specified destination IP address and / or port.

[0072] Analysis of TCP telemetry data is performed to: 1) identify possible locations of failures using differential analysis, 2) identify trends that may indicate poor communication paths, and 3) provide real-time analysis showing the health of connections and application traffic between various network nodes.

[0073] When a user reports a network-related error, differential analysis is performed. Given an error at time t, TCP telemetry data from a source node (the node that observed the error) is extracted during a sampling period. The sampling period starts from a first period before the error starts until a second period after the error starts or ends. In some examples, the first period and the second period are multiples of an initial aggregation interval (i.e., the aggregation applied by the proxy application 118) or a secondary aggregation interval (i.e., the aggregation applied at the stream processing application 166).

[0074] Examples of the extracted data that are aggregated to calculate the metrics are listed in Table 4:

[0075]

[0076]

[0077] In some examples, the ratio of all flows is used to normalize the data points. Normalization is performed to find the connections that have shown the most errors and transferred the least amount of data. As more flows are created, the probability of error increases. In other examples, the data points are composed of

[0078] Normalized by the total traffic from source to destination (eg, #TCP Syn Timeouts / Total Traffic).

[0079] After extracting the data of the source node, the TCP telemetry data of the comparison node is extracted from the data repository in the same period. Here, the comparison node: 1) is geographically close to the source node, and 2) the top N talkers are in the same destination as the source node.

[0080] The TCP telemetry data from each comparison node is aggregated to calculate the metrics listed in Table 4. The TCP telemetry data from all comparison nodes is then aggregated to formulate TCP telemetry data to be compared with the TCP telemetry data from the source node. Possible aggregations for formulating the comparison telemetry include, but are not limited to, the median or 95th percentile of the metrics listed in Table 4 (descriptive names such as "front talker" can be used for the source field). The aggregated telemetry from the comparison nodes and the source nodes are placed side by side to formulate a difference analysis table.

[0081] Reference now Figure 8 , a method 450 for performing data analysis is shown. At 460, the method determines whether a network error is encountered. When 460 is true, at 464, starting before the error begins and ending after the error begins, data related to the source node is extracted. At 468, a metric of the source node is calculated. At 472, data related to the comparison node is extracted during the same period. At 476, a metric of the comparison node is calculated. At 478, a differential analysis is generated by comparing the metrics of the source node and the comparison node. In some examples, the differential analysis compares the metric of the source node, the metric of the comparison node, and / or a predetermined threshold. At 482, the network is diagnosed based on the differential analysis.

[0082] Combined with expert knowledge about network performance, the differential table shows which metrics peaked during the reported error period and whether similar peaks were seen from some other nodes. Observing the peaks can yield pattern discoveries that indicate the location of the problem.

[0083] Common patterns often indicate problems with the server, network, or client device. Examples are listed in Table 5 below:

[0084]

[0085]

[0086] In some examples, a trend analysis is performed to identify possible bad communication paths. The trend analysis includes a first phase and a second phase. The first phase is performed using a first timer period to extract a connection quality metric for each source and destination pair. In some examples, the source and destination pairs are IP-IP, pod-pod, etc. In some examples, the first period is equal to one hour, but other periods can be used.

[0087] Table 6 lists a set of example metrics that may be extracted during the first phase of trend analysis:

[0088]

[0089]

[0090] The second phase of the trend analysis is performed using a second timer period that is longer than the first timer period. For example, the first period can be 1 hour and the second period is 24 hours, but other periods can also be used. In the second phase, the top N (e.g., top 10, 20, etc.) pairs of the metrics listed in Table 3 are identified for each period in which the first phase is performed. In other words, the top N pairs with the largest standardized failed flows, standardized retransmissions, standardized TCP connection terminations, and the smallest standardized congestion window size are calculated. Then, the source and destination pairs that appear the most in any of the top N lists are identified. Because these source and destination pairs have shown some abnormal behavior (e.g., more retransmissions compared to other pairs), the network path between the source and destination is further investigated.

[0091] Reference now Fig. 9 , a method 500 for performing trend analysis is shown. At 510, if the first period of the first phase ends, then at 514, a set of connection quality metrics are generated for the source and destination pairs. At 516, a list is generated and sorted to identify the source and destination pairs with the highest failures for one or more metrics for each first timer period at 561. At 520, the method continues from 510 (if no) and 516. If the second timer period of the second phase ends at 520, then at 524, a predetermined number of top source and destination pairs with the highest failures in the stored set of metrics and lists are determined.

[0092] The system and method according to the present disclosure provide network engineers with insights about the state of the network in near real time. The analysis is achieved by coupling the data repository with a user interface (UI) front end. In some examples, the UI front end is generated using frameworks such as Cabana, powerBI, etc. The UI provides a summary view of the network status. The summary includes aggregated TCP telemetry data or a comparison view to show the network connection health of the node with other nodes in the cluster (the average value in the cluster or 90% of the other nodes).

[0093] In some examples, the summary view is used to identify possible network locations for further investigation. Once identified, the detailed analysis provided above can be used to further pinpoint the problem. This view contains aggregated TCP telemetry data, and the aggregation can be cluster-to-cluster and / or rack-to-rack. The summary view can include, for example, near real-time insights into total traffic, TCP measured RTT, syn timeouts, and connection terminations. On the one hand, total traffic volume can be used to identify applications that are heavily utilizing the network, and RTT can show busy servers. On the other hand, TCP terminations and failed flows (syn timeouts) can indicate reachability issues.

[0094] The comparison view is always used to visually compare a node in a cluster / rack with all other nodes in the same cluster / rack (or nearby) in near real-time. Here, the network status of the cluster / rack can be calculated using aggregation functions such as average or percentile (i.e., 90th percentile of TCP telemetry such as RTT, bytes / s, and connection terminations). Figure 2 An example UI is depicted showing how Node 1 compares to all nodes in the same cluster (Cluster 1).

[0095] The foregoing description is essentially illustrative only and is by no means intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in various forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be limited thereto, because after studying the drawings, the specification and the appended claims, other modifications will become clear. It should be understood that one or more steps in the method can be performed in different orders (or simultaneously) without changing the principles of the present disclosure. In addition, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in the features of any other embodiment and / or implemented in combination with it, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments to each other is still within the scope of the present disclosure.

[0096] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "adjacent," "above," "above," "below," and "disposed." Unless explicitly described as "directly," when describing a relationship between a first element and a second element in the above disclosure, the relationship may be a direct relationship in which no other intermediate elements exist between the first element and the second element, but may also be an indirect relationship in which one or more intermediate elements exist (spatially or functionally) between the first element and the second element. As used herein, the phrase at least one of A, B, and C should be interpreted as representing a logical (A or B or C) using a non-exclusive logical OR, and should not be interpreted as representing "at least one A, at least one B, and at least one C."

[0097] In the drawings, the direction of the arrow indicated by the arrow generally represents the flow of information (such as data or instructions) of interest in the diagram. For example, when component A and component B exchange various information but the information transmitted from component A to component B is relevant to the diagram, the arrow may point from component A to component B. This unidirectional arrow does not mean that no other information is transmitted from component B to component A. In addition, for information sent from component A to component B, component B may send a request for information to component A or receive an acknowledgment of the information.

[0098] The term memory or memory circuit is a subset of the term computer-readable medium. The term computer-readable medium as used herein does not include transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term computer-readable medium may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0099] The apparatus and methods described in this application can be implemented in part or in whole by a special-purpose computer, which is created by configuring a general-purpose computer to perform one or more specific functions contained in a computer program. The above-mentioned function blocks, flow chart components and other elements are used as software specifications, which can be converted into computer programs through routine work of skilled technicians or programmers.

[0100] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0101] A computer program may include: (i) descriptive text to be parsed, such as JSON (Javascript Object Notation) HTML (Hypertext Markup Language) or XML (Extensible Markup Language), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code executed by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from a variety of languages, including C, C++, C#, Objective C, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5, Ada, ASP (Active Server Pages), PHP, Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua and

[0102] No element recited in a claim is a means-plus-function element within the meaning of 35 U.S.C. §112(f) unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the phrase “operation for” or “step for.”

Claims

1. A method for network management, comprising: receiving transmission control protocol (TCP) telemetry data including associated geotags from a remote server and storing it in a database; In response to a network error at the identified origin server: retrieving the TCP telemetry data and associated geotags for the origin server during a first time period before the network error begins and during a second time period after the network error begins or ends; as well as identifying a comparison server in communication with the source server, and retrieving the TCP telemetry data and associated geotags for the comparison server during the first period before the error began and during the second period after the error began or ended; as well as At least one function is performed on the TCP telemetry data and the associated geotags for at least one of the source server and the comparison server.

2. The method of claim 1 , wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are aggregated for the comparison server.

3. The method of claim 2, wherein aggregating the TCP telemetry data comprises: The TCP telemetry data is grouped in a table based on the associated geo-tags for the TCP telemetry data.

4. The method according to claim 3, further comprising: For corresponding packets of the TCP telemetry data based on the associated geotags, a diagnosis for the network error is generated based on an analysis of the TCP telemetry data.

5. The method of claim 2, wherein aggregating the TCP telemetry data comprises: The telemetry data is aggregated based on identified pairs of IP information between source Internet Protocol IPs and destination IPs from the associated geo-tags.

6. The method of claim 1 , wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are normalized for the source server and the comparison server.

7. The method according to claim 6, further comprising: A table is generated that includes the normalized metrics of the source server arranged adjacent to the normalized metrics of the comparison server.

8. The method according to claim 7, further comprising: comparing the normalized metrics of the source server and the comparison server with predetermined values; as well as Based on the comparison of the normalized metrics, a proposed diagnosis for the network error is generated.

9. The method of claim 7, further comprising configuring a plurality of remote servers in a push-based configuration or a pull-based configuration.

10. The method of claim 1, wherein the associated geotag comprises location information about the remote server, wherein the location information comprises one or more of: a cluster name, a cluster identifier, a domain name, a data center name, a data center location, a rack number, a rack name, a rack location, or an Internet Protocol (IP) address.

11. A system for network management, comprising: one or more processors; a memory in electronic communication with the one or more processors; as well as Instructions stored in the memory, the instructions being executable by the one or more processors to cause the server device to: receiving transmission control protocol (TCP) telemetry data including associated geotags from a remote server and storing it in a database; In response to a network error at the identified origin server: retrieving the TCP telemetry data and associated geotags for the origin server during a first time period before the network error begins and during a second time period after the network error begins or ends; as well as identifying a comparison server in communication with the source server, and retrieving the TCP telemetry data and associated geotags for the comparison server during the first period before the error began and during the second period after the error began or ended; as well as At least one function is performed on the TCP telemetry data and the associated geotags for at least one of the source server and the comparison server.

12. The system of claim 11, wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are aggregated for the comparison server.

13. The system of claim 12, wherein aggregating the TCP telemetry data comprises: The TCP telemetry data is grouped in a table based on the associated geotags for the TCP telemetry data, and also includes instructions executable by the one or more processors to cause the server device to perform the following: for corresponding groups of the TCP telemetry data based on the associated geotags, generate a diagnosis for the network error based on an analysis of the TCP telemetry data.

14. The system of claim 11, wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are normalized for the source server and the comparison server.

15. The system of claim 14, further comprising instructions executable by the one or more processors to cause the server device to: generating a table including the normalized metrics of the source server arranged adjacent to the normalized metrics of the comparison server; comparing the normalized metrics of the source server and the comparison server to predetermined values; and Based on the comparison of the normalized metrics, a proposed diagnosis for the network error is generated.

16. A non-transitory computer-readable storage medium having instructions stored thereon, which when executed by one or more processors cause a server device to: receiving transmission control protocol (TCP) telemetry data including associated geotags from a remote server and storing it in a database; In response to a network error at the identified origin server: retrieving the TCP telemetry data and associated geotags for the origin server during a first time period before the network error begins and during a second time period after the network error begins or ends; as well as identifying a comparison server in communication with the source server, and retrieving the TCP telemetry data and associated geotags for the comparison server during the first period before the error began and during the second period after the error began or ended; as well as At least one function is performed on the TCP telemetry data and the associated geotags for at least one of the source server and the comparison server.

17. The non-transitory computer-readable storage medium of claim 16, wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are aggregated for the comparison server.

18. The non-transitory computer-readable storage medium of claim 17, wherein aggregating the TCP telemetry data comprises: The TCP telemetry data is grouped in a table based on the associated geotags for the TCP telemetry data, and also includes instructions that, when executed by the one or more processors, cause the server device to perform the following instructions: for corresponding groups of the TCP telemetry data based on the associated geotags, generate a diagnosis for the network error based on an analysis of the TCP telemetry data.

19. The non-transitory computer-readable storage medium of claim 16, wherein performing the at least one function on the TCP telemetry data comprises: The TCP telemetry data and associated geotags are normalized for the source server and the comparison server.

20. The non-transitory computer-readable storage medium of claim 19, further comprising instructions that, when executable by the one or more processors, cause the server device to: generating a table including the normalized metrics of the source server arranged adjacent to the normalized metrics of the comparison server; comparing the normalized metrics of the source server and the comparison server to predetermined values; and Based on the comparison of the normalized metrics, a proposed diagnosis for the network error is generated.

Images