Detection of network measurement inaccuracies

Abstract

A detection of network measurement inaccuracy. The present disclosure describes techniques for detecting network measurement inaccuracy by detecting sender delay or packet loss. For example, a sender device of a test packet can determine whether the sender device encounters any problems when sending the test packet to a receiver device, and notify a controller of these problems so that the controller can generate an indication that one or more key performance indicator (KPI) measurements based on the test packet from the sender device are inaccurate and / or unreliable, remove the inaccurate KPI measurements, and / or adjust the inaccurate KPI measurements.

Description

[0001] This application claims the benefit of U.S. patent application No. 17 / 364,178, filed on June 30, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to computer networks and, more particularly, to measuring network performance. Background Art

[0003] A computer network is a collection of interconnected computing devices that can exchange data and share resources. In some examples, computing devices can provide services to measure the network performance of a computer network. For example, computing devices can exchange data to calculate key performance indicators (KPIs), such as latency, inter-frame gap (inter-frame delay variation), jitter, packet loss, throughput, etc.

[0004] For example, to measure a KPI, a computing device may exchange test packets including a timestamp to indicate the time when the test packet was sent or received. For example, to calculate latency between devices, a sending device sends a test packet including a timestamp indicating the time when the message was sent, and a receiving device may generate a timestamp indicating the time when the message was received. The timestamp is used to calculate latency and / or other KPIs of network performance between computing devices.

[0005] Public Content

[0006] In general, the present disclosure describes techniques for detecting network measurement inaccuracies by detecting sender delays or packet losses. A computing device of a network is configured as a measurement point (e.g., a test agent) for measuring network performance. For example, a test agent of a sender computing device (referred to herein as a "sender device") may send a test packet to a test agent of a receiving device (referred to herein as a "receiver device"), which in turn may use the test packet to calculate one or more KPIs. As an example, a sender device may send a test packet including a timestamp (referred to herein as a "send timestamp") indicating the time when the test packet was sent. When the receiver device receives the test packet, the receiver device generates a timestamp (referred to herein as a "receive timestamp") indicating the time when the test packet was received. The receiver device may use the timestamp to calculate one or more KPIs, such as delays. A controller may obtain measured KPIs from a receiver device, and may output the measurement results to a user or a network administrator for display, for example, via a graphical user interface.

[0007] In some cases, the sending device may encounter problems with sending test packets, which may result in inaccurate and / or unreliable KPI measurements. For example, when sending a send timestamp to a receiving device, an application of the sending device generates a test packet including a first timestamp, which indicates the time when the test packet is generated by the application. The sending device forwards the test packet to a network interface (e.g., a network interface card) of the sending device to output the test packet to the receiving device. In some cases, the network interface may be delayed when sending a test packet including a send timestamp to the receiving device. In another example, the sending device may experience a pause (referred to herein as a "scheduled pause") during the measurement process, such as when an application other than the test application is running on the sending device and requires CPU cycles, wherein such a pause may cause the sending device to lose the test packet due to, for example, a queue overflow. The receiving device may calculate one or more KPI measurements based on the test packets without taking into account sender delays or packet losses caused by the scheduling pause, which may cause the controller to present one or more KPI measurements, but no KPI measurements are not attributable to network performance but to indications of sender problems.

[0008] According to the techniques described herein, a sending device can determine whether the sending device is encountering any problems when sending test packets to a receiving device, and notify a controller of the problem, so that the controller can generate an indication that one or more KPI measurement values ​​of the test packets from the sending device are inaccurate and / or unreliable, remove the inaccurate KPI measurement values, and / or adjust the inaccurate KPI measurement values.

[0009] In an example where a sender device may experience a delay when sending a test packet, the technology describes that the sender device obtains a second timestamp from a kernel or hardware component of a network interface, the second timestamp indicating the time when the test packet is actually output from the sender device (also referred to herein as the "actually sent timestamp"). The sender device can calculate a time difference (also referred to herein as a "delta") between a first timestamp (a time test packet generated by an application) and a second timestamp (a time test packet output from the sender device), and compare the time difference with a threshold time value. If the sender device determines that there is a sender delay problem (e.g., the time difference exceeds the threshold time value), the sender device sends a message to notify the controller of the sender delay problem, and the controller in turn can, for example, generate and output a visual indication for display within a graphical user interface to indicate that one or more KPI measurements based on the test packet from the sender device are inaccurate and / or unreliable, or remove the KPI measurements.

[0010] In an example where a sender device experiences packet loss (e.g., due to a scheduling pause), the technology describes that the sender device obtains a count of test packets sent or lost from a network interface from a network interface. The sender device compares the number of packets sent from the network interface counter with the number of test packets sent from a test application, and sends a message to notify the controller of the result of the comparison.

[0011] For example, if the number of packets sent from the network interface counter is less than the number of packets sent from the test application, the sender device may notify the controller that one or more KPI measurements based on the test packets sent from the sender device are inaccurate, and the controller may in turn generate and output an indication for display on a graphical user interface to indicate that the KPI measurement is inaccurate and / or unreliable, or remove the KPI measurement. If the number of packets sent from the network interface counter is greater than the number of packets sent from the test application, and the sender device detects that one or more packets from the network interface are lost, the sender device may notify the controller that one or more KPI measurements based on the test packets sent from the sender device may be inaccurate, and the controller may in turn generate and output an indication (e.g., a warning) for display on a graphical user interface to indicate that the KPI measurement may be inaccurate and / or unreliable.

[0012] The techniques described herein may provide one or more technical advantages. For example, by implementing a device that detects sender problems and notifies a controller of sender problems, the controller may present additional information to a user or network administrator to enable the user or network administrator to better assess the accuracy of network measurements. This provides assurance to the user or network administrator that network measurements accurately reflect the performance of the network and dispels suspicion of measurements that appear to be attributed to the network when the problem is actually attributed to the sender device.

[0013] In one example of the technology described herein, a method includes: sending a test packet by an application of a sending device to a receiving device, the test packet including a first timestamp indicating the time when the test packet was generated by the application; obtaining a second timestamp by the sending device from a network interface of the sending device, the second timestamp indicating the time when the test packet was output from the sending device via the network interface; calculating a time difference between the second timestamp and the first timestamp by the sending device; comparing the time difference with a threshold time value by the sending device; and if the time difference exceeds the threshold time value, sending a message by the sending device to a controller to notify the controller that one or more performance measurements based on the test packet are inaccurate, wherein the controller is different from the sending device and the receiving device.

[0014] In another example of the technology described herein, a sending device comprises: a network interface; and one or more processors operably coupled to a memory, wherein the one or more processors are configured to: send a test packet by an application of the sending device to a receiving device, the test packet comprising a first timestamp indicating the time when the test packet was generated by the application; obtain a second timestamp from the network interface of the sending device, the second timestamp indicating the time when the test packet was output from the sending device via the network interface; calculate a time difference between the second timestamp and the first timestamp; compare the time difference with a threshold time value; and if the time difference exceeds the threshold time value, send a message to a controller to notify the controller that one or more performance measurements based on the test packet are inaccurate, wherein the controller is different from the sending device and the receiving device.

[0015] In another example of the technology described herein, a controller comprises: one or more processors operably coupled to a memory, wherein the one or more processors are configured to: receive from a receiving device that receives a test packet one or more performance measurement values ​​determined by the receiving device at least partially based on the test packet; receive a message from a sending device that sends the test packet to the receiving device to notify the controller that one or more performance measurement values ​​are inaccurate due to internal delays of the sending device; and output an indication of the one or more performance measurement values ​​and an indication of the inaccuracy of one or more performance measurement values ​​for display.

[0016] In another example of the technology described in this article, a method includes: sending one or more test packets by an application executed on a sending device to a receiving device; obtaining, by the sending device, a count of packets sent from a network interface of the sending device; obtaining, by the sending device, a count of one or more test packets sent by the application from the application; comparing, by the sending device, the count of packets sent from the network interface and the count of one or more test packets sent by the application to obtain a comparison result; and sending, by the sending device, a message to a controller, the message notifying the controller of the result of the comparison.

[0017] In another example of the technology described herein, a sending device comprises: a network interface; and one or more processors operably coupled to a memory, wherein the one or more processors are configured to: send one or more test packets from an application executed on the sending device to a receiving device; obtain, from the network interface of the sending device, a count of packets sent from the network interface; obtain, from the application, a count of one or more test packets sent by the application; compare the count of packets sent from the network interface with the count of one or more test packets sent by the application to obtain a result of the comparison; and send a message to a controller, the message notifying the controller of the result of the comparison.

[0018] In another example of the technology described herein, a controller includes: one or more processors operably coupled to a memory, wherein the one or more processors are configured to: receive from a receiving device that receives a test packet one or more performance measurement values ​​determined by the receiving device at least in part based on the one or more test packets; and receive from a sending device that sends the one or more test packets to the receiving device a message indicating a problem in sending the one or more test packets.

[0019] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram illustrating an example network system that provides network measurement inaccuracy detection according to one or more aspects of the techniques described in this disclosure.

[0021] Figure 2 is a block diagram illustrating an example network system for detecting network measurement inaccuracies by detecting sender delays according to one or more aspects of the techniques described in this disclosure.

[0022] Figure 3 is a block diagram illustrating an example network system for detecting network measurement inaccuracies by detecting sender packet losses according to one or more aspects of the techniques described in this disclosure.

[0023] Figure 4 is a block diagram illustrating an example computing device that performs one or more functions to detect network measurement inaccuracies in accordance with the techniques described in this disclosure.

[0024] Figure 5 is a block diagram illustrating an example of a controller capable of operating according to the techniques described in this disclosure.

[0025] Figure 6 is a flow chart illustrating example operations for detecting network measurement inaccuracies by detecting sender delays in accordance with one or more aspects of the techniques described in this disclosure.

[0026] Figure 7 is a flow chart illustrating example operations for detecting network measurement inaccuracies by detecting sender packet losses in accordance with one or more aspects of the techniques described in this disclosure. DETAILED DESCRIPTION

[0027] Figure 1 is a block diagram illustrating an example network system 2 ("network system 2") that provides network measurement inaccuracy detection according to techniques described herein.

[0028] The network system 2 may operate as a private network to provide packet-based network services to the subscriber device 16. That is, the network system 2 provides authentication and establishment of network access for the subscriber device 16 so that the subscriber device can begin exchanging packets with the public network 12, which may be a packet-based internal network or an external network such as the Internet.

[0029] exist Figure 1 In the example of , the network system 2 includes an access network 6 that provides connectivity to a public network 12 through a wide area network 7 (hereinafter referred to as "WAN 7"). The WAN 7 and the public network 12 can provide packet-based services that can be requested and used by subscriber devices 16. As examples, the WAN 7 and / or the public network 12 can provide bulk data transfer, voice over Internet Protocol (VoIP), Internet Protocol Television (IPTV), Short Message Service (SMS), Wireless Application Protocol (WAP) services, or customer-specific application services. The public network 12 can include, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual local area network (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the access network 6, an enterprise IP network, or some combination thereof. In various examples, the public network 12 is connected to a public WAN, the Internet, or to other networks. The public network 12 implements one or more packet data protocols (PDPs), such as IP (IPv4 and / or IPv6), X.25, or Point-to-Point Protocol (PPP), to enable packet-based delivery of public network 12 services.

[0030] Typically, subscriber devices 16 are connected to network devices 18A to 18B (collectively referred to as "network devices 18") through access network 6 to receive connectivity for user services of applications hosted by public network 12. Subscribers may represent, for example, businesses, residential subscribers, or mobile subscribers. Subscriber devices 16 may be, for example, personal computers, laptop computers, or other types of computing devices located behind customer equipment (CE) 11 that may provide local routing and switching functions. Each of subscriber devices 16 may run various software applications, such as word processing and other office support software, web browsing software, software that supports voice calls, video games, video conferencing, and email, etc. For example, subscriber devices 16 may be various network-enabled devices, commonly referred to as "Internet of Things" (IoT) devices, such as cameras, sensors (S), televisions, appliances, etc. In addition, subscriber devices 16 may include mobile devices that access data services of network system 2 via radio access network (RAN) 4. Example mobile subscriber devices include mobile phones, laptop or desktop computers with, for example, a wireless card, wireless-enabled netbooks, tablet computers, video gaming devices, pagers, smart phones, personal data assistants (PDAs), and the like.

[0031] The network service provider operates or, in some cases, leases elements of the access network 6 (e.g., network equipment) to provide packet transmission between the subscriber device 16 and the network equipment 18A. The access network 6 represents a network that aggregates data traffic from one or more of the subscriber devices 16 for transmission to / from the service provider's WAN 7. The access network 6 includes network nodes that execute communication protocols to transmit control and user data to facilitate communication between the subscriber device 16 and the network equipment 18A. The access network 6 may include a broadband access network, a wireless LAN, a public switched telephone network (PSTN), a customer premises equipment (CPE) network, or other types of access networks, and may include or otherwise provide connectivity to a cellular access network such as a radio access network (RAN) (e.g., RAN 4). Examples of RANs include networks compliant with fifth generation (5G) mobile networks, fourth generation (4G) mobile network Universal Mobile Telecommunications System (UMTS) architecture, an evolution of UMTS known as Long Term Evolution (LTE), Mobile IP standardized by the Internet Engineering Task Force (IETF), and other standards proposed by the 3rd Generation Partnership Project (3GPP), 3rd Generation Partnership Project 2 (3GGP / 2), and the WiMAX Forum.

[0032] Network devices 18A to 18B (collectively referred to as "network devices 18") may each be a customer edge (CE) router, a provider edge (PE) router, or other network device that provides connectivity between networks (e.g., access network 6 and public network 12). WAN 7 provides packet-based connectivity to subscriber devices 16 attached to access network 6 for access to public network 12 (e.g., the Internet). WAN 7 may represent a public network owned and operated by a service provider to interconnect multiple networks (which may include access network 6). In some examples, WAN 7 may implement multi-protocol label switching (MPLS) forwarding, and in such a case may be referred to as an MPLS network or MPLS backbone. In some cases, WAN 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers. Public network 12 may represent the Internet. Public network 12 may represent an edge network coupled to WAN 7 via a transport network 22 and one or more network devices (e.g., customer edge devices such as customer edge switches or routers). Public network 12 may include a data center. Figure 1 In the example of , network device 18B may exchange packets with service node 10 via virtual network 20 , and network device 18B may forward the packets to public network 12 via transit network 22 .

[0033] In the example of the network system 2 including the wired / broadband access network, the network device 18A or 18B can represent a broadband network gateway (BNG), a broadband remote access server (BRAS), an MPLS PE router, a core router or a gateway, or a cable modem termination system (CMTS). In the example of the network system 2 including the cellular access network as the access network 6, the network device 18A or the network device 18B can represent a mobile gateway, such as a gateway general packet radio service (GPRS) service node (GGSN), an access gateway (aGW), or a packet data network (PDN) gateway (PGW). In other examples, the functions described about the network device 18B can be implemented in a switch, a service card, or another network element or component. In some examples, the network device 18B itself can be a service node.

[0034] A network service provider that manages at least a portion of the network system 2 typically provides services to subscribers associated with systems (e.g., subscriber devices 16) that access the network system 2. The services provided may include, for example, traditional Internet access, VoIP, video and multimedia services, and security services. As described above with respect to the WAN 7, the WAN 7 may support multiple types of access network infrastructures connected to a service provider network access gateway to provide access to the provided services (e.g., services provided by the service node 10). In some cases, the network system may include subscriber devices 16 attached to multiple different access networks 6 having different architectures.

[0035] Typically, any one or more of the subscriber devices 16 may request authorization and data services by sending a session request to a gateway device such as network device 18A or 18B. The network device, in turn, may access a central server (not shown) such as an authentication, authorization, and accounting (AAA) server to authenticate one of the subscriber devices 16 requesting network access. Once authenticated, any of the subscriber devices 16 may send subscriber data traffic to WAN 7 to access and receive services provided by public network 20, and such packets may traverse network device 18A or network device 18B as part of at least one packet flow. In some examples, network device 18A may forward all authenticated subscriber traffic to public network 12, and if the subscriber traffic requires services on service node 10, network device 18B may apply services to data center 9 and / or direct specific subscriber traffic to data center 9. Applications (e.g., service applications) to be applied to subscriber traffic may be hosted on service node 10.

[0036] For example, the network system 2 includes a data center 9 with a cluster of service nodes 10, and the service nodes 10 provide an execution environment for virtualized network services. In some examples, each service node in the service node 10 represents a service instance. Each service node in the service node 10 can apply one or more services to a business flow. In this way, the network device 18B can guide the subscriber packet flow through the defined service set provided by the service node 10. That is, in some examples, each subscriber packet flow can be forwarded through a specific ordered combination of services provided by the service node 10, and each ordered set is referred to as a "service chain" in this article. As an example, the service node 10 can apply a stateful firewall (SFW) and security service, deep packet inspection (DPI), carrier-grade network address translation (CGNAT), traffic destination function (TDF) service, media (voice / video) optimization, Internet Protocol Security (IPSec) / Virtual Private Network (VPN) service, Hypertext Transfer Protocol (HTTP) filtering, counting, accounting, charging and / or load balancing of packet flows, or other types of services applied to network traffic.

[0037] In some examples, the network system 2 includes a software defined network (SDN) and a network function virtualization (NFV) architecture. In these examples, an SDN controller (not shown) can provide a controller for configuring and managing the routing and switching infrastructure of the network system 2.

[0038] Although shown as part of the data center 9, the service node 10 can be a network device coupled by one or more switches or virtual switches of the WAN 7. In one example, each service node in the service node 10 can be run as a virtual machine (VM) in a virtual computing environment. In addition, the computing environment can include an extensible cluster of general-purpose computing devices, such as services based on x86 processors. As another example, the service node 10 can include a combination of general-purpose computing devices and special-purpose devices. As a virtualized network service, the various network services provided by the service node 10 can be extended by the allocation of virtualized memory, processor utilization, storage and network policies, and horizontally by adding additional load-balanced VMs, just like in a modern data center. In other examples, the service node 10 can be a gateway device or other router. In a further example, the functions described for each service node in the service node 10 can be implemented in a switch, a service card, or another network element or component.

[0039] As described herein, network elements within the network system 2 can provide network monitoring services. For example, the network devices 18 and / or the service nodes 10 are configured as measurement points (e.g., test agents) to provide network monitoring services to determine, for example, performance and functionality, and interconnection of service chains. The test agents can be delivered from the cloud as a software as a service (SaaS) solution or can be deployed locally in an NFV environment. The network elements configured with the test agents can send and / or receive test packets to calculate one or more key performance indicators (KPIs) of the network, such as latency, delay (inter-frame gap), jitter, packet loss, throughput, etc. The test agent can send test packets according to various protocols, such as Hypertext Transfer Protocol (HTTP), Internet Control Message Protocol (ICMP), Speedtest, User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Operations, Administration and Maintenance (OAM) functions (e.g., Y.1731), Two-Way Active Measurement Protocol (TWAMP), Internet Protocol Television (IPTV) and Over the Top (OTT) protocols, VoIP phones and Session Initiation Protocol (SIP), mobile radio, remote packet inspection, and other protocols for measuring network performance.

[0040] exist Figure 1 In the example of FIG, network device 18A is configured with test agent 19A, network device 18B is configured with test agent 19B, service node 10A is configured with test agent 19C, service node 10B is configured with test agent 19D, service node 10C is configured with test agent 19E, and service node 10D is configured with test agent 19F. Although each of network device 18 and service node 10 is described as being configured with a test agent, any number of network devices and / or service nodes may be configured with a test agent to provide monitoring services.

[0041] For example, in order to test the network performance between the network device 18A and the service node 10A, the test agent 19A of the network device 18A may send a test packet to the test agent 19C of the service node 10A. The test packet may include a timestamp indicating the time when the test packet was sent. The timestamp sent by the network device 18A may be referred to as a "send timestamp" in this document, and the test agent 19C of the service node 10A may receive the test packet including the send timestamp and generate a timestamp indicating the time when the test packet was received. The timestamp generated by the test agent 19C of the service node 10A may be referred to as a "receive timestamp" in this document. The test agent 19C of the service node 10A may use the send timestamp and the receive timestamp to calculate one or more KPIs. For example, the test agent 19C of the service node 10A may determine the delay between the network device 18A and the service node 10A based on the difference between the receive timestamp and the send timestamp. The test agent 19C of the service node 10A sends one or more KPI measurements to the controller 14 (in Figure 1 24), the controller 14 can in turn present one or more KPI measurements to a user and / or network administrator, for example, via a graphical user interface. In this way, a device configured with a test agent can measure network performance between a sender device and a receiver device using a unidirectional test packet.

[0042] In some cases, the network device 18A (e.g., a sender device) may encounter problems when sending a test packet, which may cause the KPI measurement value of the receiver device (e.g., the service node 10A) to be inaccurate. As an example, a test application for a test agent 19A configured on the network device 18A can generate a send timestamp and forward the send timestamp to its network interface (e.g., a network interface card (NIC) such as an Ethernet card) to send the test packet to the service node 10A. In some cases, the network interface of the network device 18A may experience a delay (referred to herein as "sender delay") when sending a timestamp to the service node 10A. The service node 10A (e.g., a receiver device) can receive the test packet after the sender delay, generate a receive timestamp to indicate the time when the test packet is received, and calculate one or more KPI measurements (e.g., delay) based on the send timestamp and the receive timestamp, without considering the sender delay. The service node 10A can send the delay measurement value to the controller 14, and the controller 14 presents the delay measurement value to the user or network administrator, for example, through a graphical user interface. In these examples, without an indication that the delay measurements are inaccurate and / or untrustworthy, a user or network administrator may incorrectly attribute delay issues to the network between network device 18A and service node 10A, which may result in unnecessary troubleshooting and corrective actions.

[0043] In some examples, there may be pauses in the sender device performing the measurement process, for example, because the device is servicing other processes. For example, when other applications executing on network device 18A may require CPU cycles, network device 18A may experience pauses in sending test packets. When network monitoring services are resumed, network device 18A may typically send test packets in bursts to maintain a specific average packet rate per second. This may result in test packets being lost in network device 18A due to queue overflow.

[0044] According to the techniques described in the present disclosure, the network system 2 can provide detection of network measurement inaccuracies. For example, the network device 18A can detect sender delay or packet loss and notify the controller 14 that one or more KPI measurements based on the test packets sent from the network device 18A are inaccurate and / or unreliable.

[0045] In one example, network device 18A may detect a delay in outputting a test packet from network device 18A. Figure 2 As further described in , the test application of the network device 18A can generate a test packet including a first timestamp (also referred to herein as an "application send timestamp"), which indicates the time when the test packet was generated by the test application. The network device 18A can obtain a second timestamp generated by the network interface of the network device 18A, which indicates the time when the test packet was output from the network device 18A (also referred to herein as the "actual send timestamp"). The network device 18A can calculate the time difference between the first timestamp (actual send timestamp) and the second timestamp (application send timestamp), and compare the time difference with the threshold time value. If the network device 18A detects a delay in outputting the test packet from the network device 18A to the service node 10A, for example based on an evaluation of the time difference and the threshold time value, the network device 18A sends a message 26 to the controller 14 to notify the controller 14 that the network device 18A has detected a problem (e.g., a sender delay) when sending the test packet. In some examples, the network device 18A can send a message 26 including additional information associated with the sender delay, such as a delay value. In some examples, network device 18A may send message 26 after a defined measurement interval, wherein a plurality of test packets are delayed during the defined time interval.

[0046] In some examples, in response to receiving message 26 from network device 18A, controller 14 may generate a visual indication within a graphical user interface that one or more KPI measurements 24 are inaccurate and / or untrustworthy. In other examples, controller 14 may remove one or more KPI measurements 24. In some examples, controller 14 may adjust one or more KPI measurements 24 based on additional information (e.g., latency values) included in message 26.

[0047] In another example, network device 18A may detect packet loss (eg, due to a scheduling pause) while sending a test packet to service node 10A. Figure 3 As further described in , network device 18A may compare a count of test packets sent by a network interface (e.g., hardware of the network interface) with a count of test packets sent from a test application (e.g., whether the number of test packets sent by the network interface is less than the number of test packets sent from the test application). Based on the comparison, network device 18A may send a message 26 to controller 14 to notify controller 14 that network device 18A has detected a problem in sending test packets (e.g., a lost test packet). In some examples, network device 18A may send message 26 including additional information associated with the detected problem, such as the number of packets sent or lost.

[0048] In some examples, in response to receiving message 26 from network device 18A, controller 14 may generate a visual indication within a graphical user interface that one or more KPI measurements 24 are inaccurate and / or untrustworthy. In other examples, controller 14 may remove one or more KPI measurements 24. In some examples, controller 14 may adjust one or more KPI measurements 24 based on additional information included in message 26 (e.g., the number of packets sent or lost).

[0049] Although for example purposes Figure 1 The technology is described in the context of an SD-WAN in the present invention, but the technology of the present invention can be applied to any WAN assurance or network assurance environment.

[0050] Figure 2 is a block diagram illustrating an example network system for detecting network measurement inaccuracies by detecting sender delays according to one or more aspects of the techniques described in this disclosure. Figure 2 In the example of Figure 1 An example of a controller 14, and devices 204A to 204B may represent Figure 1 Any network device 18, service node 10, or any network element configured as a measurement point for measuring network performance. Network 207 may represent Figure 1The WAN 7, and / or other network connecting at least two network elements configured as measurement points.

[0051] In this example, the device 204A may include a confidence engine 220 to detect sender inaccuracies due to sender delays. As an example, a test application for a test agent 210A of the device 204A may generate a test packet 212 including a first timestamp 214 (application send timestamp) at the application layer of the open system interconnection (OSI) model. In response to the generation of the test packet 212, the test agent 210A forwards the test packet 212 to the network interface 208A (at Figure 2 2 (shown as "NI 208A" in the figure) to output the test packet to the device 204B at the data link layer or layer 2 of the OSI model. The network interface 208A outputs the test packet 212 to the device 204B and generates a second timestamp 216 (actual transmission timestamp) indicating the time when the test packet 212 was output from the network interface 208A. The confidence engine 220 of the device 204A obtains the first timestamp 214 from the test application and obtains the second timestamp 216 from the network interface 208A (e.g., from the kernel or hardware of the network interface).

[0052] The confidence engine 220 of the device 204A can compare the first timestamp 214 with the second timestamp 216 to determine whether to notify the controller 202 that one or more KPI measurements based on the test packet sent by the device 204A are inaccurate and / or untrustworthy. For example, the confidence engine 220 of the device 204A can calculate a time difference between the second timestamp 216 and the first timestamp 214, and use a threshold time value to evaluate the time difference. For example, a user or network administrator can define a delay value (e.g., in milliseconds) that controls whether the device 204A is to notify the controller 202 of a sender delay. In some examples, the confidence engine 220 can calculate multiple time differences between multiple actual send timestamps and multiple application send timestamps, and dynamically adjust the threshold based on the evaluation of the multiple time differences. If the time difference value exceeds the threshold time value (e.g., the time difference value is greater than the threshold time value), device 204A sends a message 218 to controller 202 to notify controller 202 that one or more KPI measurements based on the test packet 212 sent by device 204A are inaccurate and / or unreliable. In some examples, message 218 may include a flag or any type of notification to indicate the presence of sender delay when sending the test packet, the time difference value, and / or other information about the sender delay. In some examples, device 204A may send the message using a network configuration protocol (NETCONF) or other message protocol.

[0053] The controller 202 may receive the message 218 from the device 204A and receive one or more KPI measurements 224 from the device 204B. The one or more KPI measurements 224 are calculated using the application send timestamp 214 included in the test packet 212 and the receive timestamp generated by the device 204B in response to receiving the test packet 212. In response to receiving the message 218, in some examples, the controller 202 may generate a visual indication that the one or more KPI measurements 224 are inaccurate and / or untrustworthy (e.g., a warning symbol, highlighting the measurement in a particular color, etc.). In another example, the controller 202 may remove the one or more KPI measurements 224 in response to receiving the message 218. In another example, the controller 202 may receive a delta value included in the message 218 and use the delta value to adjust the one or more KPI measurements 224. For example, the device 204A may calculate a plurality of deltas from a plurality of application send timestamps and a plurality of actual send timestamps, and determine from the plurality of deltas that the device 204A continues to experience a delay (e.g., an average of 0.1 milliseconds) when sending the test packets. In this example, device 204A may send the delay value to controller 202 so that controller 202 may adjust one or more KPI measurements based on the delay value.

[0054] Figure 3 is a block diagram illustrating an example network system for detecting network measurement inaccuracies by detecting sender packet loss according to one or more aspects of the techniques described in this disclosure. Figure 3 In the example of Figure 1 An example of a controller 14, and devices 304A to 304B may represent Figure 1 Any network device 18, service node 10, or any network element configured as a measurement point for measuring network performance. Network 307 may represent Figure 1 WAN 7, or other network connecting at least two network elements configured as measurement points.

[0055] In this example, device 304A may include a confidence engine 320 to detect sender inaccuracies due to sender packet loss. As an example, a test application of a test agent 310A of device 304A may generate test packets 330, 332, and 334. Due to a pause in a network measurement process, device 304A may store the test packets in a process scheduling queue until the measurement process is resumed. For example, when the measurement process is resumed, device 304A may lose test packets 332 and 334 due to queue overflow. To detect sender inaccuracies due to sender packet loss, confidence engine 320 may retrieve test packets 330, 332, and 334 from network interface 308A (at Figure 3Confidence engine 320 may also obtain, from test agent 310A, a count of test packets sent by the test application. In this example, confidence engine 320 may obtain a count of the number of test packets sent and / or lost from the network interface (e.g., test packet 330). In this example, confidence engine 320 may obtain a count of one test packet sent (e.g., test packet 330) and / or a count of two test packets lost (e.g., test packets 332 and 334). Confidence engine 320 may also obtain, from test agent 310A, a count of test packets sent by the test application. In this example, confidence engine 320 obtains a count of three test packets sent by the test application.

[0056] The confidence engine 320 of the device 304A may compare the number of test packets sent by the network interface 308A with the number of test packets sent from the test application and send a message to the controller 302 to notify the controller of the result of the comparison. For example, if the number of test packets sent by the network interface 308A is less than the number of packets sent from the test application, the confidence engine 320 may send a message 318 to the controller 302 to notify the controller 302 that one or more KPI measurements based on the test packets from the device 304A are inaccurate and / or untrustworthy. In some examples, the confidence engine 320 of the device 304A may send a message indicating that the count of packets sent from the network interface is less than the count of test packets sent by the application, and the controller 302 may determine that one or more KPI measurements based on the test packets from the device 304A are inaccurate and / or untrustworthy.

[0057] Alternatively, or in addition, if the number of test packets sent by network interface 308A is greater than the number of test packets sent by the test application (e.g., if packets are sent from other applications and processes on a shared system), and losses are still detected on the network interface counters, confidence engine 320 can send message 318 to controller 302 to notify controller 302 that one or more KPI measurements based on the test packets from device 304A may be inaccurate and / or may not be trusted. In some examples, confidence engine 320 of device 304A can send a message indicating that the count of packets sent from the network interface is greater than the count of test packets sent by the application, and controller 302 can determine that one or more KPI measurements based on the test packets from device 304A may be inaccurate and / or may not be trusted.

[0058] In this example, confidence engine 320 determines that the number of test packets sent by network interface 308A (e.g., one) is less than the number of test packets sent by the test application (e.g., three). In response, message 318 is sent, which includes a flag or any type of notification to indicate the existence of a sender problem due to sender packet loss, the number of packets sent or lost, and / or other information about the sender problem due to sender packet loss. In some examples, device 304A can send the message using a network configuration protocol or other message protocol.

[0059] Controller 302 may receive message 318 from device 304A and receive one or more KPI measurements 324 from device 304B. One or more KPI measurements 324 are calculated using test packets 330. In some examples, controller 302 may generate a visual indication that one or more KPI measurements 324 are (or may be) inaccurate and / or untrustworthy (e.g., a warning symbol, highlighting the measurement in a particular color, etc.). In another example, controller 302 may remove one or more KPI measurements 324 in response to receiving message 318. In another example, controller 302 may receive a number of packets sent or lost included in message 318, and adjust one or more KPI measurements 324 using the number of packets sent or lost.

[0060] Figure 4 is a block diagram illustrating an example computing device that performs one or more functions to detect network measurement inaccuracies in accordance with the techniques described in this disclosure. Figure 4 The computing device 400 may represent Figure 1 any network device 18 and / or hosting service node 10, Figure 2 Device 204, and / or Figure 3 404. In this example, the computing device 400 includes a bus 442 that couples hardware components of the computing device 400 hardware environment. The bus 442 couples a network interface card (NIC) 430, a storage disk 446, and one or more microprocessors 410 (hereinafter referred to as "microprocessors 410"). The NIC 430 can be SR IOV-capable. In some cases, the front-end bus can couple the microprocessor 410 and the storage device 444. In some examples, the bus 442 can couple the storage device 444, the microprocessor 410, and the NIC 430. The bus 442 can represent a peripheral component interface (PCI) express (PCIe) bus. In some examples, a direct memory access (DMA) controller can control DMA transfers among components coupled to the bus 442. In some examples, components coupled to the bus 442 control DMA transfers among components coupled to the bus 442.

[0061] The microprocessor 410 may include one or more processors, each processor including an independent execution unit to execute instructions that conform to an instruction set architecture, and the instructions are stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC), or may be combined in one or more multi-core processors (or "many-core" processors), each of which is implemented using a single IC (i.e., a chip multiprocessor).

[0062] Disk 446 represents a computer-readable storage medium, which includes volatile and / or nonvolatile, removable and / or non-removable media implemented in any method or technology for storing information, such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disk (DVD) or other optical storage devices, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by microprocessor 410.

[0063] The main memory 444 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM) (e.g., DDR2 / DDR3 SDRAM), or static RAM (SRAM), flash memory, or any other form of fixed or removable storage media that can be used to carry or store required program code and program data in the form of instructions or data structures and can be accessed by the computer. The main memory 444 provides a physical address space consisting of addressable storage locations.

[0064] The network interface card (NIC) 430 includes one or more interfaces 432 configured to exchange packets using the links of the underlying physical network. The interface 432 may include a port interface card having one or more network ports. The NIC 430 may also include on-card memory to, for example, store packet data, timestamps, and counts of packets sent and / or lost. Direct memory access transmissions between the NIC 430 and other devices coupled to the bus 442 may read / write from / to the NIC memory.

[0065] The memory 444, NIC 430, storage disk 446, and microprocessor 410 may provide an operating environment for a software stack including an operating system kernel 414 executed in kernel space. Kernel 414 may represent, for example, Linux, Berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from Microsoft Corporation. In some instances, the operating system may execute a hypervisor 416 and one or more virtual execution elements, such as virtual machines 428A to 428B (collectively referred to as "virtual machines 428") managed by the hypervisor 416. Although illustrated and described for virtual machines, VNF instances 420A to 420M (collectively referred to as "VNF instances 420") may be executed by other operating environments such as containers (e.g., DOCKER containers). Additional examples of monitoring services provided by virtual execution elements are described in U.S. Provisional Patent Application 63 / 114,444, filed on November 16, 2020, entitled “Active Assurance for Virtualized Services,” the entire contents of which are incorporated herein by reference.

[0066] The computing device 400 executes a hypervisor 416 to manage virtual machines 428. Example hypervisors include Kernel-based Virtual Machine (KVM) for the Linux kernel, Xen, ESXi available from VMware, Windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor may include a virtual machine manager (VMM). The operating system including the kernel 414 provides an execution environment for one or more processes (e.g., a test agent application, as described in the present disclosure) in the user space 445.

[0067] The virtual machine 428 hosts a corresponding VNF instance 420. In some examples, the virtual machine 428 can host one or more VNF instances 420. Each VNF instance in the VNF instance 420 is configured to apply a network function to a packet. In this example, the VNF instance 420 is configured to host a test agent, such as test agent 429A to test agent 429B (collectively referred to as "test agent 429").

[0068] The kernel 414 includes a physical driver 425 to use the network interface card 430. The network interface card 430 may also implement SR-IOV to enable sharing of physical network functions (I / O) between one or more virtual execution elements, such as virtual machines 428 and / or containers ( Figure 4430). A shared virtual device such as a virtual function may provide dedicated resources so that each of the virtual execution elements may access the dedicated resources of the NIC 430, so that the NIC 430 appears as a dedicated NIC to each of the virtual execution elements. A virtual function may represent a lightweight PCIe function that shares physical resources with a physical function used by the physical driver 425 and other virtual functions. For a NIC 430 that supports SR-IOV, the NIC 430 may have thousands of virtual functions available according to the SR-IOV standard, but for I / O intensive applications, the number of configured virtual functions is typically much smaller.

[0069] The test agent 429 can communicate with other test agents using various protocols to measure Internet performance and / or network performance. For example, protocols for measuring Internet performance may include Hypertext Transfer Protocol (HTTP), Internet Control Message Protocol (ICMP), and / or speed tests. Protocols for measuring network performance may include User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Operations, Administration, and Maintenance (OAM) functions (e.g., Y.1731), Two-Way Active Measurement Protocol (TWAMP), Internet Protocol Television (IPTV) and Over-the-Top (OTT) protocols, VoIP telephony and Session Initiation Protocol (SIP), mobile radio, remote packet inspection, and other protocols for measuring network performance. The test agent 429 may communicate with a controller (e.g., Network Configuration Protocol (NETCONF)) using a messaging protocol such as the Network Configuration Protocol (NETCONF). Figure 1 Controller 14, Figure 2 Controller 202, or Figure 3 Controller 302) communicates.

[0070] According to the techniques described herein, computing device 400 may include confidence engine 460 for detecting network measurement inaccuracies by detecting sender delay or packet loss. Confidence engine 460 may include code that may be executed by microprocessor 410. Confidence engine 460 may be one or more computer processes in user space 445.

[0071] The computing device 400 can be accessed via a management interface (e.g., a command line interface (CLI) or a graphical user interface (GUI)) Figure 4 4) to receive a command that enables the sender inaccuracy notification mode to perform the techniques described in the present disclosure. As an example, the confidence engine 460 is configured to detect sender inaccuracy due to sender delay. For example, the confidence engine 460 can obtain a first timestamp 472 (e.g., Figure 2The confidence engine 460 may also be configured to obtain a second timestamp 470 (e.g., Figure 2 426). For example, the confidence engine 460 may use the kernel 414 of the computing device 400 to perform a system call to the physical driver 425 to obtain the timestamp 470 from the hardware of the NIC 430. In some examples, the confidence engine 460 may obtain the actual send timestamp from one or more virtual network interfaces that perform packet I / O or otherwise process packets. For example, the computing device 400 may use a virtual hardware component (e.g., an SR-IOV virtual function) enabled by the NIC 430 to perform packet I / O and receive / send packets. In these examples, the confidence engine 460 may obtain the actual send timestamp from the packet I / O component.

[0072] The confidence engine 460 is also configured to compare the first timestamp 472 (the application send timestamp) with the second timestamp 470 (the actual send timestamp) to determine whether to notify the controller (e.g., Figure 2 The confidence engine 460 may be configured to determine that one or more KPI measurements of the test packet sent by the computing device 400 are inaccurate and / or unreliable based on the controller 202 of the computing device 400. For example, the confidence engine 460 is configured to calculate a time difference between the second timestamp 470 and the first timestamp 472, and to evaluate the time difference with a threshold time value. The computing device 400 may include a management interface to receive input from a user or network administrator to specify a threshold time value, and the computing device 400 may store the threshold time value. If the time difference exceeds the threshold time value (e.g., the time difference is greater than the threshold time value), the confidence engine 460 is configured to generate a message (e.g., Figure 2 400) to notify the controller that one or more KPI measurements based on the test packets sent by the computing device 400 are inaccurate and / or untrustworthy. For example, the confidence engine 460 may generate a message including a flag or any type of notification using, for example, NETCONF to indicate the presence of a sender delay, a time difference, and / or other information about the sender delay.

[0073] As another example, the confidence engine 460 is configured to detect sender inaccuracies due to packet loss. For example, the NICs 430 may each include a counter for counting the number of packets sent or lost and storing the count in a memory of the IFC. The confidence engine 460 is configured to obtain a count of packets sent or lost from the NICs 430 (e.g., Figure 3 For example, confidence engine 460 may use kernel 414 of computing device 400 to perform a system call to physical driver 425 to obtain count 474 from NIC 430. Confidence engine 460 may also obtain count 476 of test packets sent from the test application (e.g., Figure 3 Count 338).

[0074] The confidence engine 460 is also configured to compare the count 474 of packets sent or lost by the NIC 430 with the count 476 of test packets sent from the test application. For example, the confidence engine 460 is configured to determine whether the number of test packets sent by the NIC 430 is less than the number of packets sent from the test application. If the number of test packets sent by the NIC 430 is less than the number of packets sent from the test application, the confidence engine 460 is configured to generate a message (e.g., Figure 3 318) to notify the controller that one or more KPI measurements based on the test packets sent by the computing device 400 are inaccurate and / or untrustworthy. For example, the confidence engine 460 may generate a message including a flag or any type of notification using, for example, NETCONF to indicate that there is a sender problem due to packet loss, the number of packets sent or lost, and / or other information about a sender problem due to packet loss.

[0075] A user or administrator can enable (or disable) a sender inaccuracy notification mode to notify the controller of sender inaccuracies due to sender delays, to notify the controller of sender inaccuracies due to sender packet losses, or both.

[0076] In this way, computing device 400 can detect any sender inaccuracies due to sender delays or packet losses and notify the controller of the sender inaccuracies so that the controller can generate a visual indication (e.g., a warning symbol, highlighting the measurement in a particular color, etc.) that one or more KPI measurements are (or may be) inaccurate and / or untrustworthy, or adjust one or more KPI measurements based on information received from the notification message.

[0077] Figure 5Controller 500 may include, for example, a server or a network controller, and may represent Figure 1 Controller 14, Figure 2 The controller 202 and Figure 3 An example instance of a controller 302 is provided.

[0078] exist Figure 5 In the example of FIG. 5 , the controller 500 includes a control unit 502 coupled to a network interface 520 to exchange packets with other network devices via an inbound link 522 and an outbound link 524. The control unit 502 may include one or more processors ( Figure 5 ), software instructions such as those used to define software or computer programs, stored in a computer-readable medium, which includes a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or random access memory (RAM)) or any other type of volatile or non-volatile memory that stores instructions to cause one or more processors to perform the techniques described herein. Alternatively, or in addition, the control unit 202 may include dedicated hardware, such as one or more integrated circuits, one or more application-specific integrated circuits (ASICs), one or more application-specific processors (ASSPs), one or more field-programmable gate arrays (FPGAs), or any combination of the foregoing examples of one or more dedicated hardware, for performing the techniques described herein.

[0079] The control unit 502 provides an operating environment for the network service application 504 and the network measurement module 512. In one example, these modules can be implemented as one or more processes executed on one or more virtual machines of one or more servers. That is, although generally illustrated and described as being executed on a single controller 500, aspects of these modules can be delegated to other computing devices.

[0080] The network service application 504 represents one or more processes that provide services to the clients of the network, and the network includes a controller 500 to provide visualization of network performance measurements calculated based on test packets exchanged by test agents within the network. The network service application 504 provides services such as performance measurement visualization, session management of test agents, etc. Each network service application 504 may include a client interface 506, and one or more client applications request services through the client interface 506. For example, the client interface 506 can represent a command line interface (CLI) or a graphical user interface (GUI). The client interface 506 can also or alternatively provide an application programming interface (API) such as a web service to the client application. A user or administrator can enable (or disable) a sender inaccuracy notification mode through the client interface 506 to notify the controller of sender inaccuracies caused by sender delays, notify the controller of sender inaccuracies caused by sender packet loss, or both.

[0081] The network service application 504 sends a request for visual network performance measurements to the network measurement module 512. The network measurement module 512 can generate a visual indication, such as by a receiving test agent (e.g., Figure 2 Test Agent 210B or Figure 3 The controller 500 may provide an indication of a performance measurement value calculated by the test agent 310B of the receiving test agent. For example, the controller 500 may establish a session (e.g., NETCONF) with the receiving test agent and receive one or more performance measurements from the test agent via the inbound link 222. The controller 500 may store the one or more performance measurements in the performance measurement data storage repository 518. The network measurement module 512 may output a visual indication of the one or more performance measurements stored in the performance measurement data storage repository 518 for display.

[0082] In some examples, network measurement module 512 can detect the sender test agent (e.g., Figure 2 Test Agent 210A or Figure 3 The test agent 310A of the sender test agent receives a message notifying the controller 500 that one or more performance measurements are inaccurate. For example, the controller 500 may establish a session (e.g., NETCONF) with the sender test agent and receive a message notifying the controller 500 that one or more performance measurements calculated based on the test packets sent by the sender test agent are inaccurate. In some examples, the network measurement module 512 may receive a message including a flag or any type of notification to indicate that there is a sender delay, a time difference, and / or other information about the sender delay when sending the test packets.

[0083] The controller 500 may store the message in the message data store 516. In response to receiving the message, the network measurement module 512 may correlate one or more performance measurements received from the receiving test agent with the message in the message data store 516. For example, the network measurement module 512 may determine one or more performance measurements calculated based on one or more test packets sent by the sending test agent and correlate the message in the message data store 516 received from the sending test agent. The network measurement module 512 may output for display a visual indication of the one or more performance measurements and an indication that the one or more performance measurements are inaccurate. In some examples, in response to receiving the message, the network measurement module 512 may remove the one or more performance measurements calculated based on the test packets sent by the sending test agent so that the controller 500 does not output the one or more performance measurements for display to the user.

[0084] In some examples, the network measurement module 512 may receive a message that notifies the controller 500 of the result of a comparison by the sender device of a packet count sent from a network interface of the sender device and one or more test packet counts sent by an application executed by the sender device. For example, the network measurement module 512 may receive a message including an indication that the packet count sent from the network interface of the sender device is less than the count of one or more test packets sent by an application executed by the sender device. In this example, the network measurement module 512 may determine that one or more performance measurements received from the receiving test agent are inaccurate and / or unreliable based on the indication that the packet count sent from the network interface of the sender device is less than the count of one or more test packets sent by the application executed by the sender device. The network measurement module 512 may output a visual indication of the one or more performance measurements and an indication that the one or more performance measurements are inaccurate for display.

[0085] As another example, the network measurement module 512 may receive a message including an indication that a count of packets sent from a network interface of a sender device is greater than a count of one or more test packets sent by an application executed by the sender device. In this example, the network measurement module 512 may determine that one or more performance measurements received from a receiving test agent may be inaccurate and / or untrustworthy based on the indication that a count of packets sent from a network interface of a sender device is greater than a count of one or more test packets sent by an application executed by the sender device. The network measurement module 512 may output for display a visual indication of the one or more performance measurements and an indication that the one or more performance measurements may be inaccurate.

[0086] Figure 6is a flowchart illustrating example operations for detecting network measurement inaccuracies by detecting sender delays according to one or more aspects of the technology described in this disclosure. For ease of illustration, for example, with respect to Figure 2 Device 204A, Figure 2 Controller 202, Figure 4 The computing device 400 and Figure 5 The example operation is described with reference to the controller 500 of FIG. 5 , but may represent any computing device and controller described in the present disclosure. The following are the steps of the process, although other examples of processes performed in the technology of the present disclosure may include additional steps or may not include some of the steps listed below.

[0087] exist Figure 6 In the example of FIG. 6 , device 204A sends a test packet 212 including a first timestamp 214 (application send timestamp) to device 204B (602). The first timestamp 214 can indicate the time when the test application generated the test packet. Device 204A obtains a second timestamp 216 (actual send timestamp), which indicates the time when the test packet 212 was output from the network interface of device 204A (604). For example, when network interface 208A outputs test packet 212 to device 204B, device 204A can obtain a timestamp generated by hardware of network interface 208A (e.g., via physical driver 425).

[0088] Device 204A calculates a time difference between second time marker 216 and first time marker 214 (606). Device 204A then compares the time difference to a threshold time value (608). For example, a user or network administrator may use an interface (e.g., a CLI or GUI) to define a time difference (e.g., in milliseconds) that defines whether device 204A is to notify controller 202 of a sender delay. If the time difference exceeds the threshold (“yes” of step 608), device 204A sends a message to controller 202 to notify the controller that one or more KPI measurements based on the test packet are inaccurate and / or unreliable. In some examples, the message may include a flag or any type of notification to indicate the presence of a sender delay when sending the test packet, a time difference, and / or other information about the sender delay. In some examples, device 204A may send the message using NETCONF or other messaging protocols. If the time difference does not exceed the threshold (“No” at step 608), device 204A does not send a message to controller 202 and may perform a subsequent test packet. Figure 6 The process described in .

[0089] The receiving test agent 210B of the network device 204B may receive the test packet 212 including the first timestamp 214 (612). The test agent 210B of the network device 204B may generate a third timestamp (receive timestamp) indicating the time when the test packet was received (614). The test agent 210B of the network device 204B may calculate one or more performance measurements based on the first timestamp (the time when the test packet 212 was generated) and the third timestamp (the time when the test packet 212 was received) (616). The test agent 210B of the network device 204B sends the one or more performance measurements to the controller 202 (618).

[0090] Controller 202 may receive one or more performance measurements from receiving test agent 210B of network device 204B (620) and, in some examples, receive a message from sending test agent 210A of network device 204A notifying controller 202 that one or more performance measurements based on test packet 212 are inaccurate (622). In response to receiving the message notifying controller that the one or more performance measurements are inaccurate, controller 202 outputs an indication of the one or more performance measurements and an indication of the one or more performance measurements being inaccurate for display (624). In some examples, controller 202 may remove the one or more performance measurements from the display output.

[0091] Figure 7 is a flowchart illustrating example operations for detecting network measurement inaccuracies by detecting sender packet loss according to one or more aspects of the techniques described in this disclosure. Figure 3 Equipment 304A, Figure 3 The controller 302, Figure 4 The computing device 400 and Figure 4 The controller 400 of describes an example operation, but may represent any computing device and controller described in the present disclosure. The following are the steps of the process, although other examples of processes performed in the technology of the present disclosure may include additional steps or may not include some of the steps listed below.

[0092] exist Figure 7 In the example of FIG. 4 , device 304A sends one or more test packets, such as test packets 330, 332, and 334, to device 304B (702). Device 304A obtains packets sent from network interface 308A or counts of packets lost (704). For example, when network interface 308A sends packets out of the interface, the confidence engine of device 304A can obtain counts of packets sent or lost generated by hardware of network interface 308A (e.g., via physical driver 425). Device 304A also obtains counts of packets sent from a test application (706).

[0093] Device 304A compares the packet count sent from network interface 308A with the packet count sent from the test application (708). If the packet count sent from the network interface is less than the packet count sent from the test application ("Yes" of step 708), device 304A sends a message to controller 302 to notify the controller that one or more KPI measurements based on the test packet sent by device 304A are inaccurate and / or unreliable. In some examples, the confidence engine 320 of device 304A may send a message indicating the result of the comparison, for example, the count of the packet sent from the network interface is less than the count of the test packet sent by the application, and controller 302 may determine that one or more KPI measurements based on the test packet from device 304A are inaccurate and / or unreliable. In some examples, the message may include a flag or any type of notification to indicate the presence of sender packet loss when sending the test packet, the number of packets sent or lost, and / or other information about sender packet loss. In some examples, device 304A may send the message using NETCONF or other message protocols. If the packet count sent from the network interface is greater than the packet count sent from the test application, and loss is still detected on the interface counter ("No" of step 708), device 304A sends a message to controller 302 to notify the controller that one or more KPI measurements based on the test packets sent by device 304A may be inaccurate and / or unreliable (712). In some examples, confidence engine 320 of device 304A may send a message indicating the result of the comparison, for example, the packet count sent from the network interface is greater than the count of test packets sent by the application, and controller 302 may determine that one or more KPI measurements based on the test packets from device 304A may be inaccurate and / or unreliable.

[0094] Receiving test agent 310B of network device 304B may receive test packet 330 (714). Test agent 310B of network device 304B may calculate one or more performance measurements based on the test packet (test packet 330) (716). Test agent 210B of network device 204B sends the one or more performance measurements to controller 202 (718).

[0095] The controller 302 may receive one or more performance measurements from a receiver test agent 310B of the network device 304B (720), and in some examples, receive a message from a sender test agent 310A of the network device 304A notifying the controller 302 that one or more performance measurements based on a test packet sent by the sender test agent 310A of the network device 304A are inaccurate or potentially inaccurate (722). In response to receiving the message notifying the controller that the one or more performance measurements are inaccurate or potentially inaccurate, the controller 302 outputs an indication of the one or more performance measurements and an indication that the one or more performance measurements are inaccurate or potentially inaccurate for display (724). In some examples, the controller 302 may remove the one or more inaccurate performance measurements from the output display. In some examples, the controller 302 may output an indication that the one or more performance measurements may be inaccurate for display, such as a visual warning (e.g., a warning icon).

[0096] The technology described herein can be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components can be implemented together in an integrated logic device, or implemented separately as a discrete but interoperable logic device or other hardware device. In some cases, the various features of the electronic circuit can be implemented as one or more integrated circuit devices, such as an integrated circuit chip or a chipset.

[0097] If implemented in hardware, the present disclosure may be directed to a device such as a processor or an integrated circuit device such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0098] Computer-readable media may form part of a computer program product, which may include packaging materials. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0099] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may mean that the storage medium is not contained in a carrier or propagating signal. In some examples, non-transitory storage media may store data that can change over time (e.g., in RAM or cache).

[0100] The code or instructions may be software and / or firmware executed by a processing circuit, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described in the present disclosure may be provided in software modules or hardware modules.

[0101] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for measuring, comprising: Sending, by an application executed on a sender device, a test packet to a receiver device, the test packet comprising a first timestamp indicating a time when the test packet was generated by the application executed on the sender device; Obtaining, by the sender device, a second timestamp from a network interface of the sender device, the second timestamp indicating a time when the test packet is sent from the sender device via the network interface of the sender device; Calculating, by the sending device, a time difference between the second timestamp and the first timestamp; The sending device compares the time difference value with a threshold time value; as well as The sending device sends a message to a controller based on comparing the time difference value with the threshold time value and determining that the time difference value exceeds the threshold time value, wherein the message indicates that one or more performance measurement values ​​of the controller based on the test group are inaccurate, wherein the controller is different from the sending device and the receiving device. The method of claim 1 , wherein the second timestamp is generated by a hardware component of the network interface. The method of claim 2 , wherein the hardware component comprises a network interface card. The method of claim 1 , wherein the message includes a flag indicating a problem in sending the test packet. The method of claim 1 , wherein the message indicates the time difference.

6. The method according to any one of claims 1 to 5, further comprising: The threshold time value is dynamically adjusted by the sender device based on an evaluation of a plurality of time difference values.

7. The method according to any one of claims 1 to 5, further comprising: Before said comparison: receiving, via a management interface of the sender device, configuration information defining the threshold time value; as well as The threshold time value is stored by the sender device.

8. The method according to any one of claims 1 to 5, further comprising: Before said calculation: receiving, via a management interface of the sender device, a command to enable a sender inaccuracy notification mode; as well as wherein the calculating, the comparing and the sending are respectively performed in response to receiving the command.

9. The method according to any one of claims 1 and 4, further comprising: The sending device sends a plurality of test packets; obtaining, by the sender device, from the network interface of the sender device, a count of packets sent from the network interface; obtaining, by the sending device from the application, a count of the one or more test packets sent by the application; comparing, by the sender device, the count of packets sent from the network interface with the count of the one or more test packets sent by the application to obtain a result of the comparison; determining, based on the comparison, that the count of packets sent from the network interface is less than the count of the one or more test packets sent by the application; as well as A message is sent by the sending device to the controller indicating that the packet count sent from the network interface is less than the count of the one or more test packets sent by the application, the message indicating that the one or more performance measurements based on the one or more test packets are inaccurate.

10. The method according to any one of claims 1 and 4, further comprising: The sending device sends a plurality of test packets; obtaining, by the sender device, from the network interface of the sender device, a count of packets sent from the network interface; obtaining, by the sending device from the application, a count of the one or more test packets sent by the application; comparing, by the sender device, the count of packets sent from the network interface with the count of the one or more test packets sent by the application to obtain a result of the comparison; determining, based on the comparison, that the count of packets sent from the network interface is greater than the count of the one or more test packets sent by the application; detecting one or more packet losses from the network interface; as well as A message is sent by the sending device to the controller indicating that the packet count sent from the network interface is greater than the count of the one or more test packets sent by the application, the message indicating that the one or more performance measurements based on the one or more packets are inaccurate.

11. A sending device, comprising: Network interface; as well as One or more processors operably coupled to the memory, wherein the one or more processors are configured to: Sending, by an application executed on the sender device, a test packet to a receiver device, the test packet comprising a first timestamp indicating a time when the test packet was generated by the application executed on the sender device; obtaining a second timestamp from a network interface of the sender device, the second timestamp indicating a time when the test packet is sent from the sender device via the network interface of the sender device; Calculating a time difference between the second timestamp and the first timestamp; comparing the time difference value to a threshold time value; as well as Based on comparing the time difference value with the threshold time value and the time difference value exceeding the threshold time value, a message is sent to a controller to notify the controller that one or more performance measurements based on the test group are inaccurate, wherein the controller is different from the sending device and the receiving device.

12. The sender device of claim 11, wherein the second timestamp is generated by a hardware component of the network interface.

13. The sender device of claim 12, wherein the hardware component comprises a network interface card.

14. The sender device of claim 11, wherein the message includes a flag indicating the presence of a sender delay.

15. The sender device of claim 11, wherein the message indicates the time difference value.

16. The sender device of claim 11, wherein the threshold time value is dynamically adjusted by the sender device based on an evaluation of a plurality of time difference values.

17. The sender device according to any one of claims 11 to 15, wherein the one or more processors are further configured to: Before said comparison: receiving, via a management interface of the sender device, configuration information defining the threshold time value; and The threshold time value is stored.

18. The sender device according to any one of claims 11 to 15, wherein the one or more processors are further configured to: Before said calculation: receiving, via a management interface of the sent sender device, a command to enable a sender inaccuracy notification mode; and wherein the calculating, the comparing and the sending are respectively performed in response to receiving the command.

19. A controller comprising: One or more processors operably coupled to the memory, wherein the one or more processors are configured to: receiving, from a recipient device receiving the test packets, one or more performance measurements determined by the recipient device based on the test packets; receiving a message from a sender device that sent the test packet to the receiver, the message indicating to the controller that the one or more performance measurements are inaccurate due to an internal delay of the sender device, wherein the internal delay of the sender device is determined by calculating a time difference between a first timestamp indicating a time when the test packet was generated by the sender device and a second timestamp indicating a time when the test packet was sent from the sender device, and determining that the time difference exceeds a threshold time value; as well as Based on receiving the message indicating that the one or more performance measurements are inaccurate, outputting an indication that the one or more performance measurements are inaccurate.

20. The controller of claim 19, wherein the one or more processors are further configured to: determining whether the one or more performance measurements are calculated from the test group; Based on determining that the one or more performance measurements are calculated from the test packet, the message is associated with the one or more performance measurements.

Images