Application-specific network telemetry and diagnostics

Abstract

Embodiments of the present disclosure relate to application-specific network telemetry and diagnostics. In some embodiments, a method includes identifying, by an NIC of a node executing therein by an application sender of an application, a network data unit for telemetry metadata insertion based on performing a matching operation on information associated with the network data unit, wherein the network data unit is transmitted from an application transmitter to an application receiver executing on another node; inserting telemetry metadata into the network data unit based on a first success of the first matching operation, the telemetry metadata comprising network information and application specific information; stripping the telemetry metadata from the network data unit by a last hop device along the data flow path; generating, by the last hop device, a telemetry metadata report using the telemetry metadata; and providing the telemetry metadata report to an application collector executing on the second node.

Description

[0001] Cross - Reference to Related Applications

[0002] This invention was made with government support under contract number H98230-15-D-0022 / 0007 awarded by the Maryland Procurement Office. The government has certain rights in this invention. Background Art

[0003] Applications typically cause data to be sent over a network. Thus, application performance can be related to the performance of the network through which the data traverses. However, there is often a disconnect between network telemetry data and the specific application that caused the data to be sent and / or received, which makes it difficult to correlate application behavior with network operations. Brief Description of the Drawings

[0004] Certain embodiments discussed herein will be described with reference to the drawings listed below. However, the drawings illustrate only certain aspects or implementations of the embodiments described herein by way of example and are not meant to limit the scope of the claims.

[0005] Figure 1 A block diagram showing an example system for implementing techniques for obtaining application-specific telemetry metadata in accordance with one or more embodiments disclosed herein;

[0006] Figure 2 An overview showing an example method for obtaining application-specific telemetry metadata in accordance with one or more embodiments disclosed herein;

[0007] Figure 3 An example system for obtaining application-specific telemetry metadata in accordance with one or more embodiments disclosed herein;

[0008] Figure 4A An example network data unit without telemetry metadata in accordance with one or more embodiments disclosed herein;

[0009] Figure 4B An example network data unit with telemetry metadata inserted therein in accordance with one or more embodiments disclosed herein; and

[0010] Figure 5 A block diagram showing a computing device in accordance with one or more embodiments of the present disclosure. Detailed Description

[0011] For applications executing on two or more nodes (e.g., computing devices), application data is typically sent from one part of the application (e.g., an application sender) to another part of the application (e.g., an application receiver) in the form of network data units (e.g., packets). Such applications can include an increasing possible scale (e.g., tens of thousands of nodes). However, when network data units traverse the network from an application sender to an application receiver, it can be challenging to associate network data units from a particular application with network telemetry information because application instrumentation does not know (and generally cannot know) the network devices that the network data units traverse, and conversely, network devices generally do not know which particular applications are sending and / or receiving data.

[0012] Network telemetry data collected for network data units traversing the network (e.g., the network devices traversed, ingress and egress ports, ingress and egress timestamps, queue / buffer depth, routing and forwarding decision information, local and / or aggregate latency, etc.) is typically sent to a collector device that collects such information for the network. However, associating such aggregated network telemetry data with the particular applications sending and receiving the data can be challenging and requires a significant amount of post - processing technical overhead to deduce which applications contribute to which network data flows. Additionally, aggregation of network telemetry data can encounter other problems, including but not limited to: aliasing (where timestamps are associated with sampling times rather than actual event times); the fact that sampling further increases the challenge of associating network telemetry data with applications because application information may not correspond to the sampling times used; the fact that security measures must be taken to help ensure that information about a given application is not shared with users of other applications, etc. Additionally, as the scale of applications grows to a larger number of nodes, problems such as those discussed above may be exacerbated because a centralized collector may be receiving data from a large number of nodes for each application and any number of applications that can use a large - scale network.

[0013] Such characteristics can make it difficult to clearly understand what is actually being sent by an application for sending and receiving data over a shared information data path (e.g., a series of dynamically - selected hops along a data flow path to a destination). In one or more embodiments, the phrase "shared data path" refers to a path over which data can traverse within a network, which can be used by any number of entities (e.g., applications) that can send and / or receive data via the network, making the data path "shared". In one or more embodiments, the data path for a given network data unit through the network can be dynamically determined (e.g., at each hop), and thus, the data path may not be the same each time an entity sends a network data unit to the same other entity.

[0014] To address at least a portion of the above problems, embodiments disclosed herein provide techniques that allow application-specific data streams and network telemetry to be correlated in real time as network data units traverse network device hops in a network path from an application sender (e.g., executing on a first node or sender node of an application) to an application receiver (e.g., executing on a second node or receiver node of an application). Telemetry metadata (including information about the application and network telemetry information) can be inserted into the network data unit between standard packet headers (e.g., between the Layer 4 (L4) header and the Layer 3 (L3) header). Such telemetry metadata can be added by any device (e.g., a network interface card (NIC), a network device (e.g., a switch, a router, etc.)) along the data flow path of the network data unit as the network data unit traverses the network from the application sender to the application receiver.

[0015] In one or more embodiments, the telemetry metadata is removed from the network data unit before the network data unit is delivered to the application receiver. In one or more embodiments, the telemetry metadata stripped from each hop of the network data unit is provided to an application collector executing on the same node as the application receiver. Thus, in one or more embodiments, the application collector receives a report that includes various network telemetry information and application information of the application associated with the data stream of which the network data unit is a part. Thus, the network telemetry data can be directly correlated with the application that sends and receives the network data unit, without the need for a central collector to perform challenging post-processing to attempt such correlation, and avoiding the aliasing problem (e.g., as in sFlow telemetry) seen when performing network data unit sampling. Additionally, since the telemetry metadata report is provided to the application collector on the receiving node of the network data unit, the telemetry collection process can scale accordingly as the size of the application (e.g., the number of nodes) scales.

[0016] In one or more embodiments, the telemetry metadata is added to the network data unit at each hop of the data flow path (e.g., through the network from the application sender to the application receiver), which data flow path includes devices configured to perform the techniques disclosed herein. As used herein, a hop can be any device or component through which the network data unit traverses along the data flow path from the application sender to the application receiver, such as a NIC (e.g., a device installed in the node on which the application executes) and network devices of the network (e.g., switches, routers, etc.). All or any part of the hops along the data flow path can be configured / enabled to add the telemetry metadata to the network data unit.

[0017] Any hopping device configured as such can include matching action rules. In one or more embodiments, the device performs a match on any one or more parts of any header or field of a network data unit, e.g., Layer 2 (L2), L3, and / or L4 fields, which can include but are not limited to source and destination Internet Protocol (IP) addresses, source and destination Media Access Control (MAC) addresses, source and destination port numbers (e.g., Transmission Control Protocol (TCP) and / or User Datagram Protocol (UDP) port numbers), Virtual Local Area Network (VLAN) tags, Virtual Network Identifier (VNI), flow labels, Differentiated Services Code Point (DSCP) values, etc.

[0018] In one or more embodiments, once a match is identified, the corresponding action is to insert telemetry metadata into the network data unit, which occurs at each hop for which the embodiments disclosed herein are enabled. In one or more embodiments, in the first instance of the insertion of telemetry metadata, the telemetry header is present at the hop along with the telemetry metadata, thus signaling the start of the telemetry metadata within the network data unit. The total size of the added telemetry metadata added by the device at a hop can be small (e.g., sixty-four bytes), such that there is little or no impact on the overall performance of the network. Any impact that may occur can be mitigated to some extent by, for example, configuring the device to add metadata only to a specific portion of the matching network data units (e.g., one out of every N network data units) rather than all matching network data units, by not adding telemetry metadata to packets below a configured size threshold, etc. In one or more embodiments, a device after the first hop (e.g., the NIC of the node on which the application sender executes) can be capable of matching on fields in the telemetry metadata added by a previous device.

[0019] As an example, a network data unit sent from an application sender can be provided to the NIC of the node on which the application sender executes, such that the NIC can send the network data unit into the network (e.g., the network data unit enters the network). The NIC can perform a match using all or any part of the information received in association with the network data unit to be sent. When a match is identified, the NIC can insert telemetry metadata into the network data unit (e.g., between the L4 and L3 headers). The telemetry metadata can include both network information and application information, thus allowing the network data unit to be associated with a specific application (or any part thereof). As an example, the application information can include an application identifier (identifying the entire application), and can additionally or alternatively include a token or other application instrumentation indicating the application execution phase / level, application library call markers, etc. Such instrumentation can be added explicitly from the application, added implicitly through a linked communication library, and / or added by matching specific communication identifiers.

[0020] The information included in the telemetry metadata can be any type of information without departing from the scope of the embodiments disclosed herein, and this information can include telemetry data as well as telemetry-related status information. Examples include but are not limited to device identifiers and related information, ingress ports and egress ports, latency information, ingress timestamps and egress timestamps, queue and buffer status, information about transformations performed on network data units, forwarding and routing decisions, and / or any optional programmable field information (e.g., functions performed by an application, application execution phases, more specific application or application part identification information, operation codes recognizable by an application user, any kind of free text, etc.). In one or more embodiments, the telemetry metadata can include additional programmable fields that can be used for any optional scraping of any one of the device configuration status, device counters, error conditions, etc. As an example, any data saved in one or more registers of a device that adds the above telemetry metadata to a network data unit can be included in the collected metadata. In one or more embodiments, the telemetry metadata can include queue depth, utilization, or load values, which can include contributions added by unrelated or other application network traffic, and this can be the reason for sub-optimal or below-expected performance of the application for which the telemetry metadata is being collected. In one or more embodiments, the telemetry metadata can include traffic class information. As an example, the best-effort class may be affected by a dedicated class, and packets collecting metadata sent in the best-effort class may be able to see that the link is also delivering packets in the dedicated class, and this can be the reason for higher-than-expected latency.

[0021] Continuing with the above example, in one or more embodiments, once a network data unit including the above telemetry metadata added by the NIC of a node of an application transmitter enters the network, each hop of the data flow path through the network including network devices configured to perform the embodiments disclosed herein can similarly perform a match on any L2, L3, L4, or previously inserted telemetry metadata (e.g., any data in any field in any header of the network data unit). In one or more embodiments, when a network device identifies a match, the network device can perform a corresponding action of inserting its own telemetry metadata into the network data unit, and this action includes any network telemetry information (as described above), as well as any optional configured information expected to be collected for the data flow of the application (e.g., the register status of the network device).

[0022] In one or more embodiments, any device of a hop along a data flow path can be configured as the last hop where telemetry metadata is added to a network data unit. As an example, the NIC of a node on which an application receiver executes can be the last hop for adding telemetry metadata. As another example, any network device along the data flow path before the network data unit reaches the node of the application receiver can be the last hop where telemetry metadata is added.

[0023] In one or more embodiments, the device (e.g., the NIC of a receiving node, a network device, etc.) that is the last hop where telemetry metadata is inserted into the network data unit can also be configured to strip the telemetry metadata added at each hop from the network data unit, where that hop has a device configured to perform the techniques disclosed herein. In one or more embodiments, then, the network data unit without the telemetry metadata can be provided to the application receiver. As an example, when the NIC of the node of the application receiver is the last hop, the NIC can strip the telemetry metadata and pass the network data unit up the network stack to the application receiver. As another example, when the last hop is a network device along the data flow path, the network device can strip the telemetry metadata and send the network data unit to the node of the application receiver.

[0024] In one or more embodiments, the stripped telemetry metadata can be used to generate a report that includes application information and network information with the telemetry metadata. In one or more embodiments, the report is provided to an application collector executing on the node of the application receiver. Then, the application collector can store the telemetry metadata from the report (e.g., in a memory) for any purpose (e.g., for direct real-time use when the telemetry metadata reaches the application receiver, or for indirect use during post-analysis of the telemetry metadata from the stored data).

[0025] As an example, the telemetry metadata can be used to render real-time information about application performance in a user interface for viewing by a user of the application. As another example, the telemetry metadata can be stored in a data structure in a persistent storage device (e.g., a database) for later consumption and / or analysis. As another example, the telemetry metadata can be provided to a remote collector that is configured to receive application-specific telemetry metadata and that can also be configured to perform analysis on the aggregated telemetry metadata of the application. The application-specific telemetry metadata can be used for any other purpose without departing from the scope of the embodiments disclosed herein.

[0026] Certain embodiments of the present disclosure may provide the ability to associate application - specific information with network telemetry metadata to allow application users to gain insights into the performance and / or operation of an application while using the network. Such an ability can be provided by configuring the NIC and network devices to match the information of network data units and perform operations based on the match, where the operations include adding various telemetry metadata to the network data units, including network telemetry information and any other information related to the application. An application collector executed on the same node as the intended application receiver of the network data unit can receive the telemetry metadata that is inserted into the network data unit at each configured hop and stripped from the network data unit at the last such hop.

[0027] The telemetry metadata can then be stored and / or used to obtain information about the performance of the application, as the telemetry metadata includes both the network telemetry information for each hop and the application - specific information that allows the network telemetry information to be associated with a particular application. The collection of application - specific telemetry metadata can be scalable because it is collected by application collectors executed at each node of the application rather than a centralized collector that receives network telemetry data for the entire network, regardless of the application that caused the data flow, and avoids the need for the time - consuming and resource - intensive post - processing of network telemetry data by such a collector that attempts to associate the data flow with a particular application. Additionally, the telemetry metadata can provide insights that can improve the load balancing of a dynamic routing network and can also help improve tail latency.

[0028] Figure 1 A block diagram of an example system for implementing techniques for obtaining application - specific telemetry metadata in accordance with one or more embodiments disclosed herein is shown. As Figure 1 shown, the system may include any number of nodes (e.g., Node A 100, Node B 102) and a network 108. In one or more embodiments, a node (e.g., Node A) may include an application transmitter 116 and NIC A 104. In one or more embodiments, a node (e.g., Node B) may include an application receiver 118, an application collector 120, and NIC B 106. In one or more embodiments, the network 108 may include any number of network devices (e.g., Network Device A 110, Network Device B 112, Network Device N 114). Each of these components will be described below.

[0029] In one or more embodiments, nodes (e.g., node A 100, node B 102) are computing devices. In one or more embodiments, as used herein, a computing device can be any single computing device, a collection of computing devices, a part of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. In one or more embodiments, a computing device is any device, part of a device, or collection of devices capable of electronically processing instructions and can include, but is not limited to, any of the following: one or more processors (e.g., components including circuitry) (not shown), memory (e.g., random access memory (RAM)) (not shown), input and output devices (not shown), non-volatile storage hardware (e.g., solid state drive (SSD), persistent memory (Pmem) device, hard disk drive (HDD)) (not shown), one or more physical interfaces (e.g., network ports, storage ports) (not shown), any number of other hardware components (not shown), and / or any combination thereof.

[0030] Examples of computing devices include, but are not limited to, servers (e.g., blade servers in a blade server chassis, rack servers in a rack, etc.), desktop computers, mobile devices (e.g., laptops, smartphones, personal digital assistants, tablets, automotive computing systems, and / or any other mobile computing device), storage devices (e.g., disk drive arrays, Fibre Channel storage devices, Internet Small Computer System Interface (iSCSI) storage devices, tape storage devices, flash arrays, network-attached storage devices, etc.), network devices (e.g., switches, routers, multilayer switches, etc.), virtual machines, virtual computing environments, logical containers (e.g., for one or more applications), Internet of Things (IoT) devices, node arrays of computing resources, supercomputing devices, data centers or any part thereof, and / or any other type of computing device having the above requirements.

[0031] In one or more embodiments, a node (e.g., node A 100, node B 102) can be part of a collection of any number of nodes configured to operate as a high-performance computing (HPC) environment. In one or more embodiments, an HPC environment can include any number of nodes, which can be homogeneous or heterogeneous in terms of device capabilities, and provide a platform for executing HPC applications (e.g., artificial intelligence (AI), machine learning, deep learning, autonomous driving, product design and manufacturing, weather modeling and prediction, seismic data analysis, financial risk assessment, fraud detection, computational fluid dynamics, DNA sequencing, context search algorithms, traffic management, complex simulations, drug research, virtual reality, augmented reality, etc.). In one or more embodiments, an HPC environment generally provides a platform for executing application workloads that use a large number of nodes to execute various parts of an application and thus often send data to each other over a network (discussed further below).

[0032] In one or more embodiments, any or all of the above examples can be combined to create a system of such devices, or can be divided into separate logical devices that can be referred to individually or collectively as computing devices. Other types of computing devices can be used without departing from the scope of the embodiments described herein, e.g., Figure 5 the computing devices shown and described above. The system can include any number and / or type of such computing devices (e.g., nodes) in any arrangement and / or configuration without departing from the scope of the embodiments disclosed herein.

[0033] In one or more embodiments, the storage and / or memory of a computing device or computing device system can be and / or include one or more data repositories for storing any number of data structures that store any number of data (e.g., information). In one or more embodiments, a data repository is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, RAM, a hard disk drive, a solid-state drive, and / or any other storage mechanism or medium). Additionally, a data repository can include multiple different storage units and / or devices. The multiple different storage units and / or devices can be or can not be of the same type or located in the same physical location.

[0034] In one or more embodiments, any storage and / or memory of a computing device or computing device system can be regarded, in whole or in part, as a non-transitory computer-readable medium storing software and / or firmware.

[0035] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuitry) of a computing device and / or computing device system, cause the one or more processors and / or other hardware components to perform operations in accordance with one or more embodiments described herein.

[0036] The software instructions may be in the form of computer-readable program code for performing the methods, procedures, etc. of the embodiments described herein, and as an example, may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium such as a compact disc (CD), digital versatile disc (DVD), storage device, floppy disk, magnetic tape memory, flash memory, physical memory, or any other non-transitory computer-readable medium.

[0037] The system may include any number of nodes, and as used herein, any number of these nodes may be considered, individually or collectively, as a computing device. All or any part of the computing device may be of the same type or different types of computing devices.

[0038] In one or more embodiments, nodes (e.g., node A 100, node B 102) include NICs (e.g., NIC A 104, NIC B 106). In one or more embodiments, a NIC is an input and / or output component configured to provide an interface between a node (e.g., node A 100, node B 102) and a network (e.g., network 108 described below). In one or more embodiments, a NIC (e.g., NIC A 104, NIC B 106) is used to receive and / or transmit network data units. A network data unit may include a payload within any number of headers and / or trailers (e.g., data intended to be consumed by an entity receiving the network data unit), which may be information fields intended to allow the receiving entity to perform various actions to propagate the network data unit to a destination (e.g., another device, application receiver, etc.). Such information fields may include, but are not limited to, various information items related to protocols used to implement data transmission (e.g., Media Access Control (MAC), Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Address Resolution Protocol (ARP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Virtual Extensible Local Area Network (VXLAN) protocol, Multiprotocol Label Switching (MPLS) Segment Routing (SR) protocol, etc.), addresses and / or labels related to such protocols (e.g., IP addresses, MAC addresses, label stacks, etc.), fields related to error identification and / or correction, etc. A NIC (e.g., NIC A 104, NIC B 106) may be configured with any type of interface for receiving and / or transmitting network data units, such as a wireless interface, a wired interface, etc. Although Figure 1 nodes (e.g., node A 100, node B 102) are shown including a single NIC, a computing device may include any number of SmartNICs without departing from the scope of the embodiments disclosed herein.

[0039] In one or more embodiments, a NIC (e.g., NIC A 104, NIC B 106) may be a SmartNIC. In one or more embodiments, a SmartNIC is a NIC that includes additional processing resources relative to a standard NIC. A SmartNIC may include various hardware components, subsystems, etc., configured to perform processing on received network data units to offload at least some of such processing from one or more processors of a computing device. Such hardware components may include, but are not limited to, Field Programmable Gate Arrays (FPGAs), System on a Chip (SOC) Digital Signal Processors (DSPs), etc. Such hardware components may be (or be included in) one or more subsystems of the SmartNIC (e.g., a RISC-ARM subsystem).

[0040] In one or more embodiments, a NIC (e.g., NIC A 104, NIC B 106) can be configured to identify network data units using matching action rules and perform the actions specified by the rules. In one or more embodiments, the matching can be performed using any information included in and / or associated with the network data unit received at the NIC (e.g., from the application sender that sent the network data unit, the application receiver intended for the received network data unit, etc.). In one or more embodiments, corresponding actions based on such matching can include inserting telemetry metadata into the network data unit, stripping telemetry metadata from the network data unit, etc. Further discussion of the various operations of the NIC performed in accordance with one or more embodiments disclosed herein is provided below in Figure 2 、 Figure 3 and the description of FIG. 4. Although Figure 1 a node with a single NIC is shown, a node can include any number of NICs without departing from the scope of the embodiments disclosed herein.

[0041] In one or more embodiments, a node (e.g., node A 100) can include an application sender (e.g., application sender 116). In one or more embodiments, application sender 116 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof that is at least partially configured to cause data to be sent over a network. As an example, an application can be implemented to execute a workload in an HPC environment that includes thousands of nodes, where each node executes a part of the application. In such a scenario, a part of the application executed on a given node (e.g., application sender 116 executed on node A 100) can be configured to send data to another part of the application executed on another node (e.g., application receiver 118 executed on node B 102) from time to time. Although Figure 1 a node A100 with a single application sender 116 is shown, a node can include any number of application senders without departing from the scope of the embodiments disclosed herein.

[0042] In one or more embodiments, a node (e.g., node B 102) can include an application receiver (e.g., application receiver 118). In one or more embodiments, application receiver 118 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof that is at least partially configured to receive data that has been sent over a network (e.g., network 108). As an example, application receiver 118 can be a part of an application that executes a workload in a cloud environment and receives data from other parts of the application (e.g., application sender 116) from time to time. Although Figure 1Node B 102 is shown with a single application receiver 118, but a node may include any number of application receivers without departing from the scope of the embodiments disclosed herein. Additionally, a node may include any number of application transmitters and may also include any number of application receivers without departing from the scope of the embodiments disclosed herein.

[0043] In one or more embodiments, a node (e.g., node B 102) may include an application collector 120. In one or more embodiments, the application collector 120 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof configured to receive a telemetry metadata report at least in part from a last-hop device along a data flow path of a network data unit. In one or more embodiments, as used herein, a last-hop device is the last device along the data flow path (e.g., NIC, network device, etc.) configured to perform operations in accordance with one or more embodiments disclosed herein. In one or more embodiments, a telemetry metadata report is a collection of information based on telemetry metadata stripped from a network data unit before the network data unit is provided to an application receiver (e.g., application receiver 118). In one or more embodiments, telemetry metadata includes application-specific information and network telemetry information added to the network data unit by one or more devices configured to perform operations in accordance with one or more embodiments disclosed herein. The content of the telemetry metadata report and the delivery of such a report to an application collector (e.g., application collector 120) are further discussed in the following Figure 2 , Figure 3 and the description of FIG. 4. Although Figure 1 a single node (e.g., node B 102) is shown as including a single application collector (e.g., application collector 120), any number of nodes may include any number of application collectors, application receivers, and / or application transmitters without departing from the scope of the embodiments disclosed herein.

[0044] In one or more embodiments, each node of the system (e.g., node A 100, node B 102) is operably connected to a network (e.g., network 108). As an example, the NICs of the nodes (e.g., NIC A 104, NIC B 106) can provide an operable connection to network 108. The network (e.g., network 108) can refer to the entire network or any part thereof (e.g., the logical part of network devices within the topology of network devices). Network 108 can be and / or include a data center network, a wide area network, a local area network, a wireless network, a cellular phone network, an InfiniBand network, and / or any other suitable network that facilitates the exchange of information from one part of the network to another (e.g., via the transmission of network data units). Network 108 can be a combination of any of the above network types. Network 108 can be located at a single physical location or distributed across any number of physical sites. In one or more embodiments, the network can be at least partially coupled or overlapped with the Internet.

[0045] In one or more embodiments, the network includes any number of network devices (described below), which can together implement any number of protocols (e.g., routing and forwarding protocols) and / or techniques (e.g., load balancing techniques), which contribute to decisions regarding the data flow path that a particular network data unit can traverse to reach another node (e.g., node B 102) operably connected to network 108 from one node (e.g., node A 100) operably connected to network 108). Such decisions can result in network 108 having any number of possible data flow paths that can be traversed between nodes, and the decisions made by network devices at various points in time can contribute to the data flow path being dynamic in nature (e.g., not predetermined for any given network data unit).

[0046] In one or more embodiments, network 108 can include any number of network devices (e.g., network device A 110, network device B 112, network device N 114). The three points between network device B 112 and network device N 114 are intended to illustrate that network 108 is not limited to any particular number of network devices and can thus include any number of network devices, which can be a large number of network devices to facilitate complex networks often implemented in data center environments, HPC environments, etc.

[0047] In one or more embodiments, a network device (e.g., 110, 112, 114) is a device that includes and / or is operatively connected to persistent storage (not shown), memory (e.g., random access memory (RAM)) (not shown), one or more processors (e.g., integrated circuits, ASICs, etc.) (not shown), and at least one physical network interface (not shown), which may also be referred to as a port and which may provide a connection (i.e., a link) to other devices (e.g., computing devices, other network devices, etc.).

[0048] In one or more embodiments, a network device (e.g., 110, 112, 114) also includes any number of additional components (not shown), such as network chips, FPGAs, application specific integrated circuits (ASICs), indicator lights (not shown), fans (not shown), power supply units, power distribution units, etc. At least a portion of such hardware components may be included as part of one or more line cards of the network device 100. In one or more embodiments, a line card as used herein refers to a collection of hardware components (e.g., connected via a printed circuit board) that includes one or more physical interfaces (e.g., network ports) and any number of additional hardware components (e.g., ASICs, FPGAs, TCAMs, processor components, other memory components, etc.), and the collection of these hardware components is at least partially used for storing and forwarding information and processing network traffic. Forwarding information (which may include all or any part of what may be referred to as "FIB" information) may be stored, for example, (e.g., in various tables) in one or more hardware components of the line card, such as MAC tables, routing table entries, multicast forwarding entries, etc., which are sometimes referred to as part of the data plane. In one or more embodiments, a network device includes information, such as a routing information base (RIB), which includes information (e.g., obtained from various routing protocols) that may be used to program the components of the network device to propagate network data units. Such information is sometimes referred to as being in the control plane. A network device (e.g., 110, 112, 114) may include any other components without departing from the scope of the embodiments described herein.

[0049] In one or more embodiments, as discussed above, a network device (e.g., 110, 112, 114) includes at least one physical interface (and typically two or more such physical interfaces). In one or more embodiments, a physical interface is any hardware, software, or combination thereof that includes the functionality to receive and / or transmit network data units (e.g., packets, frames, etc.) or any other information from / to the network device (e.g., 110, 112, 114). A physical interface may include any interface technology, such as, for example, optical, electrical, etc. A physical interface may be configured to interface with any transmission medium (e.g., optical fiber, copper wire, etc.).

[0050] In one or more embodiments, a physical interface includes and / or is operatively connected to any number of components used in network traffic processing. For example, a given physical interface may include a PHY (not shown), which is circuitry that connects a physical information propagation medium (e.g., a wire) to other components (e.g., hardware components of a line card) that process network traffic. In one or more embodiments, the physical interface includes and / or is operatively connected to a transceiver that provides a connection between the physical information transmission medium and the PHY. The PHY may also include any number of other components, such as, for example, a serializer / deserializer (SERDES) and a numberer / decoder, etc. The PHY may in turn be operatively connected to any number of other components, such as, for example, a media access control (MAC) sublayer. Such a sublayer may in turn be operatively connected to other higher layer processing components, all of which form a series of components that are used by a network device (e.g., 110, 112, 114) for receiving, sending, or otherwise processing network traffic for any purpose.

[0051] In one or more embodiments, a network device (e.g., 110, 112, 114) includes any software (e.g., various daemons, state databases, etc.) that is configured to perform and / or allow other components to perform the various functions of the network device (e.g., process network traffic). For example, such software may be executed using one or more processors of the network device or any other hardware resources of the network device capable of executing software.

[0052] Examples of network devices (e.g., 110, 112, 114) include but are not limited to switches, routers, multilayer switches, Fibre Channel devices, devices, etc. Network devices are not limited to the above specific examples.

[0053] In one or more embodiments, a network device (e.g., 110, 112, 114) includes a function for receiving network data units (e.g., frames, packets, tunnel protocol frames, etc.) at any physical interface (e.g., port) of the network device (e.g., 110, 112, 114) and processing the network data units. In one or more embodiments, processing network traffic includes but is not limited to a series of one or more lookups (e.g., longest prefix match (LPM) lookup, forwarding equivalence class (FEC) lookup, etc.) and corresponding actions (e.g., forwarding from a certain egress port, adding a label protocol header, rewriting a destination address, encapsulation, etc.). Examples of network traffic processing include but are not limited to determining: (i) whether to take security measures (e.g., discard network traffic data units); (ii) whether to mirror network traffic data units; and / or (iii) how to route / forward network traffic data units in order to send the network traffic data units from an interface of the network device.

[0054] In one or more embodiments, a network device (e.g., 110, 112, 114) is configured with any number of match-action rules. In one or more embodiments, a match-action rule causes the network device (e.g., 110, 112, 114) to perform a match on any information included in a network data unit (e.g., any number of fields in any number of headers, any telemetry metadata previously added to the network data unit by other devices, etc.), and to perform one or more corresponding actions (e.g., add telemetry metadata) upon successful match (e.g., one or more fields in the network data unit match information that the network device is configured to include and that is associated with the match-action rule). The use of match-action rules according to one or more embodiments disclosed herein is further discussed below in Figure 2 , Figure 3 and the description of FIG. 4.

[0055] Although Figure 1 a particular configuration of components is shown, other configurations may also be used without departing from the scope of the embodiments described herein. For example, although Figure 1 certain components are shown as part of the same device, any component can be grouped into a set of one or more components, and any of these components can exist and operate as part of any number of separate and operably connected devices. As another example, a single component can be configured to perform all or any part of the functions performed by the Figure 1 shown components. Accordingly, the embodiments disclosed herein should not be limited to the Figure 1 shown configuration of components.

[0056] Figure 2 shows an overview of an example method for obtaining application-specific telemetry metadata according to one or more embodiments disclosed herein. Figure 2 All or any part of the method shown can be performed by, for example, a device or set of devices (e.g., Figure 1 NIC A 104, NIC B 106, network device A 110, network device B 112, network device N 114) configured to perform operations according to one or more embodiments disclosed herein.

[0057] Although Figure 2 the individual steps in the flowchart shown are presented and described in sequence, some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, Figure 2 other steps not shown in Figure 2 may be additionally performed, and / or some or all of the steps may be performed in parallel with other steps in

[0058] In step 200, the method includes the following steps: Figure 1 The first node (eg, Figure 1 NIC of node A 100) (e.g., Figure 1 NIC A 104 of the present invention identifies a network data unit (NDU) for telemetry metadata insertion. In one or more embodiments, identification of the NDU occurs by the NIC executing a match action rule configured for the NIC. In one or more embodiments, executing the match action rule includes performing a match operation on any information included in and / or associated with the NDU. Such information may include, but is not limited to, any one or more portions of any header or field of the NDU, such as L2, L3, and / or L4 fields, which may include, but are not limited to, source and destination IP addresses, source and destination MAC addresses, source and destination port numbers (e.g., TCP and / or UDP port numbers), VLAN tags, VNIs, flow labels, differentiated services code point (DSCP) values, application identifiers, flow identifiers, flow labels, and protocol information. As an example, a NDU sent from an application sender may be provided to a NIC of a node on which the application sender is executing, so that the NIC may send the NDU into the network (e.g., the NDU enters the network). The NIC may perform the match using all or any portion of the information received in association with the NDU to be sent. In one or more embodiments, in a scenario where the NIC of the node on which the application sender is executed is not configured with such a match action rule (e.g., performing operations according to one or more embodiments disclosed herein), the first match operation can be alternately performed by the first device (e.g., a network device) along the data flow path of the network data unit so configured.

[0059] In step 202, the method includes inserting initial telemetry metadata into the network data unit based on the success of the first matching operation. As an example, the initial telemetry metadata can be inserted into the network data unit by the NIC that provides the network data unit from the application sender. In one or more embodiments, the initial telemetry metadata includes both network information (e.g., network telemetry information and / or any other network information related to and / or associated with the device inserting the telemetry metadata) and application information (e.g., any information specific to the application of which the application sender is a part). In one or more embodiments, the initial telemetry metadata can be considered to include a metadata header that indicates the start of the telemetry metadata within the network data unit.

[0060] Initial telemetry metadata can include, but is not limited to: information identifying the application; information identifying a part of the application that caused the data to be sent (e.g., process, application phase, application level, application function, application operation, etc.); device information (e.g., device identifier of the NIC); ingress port and / or egress port of the network data unit; latency information, which can include the time when the network data unit arrives at the NIC, the time spent processing the network data unit before sending; queuing arbitration parameters such as (e.g., the time the network data unit spends in the buffer); ingress timestamp and egress timestamp, which can be high-resolution timestamps (e.g., at the nanosecond level) obtained using techniques such as the Precision Time Protocol (PTP); header transformation information, processing and / or transformation time; link utilization information; link load and congestion indicators; various queue and buffer states; network data unit pipelining operations; network data unit transformation; Virtual Routing and Forwarding (VRF) information; flow information (e.g., is the network data unit starting a new flow or part of an ongoing flow?); changing the size of the network data unit; information about the routing and forwarding decisions made; any other type of data included in the optional scrape field (e.g., status of device registers, statistics, counters, code indicating something to the user about the application, any kind of free text, etc.), information about the reason the network data unit can be blocked for a period of time, measurement information about the type and / or degree of congestion, and / or any combination thereof. Any other relevant information can be included in the initial telemetry metadata without departing from the scope of the embodiments disclosed herein. In one or more embodiments, a part of the telemetry metadata can be standard telemetry information based on a network monitoring framework (e.g., In-band Network Telemetry (INT)), while another part can be information specific to the embodiments disclosed herein, which is specific to the application sending the network data unit, and this information can change over time as the user of the application fine-tunes the information needed to evaluate the performance of the application.

[0061] In one or more embodiments, inserting the initial telemetry metadata into the network data unit can include inserting the telemetry metadata into a predefined specific location within the network data unit. As an example, the NIC can insert the initial telemetry metadata between the first part of the header of the network data unit (e.g., the header related to the application, presentation, session, and / or transport layer) and other headers used to allow the network data unit to traverse network devices (e.g., network, data link, and physical layer). For example, the initial telemetry metadata can be inserted into the network data unit below the Ethernet header (e.g., including MAC address information) and the IP header (e.g., including IP address information) and above the remaining part of the network data unit (e.g., the L4 payload). The initial telemetry metadata can be inserted into any other location within the network data unit without departing from the scope of the embodiments disclosed herein.

[0062] In one or more embodiments, including both application - specific information and network information in the initial telemetry metadata allows the collection of telemetry metadata for network data units to be associated with a specific application, such that network data units can be sent without extensive post - processing of the network data units by collectors of information not specific to that application, and can allow devices along the data flow path to identify network data units as related to a specific application and / or parts thereof, in order to customize the telemetry metadata added to the network data units for the specific application.

[0063] In one or more embodiments, a matching action rule (e.g., configured on the NIC of a node executing an application sender or any other device on the data flow path) can include that all or any part of a network data unit for which the matching operation is successful has inserted telemetry metadata. As an example, in some cases, when an application sends data over a network, it can be important to insert telemetry metadata into each such network data unit to gain insights into the application's performance. Another example is that the matching action rule can specify inserting telemetry metadata into only some network data units (e.g., so as not to adversely affect network data unit transmission performance), such as inserting network telemetry metadata into only a portion of the matching network data units (e.g., one out of every ten units), or not inserting telemetry metadata into network data units below a threshold size (e.g., such that the size of the network data unit does not increase significantly due to the insertion). In one or more embodiments, the matching action rule can specify that telemetry metadata is inserted only for certain traffic classes (which can be indicated by one or more fields in the network data unit).

[0064] In step 204, the method includes the NIC sending the network data unit to a network device (e.g., Figure 1 one of network devices 110, 112, or 114). In one or more embodiments, sending the network data unit includes preparing the network data unit for transmission (e.g., performing any transformation of the network data unit, such as adding and / or modifying header information identifying the next hop), and sending the network data unit as a stream of bits over the transmission medium.

[0065] In step 206, the method includes performing a matching operation by a network device along the data flow path of the network data unit using any information included in the network data unit. In one or more embodiments, a network device similar to the above-described NIC may be configured with one or more matching action rules and may perform a matching operation according to such rules to determine whether the network data unit successfully matches the information that the network device is configured to perform a comparison against. The network device that performs such a matching action rule may be the first network device on the data flow path configured to perform operations according to the embodiments disclosed herein, which may be the first network device that the network data unit reaches after leaving the node of the application sender, or may be any subsequent device along the data flow path. In one or more embodiments, the execution of the matching action rule may be similar to the execution discussed above with respect to the NIC of the node of the application sender. Thus, in one or more embodiments, the network device may match any information in any field of any header of the network data unit, and any telemetry metadata previously added to the network data unit according to the embodiments disclosed herein, or any combination of such information.

[0066] In step 208, the method includes inserting additional telemetry metadata into the network data unit based on a successful matching operation performed in step 206. In one or more embodiments, the additional telemetry metadata may include all or any part of the type of information inserted as initial telemetry metadata (discussed above), but specific to the network device that performs the insertion of the additional telemetry metadata. As an example, the additional telemetry metadata may include any standard in-band network telemetry data (e.g., ingress interface / egress interface, latency information, routing and forwarding decision information, buffer and queue depth, etc.), and any optional information specific to the network device when receiving, processing, and / or sending the network data unit (e.g., device register status, counter information, etc.). In one or more embodiments, the set of information included in the additional telemetry metadata is configured via the matching action rules implemented on the network device.

[0067] In one or more embodiments, the additional telemetry metadata is inserted into the network data unit in a location similar to the location of the initial telemetry metadata and / or any other telemetry metadata added by other network devices before the network data unit reaches the network device. As an example, the network device may insert the additional telemetry metadata after the initial telemetry metadata in the network data unit, where "after" refers to the location of the telemetry metadata in the telemetry metadata stack constructed when each hop adds its telemetry metadata to the network data unit.

[0068] In one or more embodiments, any number of network devices may be present along the data flow path of a network data unit, and each device configured with a match action rule that causes telemetry metadata to be added based on a successful match, as disclosed herein, may similarly insert additional telemetry metadata associated with the network device that inserts the telemetry metadata, which may result in a per-device stack of telemetry metadata within the network data unit. As an example, starting with a telemetry header and telemetry metadata that are part of the initial telemetry metadata, a first device (such as a NIC on the node where an application sender executes) may insert the initial telemetry data into a specific location within the network data unit (e.g., between the L3 header and the L4 header). The telemetry header in the initial telemetry metadata may signal the start of the telemetry metadata for the network data unit. At each subsequent hop where telemetry metadata is added here, the additional telemetry metadata may be appended to the previous telemetry metadata added by one or more previous hops. Thus, each unit of telemetry metadata added by a hop along the data path may be a discrete unit of telemetry metadata that is added in sequence after the telemetry header added by the first device as part of the initial telemetry data to the network data unit.

[0069] In step 210, the method includes a network device sending a network data unit (which includes initial telemetry metadata and additional telemetry metadata added by any number of other network devices) to a last-hop device along the data flow path of the network data unit. In one or more embodiments, any device of the hops along the data flow path of the network data unit may be configured as the last hop where telemetry metadata is added to the network data unit. As an example, a NIC of the node on which an application receiver executes may be the last hop for adding telemetry metadata. As another example, any network device along the data flow path before the network data unit reaches the node of the application receiver may be the last hop where telemetry metadata is added here. In one or more embodiments, although not shown in Figure 2 the last-hop device also adds additional telemetry metadata to the network data unit, similar to the insertion of the additional telemetry metadata discussed above in the description of step 208 (e.g., performing a match operation and, based on a successful match, performing an action to insert last-hop device-specific additional telemetry metadata).

[0070] In step 212, the method includes the last-hop device stripping the initial telemetry metadata and all additional telemetry metadata from the network data unit. In one or more embodiments, although not shown in Figure 2Although not shown in [the figure], network data units without stripped telemetry metadata can be provided to an application receiver that is intended to be the network data unit. As an example, when the NIC of a node of the application receiver is the last hop, the NIC can strip the telemetry metadata and pass the network data unit up the network stack to the application receiver. As another example, when the last hop is a network device along the data flow path, the network device can strip the telemetry metadata and send the network data unit to the node of the application receiver.

[0071] In step 214, the method includes generating, by the last-hop device, a telemetry metadata report that includes the initial telemetry metadata and all additional telemetry metadata. The telemetry metadata report can be in any suitable form (e.g., raw data formatted in a particular way, etc.). Thus, the telemetry metadata report includes, but is not limited to, application-specific information (e.g., information identifying the application that sent the network data unit or any part thereof), and network telemetry information from each hop along the data flow path where telemetry metadata was added to the network data unit. A wide variety of examples of the types of information that can be included in the telemetry metadata report (e.g., initial telemetry metadata and additional telemetry metadata) can be found, for example, in the descriptions of steps 202 and 208 above.

[0072] In step 216, the method includes providing, by the last-hop device, the telemetry metadata report to the application receiver where the network data unit is to be executed (e.g., Figure 1 application receiver 118) at the same node (e.g., Figure 1The application collector (e.g., the application collector 120 of 1) that executes on the node B 102). The application collector can then store the telemetry metadata from the reports (e.g., in memory) for any purpose. As an example, the telemetry metadata can be used to render real-time information about application performance in a user interface for the user of the application to view. As another example, the telemetry metadata can be stored in a data structure in a persistent storage device (e.g., a database) for later consumption and / or analysis. As another example, the telemetry metadata can be provided to a remote collector that is configured to receive application-specific telemetry metadata and that can also be configured to perform analysis on the aggregated telemetry metadata of the application. The application-specific telemetry metadata can be used for any other purpose without departing from the scope of the embodiments disclosed herein. In one or more embodiments, as discussed above and below, providing the telemetry metadata reports to an application collector that executes on a node where the application receiver executes rather than to a centralized collector can allow the collection of telemetry metadata associated with a particular application to be scalable because it is collected by an application collector that executes at each node of the application rather than a centralized collector that receives network telemetry data for the entire network, regardless of the application that caused the data flow, and avoids the need for such a collector that attempts to associate the data flow with a particular application to perform time-consuming and resource-intensive post-processing of the network telemetry data.

[0073] Figure 3 Shows an example system for obtaining application-specific telemetry metadata in accordance with one or more embodiments disclosed herein. Figure 4A Shows an example network data unit without telemetry metadata in accordance with one or more embodiments disclosed herein. Figure 4B Shows an example network data unit with telemetry metadata inserted in accordance with one or more embodiments disclosed herein. Figure 3 The examples shown and described below are only simplified examples for illustrative purposes and are not intended to limit the scope of the embodiments described herein. Further, although the examples show certain aspects of the embodiments described herein, not all possible aspects of such embodiments may be shown in that particular example.

[0074] Consider a scenario where an HPC application executes on a thousand nodes, two of which are Node A 300 and Node B 302. An application transmitter 322 executing on Node A 300 is configured to send network data units to an application receiver 324 executing on Node B 302. The user of the HPC application has noticed that the performance of the application is occasionally below expectations and suspects that the problem may be related to the network data units sent from the application transmitter of the application to the application receiver of the application. Thus, in accordance with embodiments disclosed herein, the user configures the NIC 304 of Node A 300, the NIC B 306 of Node B 302, and each network device of the network between the two nodes (e.g., 308, 310, 312, 316, 318, and 320) with various matching action rules that allow the data flow through the NICs and network devices to be associated with the application and its various parts (e.g., different phases of the application, different application functions, etc.).

[0075] In such a scenario, the application transmitter 322 may generate a network data unit that may include, for example, a data payload, information from various upper layer protocols (e.g., the sending layer), the destination IP address of Node B 302 where the application receiver 324 executes, etc. The network data unit may be provided to NIC A 304. In the absence of the embodiments disclosed herein, the NIC may add additional headers (e.g., an IP header, a MAC header) for sending the network data unit to the application receiver. An example of a network data unit prepared in this manner can be seen in Figure 4A where the network data unit 400 includes an upper layer payload 402 (including the data payload and any upper layer headers), an IP header 404, and a MAC header 406.

[0076] However, in this case, NIC A 304 is configured with matching action rules related to the application of the application transmitter 322. Thus, NIC A 304 performs a matching operation that identifies that the network data unit is from the application transmitter 322 of a specific phase of the executing application to send a specific category of network data units to a specific TCP port (e.g., at least four fields of the network data unit match). Based on a successful match of this information, NIC A304 identifies that initial telemetry metadata should be inserted into the network data unit. Thus, NIC A304 inserts the initial telemetry metadata between the upper layer payload 402 and the IP header 404 of the network data unit 400 and sends the network data unit to network device A 308.

[0077] Network device A 308 is also configured with an application-specific matching action rule and similarly performs a successful matching operation, inserts additional telemetry metadata A into the network data unit, and forwards the network data unit to the next hop along the data path based on the routing and forwarding configuration of network device A 308. Network device C 312 also performs a successful pairing operation and inserts additional telemetry metadata B into the network data unit. A similar process occurs along the data flow path through network device D 316 (which inserts additional telemetry metadata C into the network data unit) and network device F 320 (which inserts additional telemetry metadata D into the network data unit).

[0078] Next, the network data unit is sent to the NIC B 306 of node B 302 where the application receiver 324 is executed. NIC B is the last hop along the data flow path and also performs a matching action rule to perform a matching operation and inserts additional telemetry metadata into the network data unit. As shown Figure 4B in the resulting network data unit 400 with the initial telemetry metadata and various additional telemetry metadata. As Figure 4B shown, at this time, the network data unit 400 includes the initial telemetry metadata 408 added by NIC A 304, the additional telemetry metadata A 410 added by network device A 308, the additional telemetry metadata B 412 added by network device C 312, the additional telemetry metadata C 414 added by network device D 316, and the additional telemetry metadata E 418 added by NIC B 306 between the upper layer payloads.

[0079] As the last-hop device, NIC B 306 strips the initial telemetry metadata 408 and all additional telemetry metadata (410, 412, 414, 416, and 418) from the network data unit to obtain the network data unit 400 as shown Figure 4A in. The network data unit 400 can then be processed normally Figure 4A to provide the network data unit 400 to the application receiver 324 of node B 302.

[0080] As a last-hop device, NIC B is also configured to generate a telemetry metadata report using the initial telemetry metadata and all additional telemetry metadata, and provide the telemetry metadata report to the application collector 326 that also executes on node B 302. Thus, network information (e.g., network telemetry data) from each hop along the data flow path, as well as application-specific information, is provided to the application collector 326, which can store or use this information for any purpose, such as allowing the user of the application to evaluate possible reasons why problems seen by a network data unit as it traverses the data flow path through the network may result in application performance being lower than expected. Since the telemetry metadata includes both network information and application-specific information, this information is already relevant to the application. Additionally, the telemetry metadata for such network data units sent from the application sender to the application receiver is reconciled at the application collector to allow such analysis to scale with the size of the application (e.g., the number of nodes), thus avoiding bottlenecks and post-processing issues that occur when network telemetry data not specifically associated with the application is aggregated and sent to a remote collector configured to receive all such data for the network.

[0081] Figure 5 A block diagram of a computing device is shown in accordance with one or more embodiments of the present disclosure. As discussed above, the embodiments described herein may be implemented using a computing device. For example, Figure 1 All or any part of the components shown may be implemented, at least in part, using one or more computing devices, and Figure 2 all or any part of the methods shown may be performed using one or more computing devices such as computing device 500. Computing device 500 may include one or more computer processors 502, non-persistent storage 504 (e.g., volatile memory such as random access memory (RAM), cache memory, etc.), persistent storage 506 (e.g., hard disk, optical drive such as a compact disc (CD) drive or digital versatile disc (DVD) drive, flash memory, etc.), communication interface 512 (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device 510, output device 508, and many other elements (not shown) and functions. Each of these components will be described below.

[0082] In one or more embodiments, the computer processor(s) 502 may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or microcores of a processor. Processor 502 may be a general-purpose processor configured to execute program code included in software executed on computing device 500. Processor 502 may be a special-purpose processor in which certain instructions are incorporated into the processor design. Processor 502 may be an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a data processing unit (DPU), a tensor processing unit (TPU), an accelerated processing unit (APU), a vision processing unit (VPU), a quantum processing unit (QPU), and / or various other processing units using dedicated hardware (e.g., field-programmable gate array (FPGA), system-on-chip (SOC), digital signal processor (DSP), etc.). Although only one processor 502 is shown in Figure 5 , computing device 500 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0083] Computing device 500 may also include one or more input devices 510, such as a touch screen, keyboard, mouse, microphone, touchpad, stylus, motion sensor, or any other type of input device. Input device 510 may allow a user to interact with computing device 500. In one or more embodiments, computing device 500 may include one or more output devices 508, such as a screen (e.g., liquid crystal display (LCD), plasma display, touch screen, cathode ray tube (CRT) monitor, projector, or other display device), printer, external storage, or any other output device. One or more of the output devices may be the same as or different from the input device(s). The input and output device(s) may be locally or remotely connected to the computer processor(s) 502, non-persistent storage 504, and persistent storage 506. There are many different types of computing devices, and the input and output device(s) described above may take other forms. In some instances, a multimodal system may allow a user to provide multiple types of input / output to communicate with computing device 500.

[0084] In addition, communication interface 512 may facilitate connecting computing device 500 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device such as another computing device. Communication interface 512 may use a wired and / or wireless transceiver of any type and / or technology to perform or facilitate the reception and / or transmission of wired or wireless communication. Examples include but are not limited to those using an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Port / plug, Ethernet port / plug, fiber optic port / plug, proprietary wired port / plug, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, and Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 512 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing device 500 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS includes, but is not limited to, US-based GPS, Russia-based Global Navigation Satellite (GLONASS), China-based BeiDou Navigation Satellite System (BDS), and Europe-based Galileo GNSS. There is no limitation on the operation of any particular hardware arrangement, and thus the basic features here can easily be replaced by improved hardware or firmware arrangements developed.

[0085] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. The computer-readable medium may include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as CDs or DVDs, flash memory, memory, or storage devices. The computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a process, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0086] All or any part of the components of computing device 500 may be implemented in circuitry. For example, the components may include electronic circuits or other electronic hardware (including one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits)) and / or may be implemented using the same, and / or may include computer software, firmware, or any combination thereof and / or may be implemented using the same to perform the various operations described herein. In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals that include bitstreams and the like. However, when referred to, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0087] In the foregoing description, many details are set forth as examples of the embodiments described herein. Those skilled in the art (who also have the benefit of this disclosure) will understand that one or more of the embodiments described herein may be practiced without these specific details and that there may be many variations or modifications without departing from the scope of the embodiments described herein. Certain details known to those of ordinary skill in the art may be omitted to avoid obscuring the description.

[0088] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, those of ordinary skill in the art will understand that these aspects may be practiced without these specific details. For clarity, in some instances, the technology may be presented as including functional blocks that may include devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. In addition to the components shown and / or described herein, additional components may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the aspects with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the aspects of the embodiments disclosed herein.

[0089] The various aspects may be described above as a process or method, which is depicted as a flow chart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flow chart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but there may be additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0090] The processes and methods according to the above examples can be implemented using computer-executable instructions stored in or otherwise retrievable from a computer-readable medium. For example, such instructions can include instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used can be accessible via a network. The computer-executable instructions can be, for example, binary, intermediate format instructions (such as assembly language, firmware, source code, etc.). Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the examples include magnetic or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0091] In the above description of the drawings, in the various embodiments described herein, any component described with respect to one drawing can be equivalent to one or more components of the same or similar name and / or number described with respect to any other drawing. For the sake of brevity, the description of these components may not be repeated for each drawing. Accordingly, each embodiment of the components of each drawing is incorporated by reference and is assumed to optionally exist in each other drawing having one or more components of the same or similar name and / or number. Additionally, in accordance with the various embodiments described herein, any description of components in the figures should be interpreted as an optional embodiment, which may be implemented as a supplement to, in combination with, or in place of the embodiments described with respect to one or more components of the same or similar name and / or number in any other figure.

[0092] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in the application). The use of ordinal numbers is not intended to imply or create any particular order of the elements, nor to limit any element to only a single element, unless explicitly disclosed, such as by the use of terms like "before," "after," "single," and other such terms. Instead, the use of ordinal numbers is for the purpose of differentiating between elements. By way of example, a first element is different from a second element, and the first element can include more than one element and can come after (or before) the second element in the order of the elements.

[0093] As used herein, the phrase "operably connected" or "operably connect" means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact in some manner. For example, the phrase "operably connected" can refer to any direct (e.g., directly wired between two devices or components) or indirect (e.g., a wired and / or wireless connection between any number of devices or components that operably connect the connected devices) connection. Thus, any path through which information can pass can be considered an operable connection.

[0094] Although the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art who benefit from this disclosure will understand that other embodiments can be envisioned without departing from the scope of the embodiments disclosed herein. Accordingly, the scope of the embodiments described herein should be limited only by the appended claims.

Claims

1. A computer-implemented method, comprising: identifying, by a network interface card (NIC) of a first node on which an application sender of an application executes, a network data unit for telemetry metadata insertion based on performing a first matching operation on information associated with the network data unit, wherein the network data unit is sent from the application sender of the first node to an application receiver of the application executing on a second node; inserting, by the NIC, initial telemetry metadata into the network data unit based on a first success of the first matching operation, the initial telemetry metadata including network information and application-specific information corresponding to the application; performing, by a network device of a hop along a data flow path of the network data unit, a second matching operation using at least one of the information associated with the network data unit and the initial telemetry metadata; inserting, by the network device, additional telemetry metadata into the network data unit based on a second success of the second matching operation; stripping, by a last-hop device of a last hop along the data flow path, the initial telemetry metadata and the additional telemetry metadata from the network data unit; generating, by the last-hop device, a telemetry metadata report using the initial telemetry metadata and the additional telemetry metadata stripped from the network data unit; and providing the telemetry metadata report to an application collector executing on the second node.

2. The computer-implemented method according to claim 1, wherein the last-hop device includes a second NIC of the second node.

3. The computer-implemented method according to claim 1, wherein the last-hop device includes a second network device along the data flow path.

4. The computer-implemented method according to claim 1, wherein the application-specific information identifies the application.

5. The computer-implemented method according to claim 1, wherein the application-specific information identifies a part of the application.

6. The computer-implemented method according to claim 1, further comprising: sending, by the last-hop device, the stripped network data unit along the data flow path toward the application receiver.

7. The computer-implemented method according to claim 1, wherein: identifying the network data unit is performed as part of executing a match-action rule, and the match-action rule specifies that only a part of the network data units having a successful first matching operation are inserted with corresponding initial telemetry metadata.

8. The computer-implemented method according to claim 1, wherein: identifying the network data unit is performed as part of executing a match-action rule, and the match-action rule specifies that any network data unit smaller than a size threshold is not inserted with the initial telemetry metadata.

9. A non-transitory computer-readable medium storing a program, the program being for execution by one or more processors, the program including instructions for: The network interface card (NIC) of the first node on which the application transmitter of the application executes identifies a network data unit for telemetry metadata insertion based on performing a first matching operation on information associated with the network data unit, where the network data unit is sent from the application transmitter of the first node to the application receiver of the application executing on a second node; Based on the first success of the first matching operation, the NIC inserts initial telemetry metadata into the network data unit, the initial telemetry metadata including network information and application-specific information corresponding to the application; The last-hop device of the last hop along the data flow path strips the initial telemetry metadata from the network data unit; The last-hop device uses the initial telemetry metadata stripped from the network data unit to generate a telemetry metadata report; And The telemetry metadata report is provided to an application collector executing on the second node.

10. The non-transitory computer-readable medium according to claim 9, wherein the last-hop device includes a second NIC of the second node.

11. The non-transitory computer-readable medium according to claim 9, wherein the last-hop device includes a second network device along the data flow path.

12. The non-transitory computer-readable medium according to claim 9, wherein: The program includes additional instructions for: A network device along the hop of the data flow path of the network data unit performs a second matching operation using at least one of the information associated with the network data unit and the initial telemetry metadata; And Based on the second success of the second matching operation, the network device inserts additional telemetry metadata into the network data unit, The last-hop device also strips the additional telemetry metadata from the network data unit, The additional telemetry metadata is also used to generate the telemetry metadata report.

13. The non-transitory computer-readable medium according to claim 9, wherein the application-specific information identifies a part of the application.

14. The non-transitory computer-readable medium according to claim 9, further comprising: The last-hop device sends the stripped network data unit along the data flow path towards the application receiver.

15. The non-transitory computer-readable medium according to claim 9, wherein: Identifying that the network data unit is executed as part of performing a matching action rule, and The matching action rule specifies that only a part of the network data units with a successful first matching operation are inserted with corresponding initial telemetry metadata.

16. The non-transitory computer-readable medium according to claim 9, wherein: Identifying that the network data unit is executed as part of performing a matching action rule, and The matching action rule specifies that any network data unit smaller than a size threshold is not inserted with the initial telemetry metadata.

17. A network device, comprising: One or more processors; And One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: Receive a network data unit that includes initial telemetry metadata inserted into the network data unit by a NIC of a first node on which an application transmitter of an application executes, where: The initial telemetry metadata includes network information and application-specific information corresponding to the application, The network device is a device along a data flow path of the network data unit, and The network data unit is intended for an application receiver executing on a second node; Perform a matching operation using at least one of information associated with the network data unit and the initial telemetry metadata; Based on a second success of the matching operation, insert additional telemetry metadata into the network data unit; and Send the network data unit along the data flow path of the network data unit to a last-hop device.

18. A system comprising: The network device according to claim 17; The NIC of the first node; And The last-hop device, Wherein the last-hop device is configured to: Strip the initial telemetry metadata and the additional telemetry metadata from the network data unit; Use the initial telemetry metadata and the additional telemetry metadata stripped from the network data unit to generate a telemetry metadata report; And Provide the telemetry metadata report to an application collector executing on the second node.

19. The system according to claim 18, wherein the last-hop device is one of: a second NIC on the second node, or a second network device along the data flow path.

20. The network device according to claim 17, wherein the application-specific information identifies one of: the application, or a part of the application.

Images