Application of specific network telemetry and diagnostics

Abstract

Embodiments of this disclosure relate to application-specific network telemetry and diagnostics. In some embodiments, a method includes a NIC of a node executed by an application transmitter of an application to identify a network data unit for telemetry metadata insertion based on information associated with the network data unit, wherein the network data unit is sent from the application transmitter to an application receiver executed on another node; inserting telemetry metadata into the network data unit based on a first success of a first matching operation, the telemetry metadata including network information and application-specific information; stripping the telemetry metadata from the network data unit by a last-hop device along the data flow path; generating a telemetry metadata report by the last-hop device using the telemetry metadata; and providing the telemetry metadata report to an application collector executed on a second node.

Description

Background Technology

[0001] Applications typically cause data to be sent over a network. Therefore, application performance can be correlated with the performance of the network through which the data traverses. However, there is often a disconnect between network telemetry data and the specific application that caused the data to be sent and / or received, making it difficult to correlate application behavior with network operations. Attached Figure Description

[0002] Some embodiments discussed herein will be described with reference to the accompanying drawings listed below. However, the drawings are merely illustrative of certain aspects or implementations of the embodiments described herein and are not intended to limit the scope of the claims.

[0003] Figure 1 A block diagram is shown for implementing an example system for acquiring application-specific telemetry metadata according to one or more embodiments disclosed herein;

[0004] Figure 2 An overview of example methods for obtaining application-specific telemetry metadata according to one or more embodiments disclosed herein is shown;

[0005] Figure 3 An example system for obtaining application-specific telemetry metadata is shown, according to one or more embodiments disclosed herein;

[0006] Figure 4A An example network data unit without telemetry metadata is shown according to one or more embodiments disclosed herein;

[0007] Figure 4B Example network data units with telemetry metadata inserted according to one or more embodiments disclosed herein are shown; and

[0008] Figure 5 A block diagram of a computing device according to one or more embodiments of the present disclosure is shown. Detailed Implementation

[0009] For applications running on two or more nodes (e.g., computing devices), application data is typically transmitted from one part of the application (e.g., the application transmitter) to another part (e.g., the application receiver) in the form of network data units (e.g., packets). Such applications can include an ever-increasing scale (e.g., tens of thousands of nodes). However, as network data units traverse the network from the application transmitter to the application receiver, associating network data units from a specific application with network telemetry information can be challenging because the application instrument is unaware (and typically cannot) of the network devices the network data units are traversing, and conversely, the network devices typically do not know which specific applications are sending and / or receiving data.

[0010] Network telemetry data collected for traversing network data units (e.g., traversed network devices, ingress and egress ports, ingress and egress timestamps, queue / buffer depth, routing and forwarding decision information, local and / or aggregation latency, etc.) is typically sent to collector devices that collect such information for the network. However, associating such aggregated network telemetry data with the specific applications sending and receiving data can be challenging and requires significant post-processing overhead to deduce which applications contribute to which network data streams. Furthermore, the aggregation of network telemetry data may encounter other problems, including but not limited to: aliasing (where timestamps are related to the sampling time rather than the actual event time); the fact that sampling further increases the challenge of associating network telemetry data with applications because application information may not correspond to the sampling time used; and the need for security measures to help ensure that information about a given application is not shared with users of other applications. Moreover, as applications grow to a larger number of nodes, these problems may be exacerbated because centralized collectors may be receiving data from a large number of nodes for each application and from any number of applications that can use large-scale networks.

[0011] Such characteristics can make it difficult to clearly understand what an application is actually sending and receiving data via a shared data path (e.g., a series of dynamically selected hops along the data flow path to the destination). In one or more embodiments, the phrase "shared data path" refers to a path that data can traverse within the network, which can be used by any number of entities (e.g., applications) that can send and / or receive data via the network, making the data path "shared". In one or more embodiments, the data path of a given network data unit across the network can be dynamically determined (e.g., at each hop), and therefore, the data path can be different each time an entity sends a network data unit to the same other entity.

[0012] To address at least part of the aforementioned problems, the embodiments disclosed herein provide a technique that allows application-specific data flows and network telemetry to be correlated in real time as network data units traverse the network path from an application transmitter (e.g., executed on a first node or transmitter node of the application) to an application receiver (e.g., executed on a second node or receiver node of the application). Telemetry metadata (including information about the application and network telemetry information) can be inserted into network data units between standard packet headers (e.g., between a Layer 4 (L4) header and a Layer 3 (L3) header). Such telemetry metadata can be generated by any device (e.g., a network interface card (NIC), network device (e.g., a switch, router, etc.)) along the data flow path of the network data unit as it traverses the network from the application transmitter to the application receiver.

[0013] In one or more embodiments, telemetry metadata is removed from the network data unit before it is delivered to the application receiver. In one or more embodiments, each hop of telemetry metadata stripped from the network data unit is provided to an application collector that runs on the same node as the application receiver. Thus, in one or more embodiments, the application collector receives a report that includes various network telemetry information and application information associated with the data flow to which the network data unit is part. Therefore, network telemetry data can be directly correlated with the applications sending and receiving the network data unit without requiring challenging post-processing by a central collector to attempt such correlation, and aliasing issues (e.g., sFlow telemetry) are avoided when performing network data unit sampling. Furthermore, because the telemetry metadata report is provided to the application collector on the receiving node of the network data unit, the telemetry collection process can scale accordingly with the size of the application (e.g., the number of nodes).

[0014] In one or more embodiments, telemetry metadata is added to the network data unit at each hop of a data flow path (e.g., via a network from an application transmitter to an application receiver), the data flow path including devices configured to perform the techniques disclosed herein. As used herein, a hop can be any device or component traversed by the network data unit along the data flow path from the application transmitter to the application receiver, such as a NIC (e.g., a device installed in a node on which an application executes) and network devices of the network (e.g., switches, routers, etc.). All or any portion of the hops along the data flow path can be configured / enabled to add telemetry metadata to the network data unit.

[0015] Any hop device configured in this way can include matching action rules. In one or more embodiments, the device performs matching on any one or more portions of any header or field of a network data unit, such as Layer 2 (L2), L3, and / or L4 fields, which may include, but are not limited to, source and destination Internet Protocol (IP) addresses, source and destination Media Access Control (MAC) addresses, source and destination port numbers (e.g., Transmission Control Protocol (TCP) and / or General Datagram Protocol (UDP) port numbers), Virtual Local Area Network (VLAN) tags, Virtual Network Identifiers (VNIs), flow tags, Differential Service Code Point (DSCP) values, etc.

[0016] In one or more embodiments, once a match is identified, the corresponding action is to insert telemetry metadata into the network data unit, which occurs at each hop where it is enabled, as disclosed in the embodiments herein. In one or more embodiments, in the first instance of telemetry metadata insertion, the telemetry header, along with the telemetry metadata, is sent at that hop to signal the start of telemetry metadata within the network data unit. The total size of the telemetry metadata added by the device at a hop can be very small (e.g., 64 bytes), resulting in little or no impact on the overall network performance. Any potential impact can be mitigated to some extent by, for example, configuring the device to add metadata only to a specific portion of the matching network data units (e.g., one out of every N network data units) instead of all matching network data units, or by not adding telemetry metadata to packets below a configured size threshold. In one or more embodiments, devices after the first hop (e.g., the NIC of a node on which an application transmitter is executed) may be able to match on fields in the telemetry metadata added by the previously added device.

[0017] As an example, a network data unit sent from an application transmitter can be provided to the NIC of a node on which the application transmitter executes, allowing the NIC to send the network data unit into the network (e.g., the network data unit enters the network). The NIC can perform a match using all or any portion of the information received in association with the network data unit to be sent. When a match is identified, the NIC can insert telemetry metadata into the network data unit (e.g., between the L4 and L3 headers). The telemetry metadata can include both network information and application information, allowing the network data unit to be associated with a specific application (or any part thereof). As an example, application information can include an application identifier (identifying the entire application) and may additionally or alternatively include tokens or other application instruments, application library call tokens, etc., indicating the application execution stage / level. Such instruments can be added explicitly from the application, implicitly through linked communication libraries, and / or by matching specific communication identifiers.

[0018] The information included in the telemetry metadata can be of any type without departing from the scope of the embodiments disclosed herein, and may include telemetry data and telemetry-related status information. Examples include, but are not limited to, device identifiers and related information, ingress and egress ports, latency information, ingress and egress timestamps, queue and buffer states, information about transformations performed on the network data unit, forwarding and routing decisions, and / or any optional programmable field information (e.g., functions performed by an application, application execution phase, more specific application or application portion identification information, operation codes recognizable by the application user, free text of any kind, etc.). In one or more embodiments, the telemetry metadata may include additional programmable fields that may be used for any optional scraping of any of the device's configuration status, device counters, error conditions, etc. As an example, any data stored in one or more registers of the device that adds the above-described telemetry metadata to the network data unit may be included in the collected metadata. In one or more embodiments, the telemetry metadata may include queue depth, utilization, or load values, which may include contributions added by unrelated or other application network services, which may be the reason for suboptimal or below-expected performance of the application collecting the telemetry metadata. In one or more embodiments, the telemetry metadata may include service class information. As an example, the best-effort class may be affected by the dedicated class, and the packets for collecting metadata sent in the best-effort class may be able to see that the link is also delivering packets in the dedicated class, which may be the reason for higher-than-expected latency.

[0019] Continuing the example above, in one or more embodiments, once a network data unit including the aforementioned telemetry metadata added by the NIC of the node sending the application enters the network, each hop of the data flow path through the network, including network devices configured to perform the embodiments disclosed herein, can similarly perform a match on any L2, L3, L4, or previously inserted telemetry metadata (e.g., any data in any field of any header of the network data unit). In one or more embodiments, when the network device identifies a match, the network device can perform a corresponding action of inserting its own telemetry metadata into the network data unit, including any network telemetry information (as described above) and any optional configuration information (e.g., the network device's register state) that is expected to be collected for the application's data flow.

[0020] In one or more embodiments, any device along the data flow path can be configured as the last hop on which telemetry metadata is added to the network data unit. As an example, the NIC of the node on which the application receiver operates can be the last hop for adding telemetry metadata. As another example, any network device along the data flow path before the network data unit reaches the application receiver's node can be the last hop on which telemetry metadata is added.

[0021] In one or more embodiments, the last-hop device (e.g., the NIC of the receiving node, a network device, etc.) to which telemetry metadata is inserted in the network data unit may also be configured to strip the telemetry metadata added at each hop from the network data unit, the hop having a device configured to perform the techniques disclosed herein. In one or more embodiments, the network data unit without telemetry metadata may then be provided to the application receiver. As an example, when the NIC of the application receiver's node is the last hop, the NIC may strip the telemetry metadata and pass the network data unit up the network stack to the application receiver. As another example, when the last hop is a network device along the data flow path, the network device may strip the telemetry metadata and send the network data unit to the application receiver's node.

[0022] In one or more embodiments, the stripped telemetry metadata can be used to generate a report that includes application information and network information, including the telemetry metadata. In one or more embodiments, the report is provided to an application collector that executes on a node of the application receiver. The application collector can then store the telemetry metadata from the report (e.g., in memory) for any purpose (e.g., directly in real time when the telemetry metadata arrives at the application receiver, or indirectly during post-analysis of the telemetry metadata from the stored data).

[0023] As an example, telemetry metadata can be used to render real-time information about application performance in a user interface for viewing by the application's user. As another example, telemetry metadata can be stored in a data structure on a persistent storage device (e.g., a database) for later consumption and / or analysis. As yet another example, telemetry metadata can be provided to a remote collector configured to receive application-specific telemetry metadata, which can also be configured to perform analysis on aggregated telemetry metadata for the application. Application-specific telemetry metadata can be used for any other purpose without departing from the scope of the embodiments disclosed herein.

[0024] Certain embodiments of this disclosure can provide the ability to associate application-specific information with network telemetry metadata to allow application users to gain in-depth insights into application performance and / or operation while using the network. Such capability can be provided by configuring the NIC and network devices to match information from network data units and performing operations based on the match, wherein the operations include adding various telemetry metadata to the network data unit, including network telemetry information and any other application-related information. An application collector, executing on the same node as the intended application receiver for the network data unit, can receive telemetry metadata inserted into the network data unit at each configured hop and stripped from the network data unit at the last such hop.

[0025] Telemetry metadata can then be stored and / or used to obtain information about application performance, as it includes both network telemetry information for each hop and application-specific information that allows the network telemetry information to be associated with a specific application. The collection of application-specific telemetry metadata can be scalable because it is collected by an application collector that executes at each node of the application, rather than a centralized collector that receives network telemetry data for the entire network, regardless of the application causing the data flow. This avoids the time-consuming and resource-intensive post-processing of network telemetry data required by collectors attempting to associate data flows with specific applications. Furthermore, telemetry metadata can provide insights that lead to improved load balancing in dynamically routed networks and can also help improve tail latency.

[0026] Figure 1 A block diagram is shown for implementing an example system for acquiring application-specific telemetry metadata, according to one or more embodiments disclosed herein. Figure 1 As shown, the system may include any number of nodes (e.g., node A 100, node B 102) and network 108. In one or more embodiments, a node (e.g., node A) may include application transmitter 116 and NIC A 104. In one or more embodiments, a node (e.g., node B) may include application receiver 118, application collector 120, and NIC B 106. In one or more embodiments, network 108 may include any number of network devices (e.g., network device A 110, network device B 112, network device N 114). Each of these components will be described below.

[0027] In one or more embodiments, a node (e.g., node A 100, node B 102) is a computing device. In one or more embodiments, as used herein, a computing device can be any single computing device, a collection of computing devices, a portion of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. In one or more embodiments, a computing device is any device, a portion of a device, or any collection of devices capable of electronically processing instructions, and may include, but is not limited to, any of the following: one or more processors (e.g., components including circuitry systems) (not shown), memory (e.g., random access memory (RAM)) (not shown), (a plurality of) input and output devices (not shown), non-volatile storage hardware (e.g., solid-state drives (SSDs), persistent memory (Pmem) devices, hard disk drives (HDDs) (not shown)), one or more physical interfaces (e.g., network ports, storage ports) (not shown), any number of other hardware components (not shown), and / or any combination thereof.

[0028] Examples of computing devices include, but are not limited to, servers (e.g., blade servers in blade server chassis, rack servers in racks, etc.), desktop computers, mobile devices (e.g., laptops, smartphones, personal digital assistants, tablets, automotive computing systems and / or any other mobile computing devices), storage devices (e.g., disk drive arrays, Fibre Channel storage devices, Internet Small Computer System Interface (iSCSI) storage devices, tape storage devices, flash memory arrays, network-attached storage devices, etc.), network devices (e.g., switches, routers, multilayer switches, etc.), virtual machines, virtual computing environments, logical containers (e.g., for one or more applications), Internet of Things (IoT) devices, node arrays of computing resources, supercomputing devices, data centers or any part thereof, and / or any other type of computing device having the above requirements.

[0029] In one or more embodiments, nodes (e.g., node A 100, node B 102) may be part of a collection of any number of nodes configured to operate as a high-performance computing (HPC) environment. In one or more embodiments, an HPC environment may include any number of nodes that may be homogeneous or heterogeneous in terms of device capabilities and provide a platform for executing HPC applications, such as artificial intelligence (AI), machine learning, deep learning, autonomous driving, product design and manufacturing, weather modeling and forecasting, seismic data analysis, financial risk assessment, fraud detection, computational fluid dynamics, DNA sequencing, contextual search algorithms, traffic management, complex simulations, drug research, virtual reality, augmented reality, etc. In one or more embodiments, an HPC environment typically provides a platform for executing application workloads that use a large number of nodes to execute various parts of the application and therefore frequently send data to each other over a network (discussed further below).

[0030] In one or more embodiments, any or all of the above examples can be combined to create a system with such a device, or they can be divided into separate logical devices, which may be individually or collectively referred to as computing devices. Other types of computing devices may be used without departing from the scope of the embodiments described herein, for example... Figure 5 The computing device shown and described below. The system may include any number and / or type of such computing devices (e.g., nodes) in any arrangement and / or configuration without departing from the scope of the embodiments disclosed herein.

[0031] In one or more embodiments, the storage and / or memory of the computing device or computing device system may be and / or include one or more data repositories for storing any number of data structures that store any number of data (e.g., information). In one or more embodiments, the data repository is any type of storage unit and / or device for storing data (e.g., file system, database, collection of tables, RAM, hard disk drive, solid-state drive, and / or any other storage mechanism or medium). Furthermore, the data repository may include multiple different storage units and / or devices. These multiple different storage units and / or devices may or may not be of the same type or located in the same physical location.

[0032] In one or more embodiments, any storage and / or memory of a computing device or computing device system may be regarded, in whole or in part, as a non-transitory computer-readable medium storing software and / or firmware.

[0033] Such software and / or firmware may include instructions that, when executed by one or more processors (not shown) and / or other hardware (e.g., circuitry) of a computing device and / or a computing device system, cause one or more processors and / or other hardware components to perform operations according to one or more embodiments described herein.

[0034] The software instructions may be in the form of computer-readable program code that performs the methods, processes, etc. of the embodiments described herein, and by way of example, may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer-readable medium, such as an optical disc (CD), a digital multifunction disc (DVD), a storage device, a floppy disk, a magnetic tape storage device, flash memory, physical memory, or any other non-transitory computer-readable medium.

[0035] The system may include any number of nodes, as used herein, and any number of these nodes may be considered individually or collectively as computing devices. All or any part of the computing devices may be computing devices of the same or different types.

[0036] In one or more embodiments, a node (e.g., node A 100, node B 102) includes a NIC (e.g., NIC A 104, NIC B 106). In one or more embodiments, the NIC is an input and / or output component configured to provide an interface between a node (e.g., node A 100, node B 102) and a network (e.g., network 108 described below). In one or more embodiments, the NIC (e.g., NIC A 104, NIC B 106) is used to receive and / or transmit network data units. A network data unit may include any number of payloads (e.g., data intended to be consumed by an entity receiving the network data unit) within a header and / or trailer, which may be information fields designed to allow the receiving entity to perform various actions to propagate the network data unit to a destination (e.g., another device, application receiver, etc.). Such information fields may include, but are not limited to, various information items related to the protocols used to implement data transmission (e.g., Media Access Control (MAC), Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Address Resolution Protocol (ARP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Virtual Extensible Local Area Network (VXLAN), Multiprotocol Label Switching (MPLS) Segment Routing (SR), etc.), addresses and / or labels related to such protocols (e.g., IP address, MAC address, label stack, etc.), fields related to error identification and / or correction, etc. The NIC (e.g., NIC A 104, NIC B 106) can be configured with any type of interface for receiving and / or transmitting network data units, such as wireless interfaces, wired interfaces, etc. Although Figure 1 A node including a single NIC (e.g., node A 100, node B 102) is shown. Computing devices may include any number of SmartNICs without departing from the scope of the embodiments disclosed herein.

[0037] In one or more embodiments, the NIC (e.g., NIC A 104, NIC B 106) may be a SmartNIC. In one or more embodiments, a SmartNIC is a NIC that includes additional processing resources relative to a standard NIC. A SmartNIC may include various hardware components, subsystems, etc., configured to perform processing on received network data units to offload at least some of such processing from one or more processors of a computing device. Such hardware components may include, but are not limited to, field-programmable gate arrays (FPGAs), system-on-chip (SoC) digital signal processors (DSPs), etc. Such hardware components may be (or be included in) one or more subsystems (e.g., RISC-ARM subsystems) of the SmartNIC.

[0038] In one or more embodiments, a NIC (e.g., NIC A 104, NIC B 106) can be configured to identify network data units using matching action rules and perform actions specified by those rules. In one or more embodiments, matching can be performed using any information included in and / or associated with the network data unit received at the NIC (e.g., an application transmitter from the sending network data unit, an application receiver designed for the receiving network data unit, etc.). In one or more embodiments, corresponding actions based on such matching can include inserting telemetry metadata into the network data unit, stripping telemetry metadata from the network data unit, etc. (The following is a summary of the previous embodiments.) Figure 2 , Figure 3 Various operations of the NIC performed according to one or more embodiments disclosed herein are further discussed in the description of Figure 4. Although Figure 1 A node with a single NIC is shown, but a node may include any number of NICs without departing from the scope of the embodiments disclosed herein.

[0039] In one or more embodiments, a node (e.g., node A 100) may include an application transmitter (e.g., application transmitter 116). In one or more embodiments, application transmitter 116 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof, at least partially configured to cause data to be transmitted over a network. As an example, an application may be implemented to perform a workload in an HPC environment comprising thousands of nodes, where each node performs a portion of the application. In such a scenario, a portion of the application executing on a given node (e.g., application transmitter 116 executing on node A 100) may be configured to periodically send data to another portion of the application executing on another node (e.g., application receiver 118 executing on node B 102). Although Figure 1 A node A100 with a single application transmitter 116 is shown, but a node may include any number of application transmitters without departing from the scope of the embodiments disclosed herein.

[0040] In one or more embodiments, a node (e.g., node B 102) may include an application receiver (e.g., application receiver 118). In one or more embodiments, application receiver 118 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof, at least partially configured to receive data that has been transmitted over a network (e.g., network 108). As an example, application receiver 118 may be part of an application performing workloads in a cloud environment and may receive data from other parts of the application (e.g., application transmitter 116) from time to time. Although Figure 1A node B 102 with a single application receiver 118 is shown, but a node may include any number of application receivers without departing from the scope of the embodiments disclosed herein. Furthermore, a node may include any number of application transmitters and may also include any number of application receivers without departing from the scope of the embodiments disclosed herein.

[0041] In one or more embodiments, a node (e.g., node B 102) may include an application collector 120. In one or more embodiments, the application collector 120 is any hardware (e.g., circuitry), software, firmware, and / or any combination thereof configured to receive telemetry metadata reports at least partially from the last-hop device along the data flow path of the network data unit. In one or more embodiments, as used herein, the last-hop device is the last device along the data flow path (e.g., a NIC, network device, etc.) configured to perform operations according to one or more embodiments disclosed herein. In one or more embodiments, the telemetry metadata report is a collection of information based on telemetry metadata stripped from the network data unit before the network data unit is provided to an application receiver (e.g., application receiver 118). In one or more embodiments, the telemetry metadata includes application-specific information and network telemetry information added to the network data unit by one or more devices configured to perform operations according to one or more embodiments disclosed herein. The content of the telemetry metadata report, and the delivery of such a report to the application collector (e.g., application collector 120), are described below. Figure 2 , Figure 3 This will be discussed further in the description of Figure 4. Although Figure 1 A single node (e.g., node B 102) is shown as including a single application collector (e.g., application collector 120), but any number of nodes may include any number of application collectors, application receivers, and / or application senders without departing from the scope of the embodiments disclosed herein.

[0042] In one or more embodiments, each node of the system (e.g., node A 100, node B 102) is operatively connected to a network (e.g., network 108). As an example, the NICs (e.g., NIC A 104, NIC B 106) of the nodes (e.g., node A 100, node B 102) can provide operative connectivity to network 108. A network (e.g., network 108) can refer to the entire network or any part thereof (e.g., a logical portion of network devices within the topology of network devices). Network 108 can be and / or include data center networks, wide area networks, local area networks, wireless networks, cellular telephone networks, InfiniBand networks, and / or any other suitable network that facilitates the exchange of information from one part of a network to another (e.g., transmission via network data units). Network 108 can be any combination of the above network types. Network 108 can be located at a single physical location or distributed across any number of physical sites. In one or more embodiments, the network can be at least partially coupled to or overlap with the Internet.

[0043] In one or more embodiments, the network includes any number of network devices (described below) that may collectively implement any number of protocols (e.g., routing and forwarding protocols) and / or technologies (e.g., load balancing techniques) that contribute to decisions regarding the data flow path that a particular network data unit can traverse from one node (e.g., node A 100) operatively connected to network 108 to another node (e.g., node B 102) operatively connected to the network. Such decisions may result in network 108 having any number of possible data flow paths that can traverse between nodes, and the decisions made by the network devices at various points in time may contribute to the fact that the data flow paths are inherently dynamic (e.g., not predetermined for any given network data unit).

[0044] In one or more embodiments, network 108 may include any number of network devices (e.g., network device A110, network device B112, network device N114). The three points between network device B112 and network device N114 are intended to illustrate that network 108 is not limited to any particular number of network devices, and therefore may include any number of network devices, which can be a large number of network devices to facilitate complex networks often implemented in data center environments, HPC environments, etc.

[0045] In one or more embodiments, a network device (e.g., 110, 112, 114) is a device that includes and / or is operatively connected to persistent storage (not shown), memory (e.g., random access memory (RAM)) (not shown), one or more processors (e.g., integrated circuits, ASICs, etc.) (not shown), and at least one physical network interface (not shown), which may also be referred to as a port and may provide connectivity (i.e., a link) to other devices (e.g., computing devices, other network devices, etc.).

[0046] In one or more embodiments, the network device (e.g., 110, 112, 114) also includes any number of additional components (not shown), such as network chips, FPGAs, application-specific integrated circuits (ASICs), indicator lights (not shown), fans (not shown), power supply units, power distribution units, etc. At least a portion of such hardware components may be included as part of one or more line cards of the network device 100. In one or more embodiments, as used herein, a line card refers to a collection of hardware components (e.g., connected via a printed circuit board) including one or more physical interfaces (e.g., network ports) and any number of additional hardware components (e.g., ASICs, FPGAs, TCAMs, processor components, other memory components, etc.) that are at least partially used for storing forwarding information and processing network traffic. Forwarding information (which may include all or any part of information that may be referred to as “FIB”) may be stored, for example (e.g., in various tables) in one or more hardware components of the line card, such as MAC tables, routing table entries, multicast forwarding entries, etc., which are sometimes referred to as part of the data plane. In one or more embodiments, the network device includes information such as a routing information base (RIB) that includes information (e.g., obtained from various routing protocols) that can be used to program components of the network device to propagate network data units. Such information is sometimes referred to as being in the control plane. The network device (e.g., 110, 112, 114) may include any other components without departing from the scope of the embodiments described herein.

[0047] In one or more embodiments, as discussed above, the network device (e.g., 110, 112, 114) includes at least one physical interface (and typically two or more such physical interfaces). In one or more embodiments, the physical interface is any hardware, software, or combination thereof that includes the ability to receive and / or send network data units (e.g., packets, frames, etc.) or any other information to the network device (e.g., 110, 112, 114). The physical interface can include any interface technology, such as, for example, optical, electrical, etc. The physical interface can be configured to interface with any transmission medium (e.g., fiber optic, copper wire, etc.).

[0048] In one or more embodiments, the physical interface includes and / or is operatively connected to any number of components used in network traffic processing. For example, a given physical interface may include a PHY (not shown), which is a circuit system connecting a physical information propagation medium (e.g., wires) to other components (e.g., hardware components of a line card) that process network traffic. In one or more embodiments, the physical interface includes and / or is operatively connected to a transceiver that provides a connection between the physical information transmission medium and the PHY. The PHY may also include any number of other components, such as serializers / deserializers (SERDES) and numberers / decoders. The PHY may in turn be operatively connected to any number of other components, such as a Media Access Control (MAC) sublayer. Such a sublayer may in turn be operatively connected to other higher-level processing components, all of which form a series of components used by network devices (e.g., 110, 112, 114) for any purpose to receive, transmit, or otherwise process network traffic.

[0049] In one or more embodiments, the network device (e.g., 110, 112, 114) includes any software (e.g., various daemons, state databases, etc.) configured to perform and / or allow other components to perform various functions of the network device (e.g., processing network traffic). For example, such software can be executed using one or more processors of the network device or any other hardware resource of the network device capable of executing the software.

[0050] Examples of network devices (e.g., 110, 112, 114) include, but are not limited to, switches, routers, multilayer switches, Fibre Channel devices, InfiniBand® devices, etc. Network devices are not limited to the specific examples mentioned above.

[0051] In one or more embodiments, the network device (e.g., 110, 112, 114) includes functionality for receiving and processing network data units (e.g., frames, packets, tunneling protocol frames, etc.) at any physical interface (e.g., port) of the network device (e.g., 110, 112, 114). In one or more embodiments, processing network traffic includes, but is not limited to, a series of one or more lookups (e.g., longest prefix match (LPM) lookup, forwarding equivalence class (FEC) lookup, etc.) and corresponding actions (e.g., forwarding from an egress port, adding a tag header, rewriting the destination address, encapsulation, etc.). Examples of network traffic processing include, but are not limited to, determining: (i) whether to take security measures (e.g., discarding network traffic data units); (ii) whether to mirror network traffic data units; and / or (iii) how to route / forward network traffic data units to send them from the network device's interface.

[0052] In one or more embodiments, network devices (e.g., 110, 112, 114) are configured with any number of matching action rules. In one or more embodiments, a matching action rule is a rule that causes network devices (e.g., 110, 112, 114) to perform a match on any information included in a network data unit (e.g., any number of fields in any number of headers, any telemetry metadata previously added to the network data unit by other devices, etc.), and, upon successful matching, to perform one or more corresponding actions (e.g., adding telemetry metadata) (e.g., one or more fields in the network data unit match information that the network device is configured to include and associate with the matching action rule). The following... Figure 2 , Figure 3 The use of matching action rules according to one or more embodiments disclosed herein is further discussed in the description of Figure 4.

[0053] Although Figure 1 A specific configuration for the component is shown, but other configurations can also be used without departing from the scope of the embodiments described herein. For example, although Figure 1 Some components may be shown as part of the same device, but any component can be grouped into a collection of one or more components, any of which can exist and perform as part of any number of separate and operatively connected devices. As another example, a single component can be configured to perform... Figure 1 The components shown may perform all or any part of the functionality. Therefore, the embodiments disclosed herein should not be limited to... Figure 1 The configuration of the components shown.

[0054] Figure 2 An overview of example methods for obtaining application-specific telemetry metadata according to one or more embodiments disclosed herein is shown. Figure 2 All or any part of the methods shown can be performed by, for example, a device or collection of devices configured to perform operations according to one or more embodiments disclosed herein (e.g., Figure 1 The network devices (NIC A 104, NIC B 106, network device A 110, network device B 112, and network device N 114) are used to execute this.

[0055] Although Figure 2 The steps in the flowchart shown are presented and described in sequence, but some or all of the steps may be performed in a different order, and some or all of the steps may be combined or omitted. Figure 2 Other steps not shown may be performed in addition, and / or some or all of the steps may be performed in conjunction with... Figure 2 The other steps are executed in parallel.

[0056] In step 200, the method includes being sent by an application transmitter (e.g., Figure 1 The application transmitter 116) executes on the first node (e.g., Figure 1 The NIC of node A 100 (e.g., Figure 1 The NIC (NIC A 104) identifies the network data unit used for telemetry metadata insertion. In one or more embodiments, the identification of the network data unit occurs by the NIC executing matching action rules configured by the NIC. In one or more embodiments, executing the matching action rules includes performing a matching operation on any information included in and / or associated with the network data unit. Such information may include, but is not limited to, any one or more portions of any header or field of the network data unit, such as L2, L3, and / or L4 fields, which may include, but are not limited to, source and destination IP addresses, source and destination MAC addresses, source and destination port numbers (e.g., TCP and / or UDP port numbers), VLAN tags, VNIs, flow tags, Differential Service Code Point (DSCP) values, application identifiers, flow identifiers, flow tags, and protocol information, etc. As an example, a network data unit sent from an application sender may be provided to the NIC of a node on which the application sender executes, so that the NIC can send the network data unit into the network (e.g., the network data unit enters the network). The NIC may use all or any portion of the information received in association with the network data unit to be sent to perform the matching. In one or more embodiments, where the NIC of the node on which the application transmitter is executed is not configured with such matching action rules (e.g., operations performed according to one or more embodiments disclosed herein), the first matching operation may be performed alternately by a first device (e.g., a network device) along the data flow path of the network data unit thus configured.

[0057] In step 202, the method includes inserting initial telemetry metadata into the network data unit based on the success of the first matching operation. As an example, the initial telemetry metadata may be inserted into the network data unit by a NIC that provides the network data unit from an application sender. In one or more embodiments, the initial telemetry metadata includes both network information (e.g., network telemetry information and / or any other network information related to and / or associated with the device into which the telemetry metadata is inserted) and application information (e.g., any information specific to an application to which the application sender is part). In one or more embodiments, the initial telemetry metadata may be considered to include a metadata header indicating the start of telemetry metadata within the network data unit.

[0058] Initial telemetry metadata may include, but is not limited to: information identifying the application; information identifying a part of the application that caused the data transmission (e.g., process, application phase, application level, application function, application operation, etc.); device information (e.g., the NIC's device identifier); the ingress and / or egress ports of the network data unit; latency information, which may include the time the network data unit arrives at the NIC, the time spent processing the network data unit before transmission; queue arbitration parameters (e.g., the time the network data unit spends in the buffer); ingress and egress timestamps, which may be high-resolution timestamps (e.g., at the nanosecond level) obtained using techniques such as Precise Time Protocol (PTP); header conversion information, processing and / or... This may include: time of change; link utilization information; link load and congestion indicators; various queue and buffer states; network data unit pipeline operation; network data unit transformation; virtual routing and forwarding (VRF) information; flow information (e.g., whether the network data unit is initiating a new flow or is part of an ongoing flow); changes in network data unit size; information about routing and forwarding decisions made; any other types of data included in the optional scraping fields (e.g., device registers, statistics, counters, code indicating something about the application to the user, any kind of free text, etc.), information about why the network data unit may be blocked for a period of time, measurements of congestion type and / or degree, and / or any combination thereof. Any other relevant information may be included in the initial telemetry metadata without departing from the scope of the embodiments disclosed herein. In one or more embodiments, a portion of the telemetry metadata may be standard telemetry information based on a network monitoring framework (e.g., in-band network telemetry (INT)), while another portion may be information specific to the embodiments disclosed herein, specific to the application sending the network data unit, which may change over time as the user of the application fine-tunes the information needed to evaluate the application's performance.

[0059] In one or more embodiments, inserting initial telemetry metadata into a network data unit can include inserting the telemetry metadata at a predefined, specific location within the network data unit. As an example, the NIC may insert the initial telemetry metadata between a first portion of the network data unit's header (e.g., a header related to the application, presentation, session, and / or transmission layers) and other headers that allow the network data unit to traverse network devices (e.g., network, data link, and physical layers). For example, the initial telemetry metadata may be inserted below the Ethernet header (e.g., including MAC address information) and the IP header (e.g., including IP address information) and above the remainder of the network data unit (e.g., L4 payload). The initial telemetry metadata may be inserted at any other location within the network data unit without departing from the scope of the embodiments disclosed herein.

[0060] In one or more embodiments, including both application-specific information and network information in the initial telemetry metadata allows the collection of telemetry metadata for a network data unit to be associated with a specific application, thereby enabling the transmission of the network data unit without extensive post-processing of the network data unit by a collector of such information that is not application-specific, and may allow devices along the data flow path to identify the network data unit as being associated with a specific application and / or a portion thereof, so as to customize the telemetry metadata added to the network data unit for the specific application.

[0061] In one or more embodiments, the matching action rule (e.g., configured on the NIC of the node performing the application sender or on any other device along the data flow path) may include all or any portion of the network data units for which the matching operation succeeded, with inserted telemetry metadata. As an example, in some cases, it may be important to insert telemetry metadata into each such network data unit to gain insight into the application's performance as the application sends data over the network. Another example is that the matching action rule may specify that only some network data units' telemetry metadata is inserted (e.g., so as not to adversely affect the network data unit's transmission performance), such as inserting network telemetry metadata only into a portion of the matched network data units (e.g., one in every ten units), or not inserting telemetry metadata into network data units below a threshold size (e.g., so that the size of the network data unit does not significantly increase due to insertion). In one or more embodiments, the matching action rule may specify that telemetry metadata is inserted only for certain service classes (which may be indicated by one or more fields in the network data unit).

[0062] In step 204, the method includes the NIC sending network data units to network devices (e.g., Figure 1 (One of network devices 110, 112, or 114). In one or more embodiments, transmitting a network data unit includes preparing the network data unit for transmission (e.g., performing any transformations on the network data unit, such as adding and / or modifying header information identifying the next hop), and transmitting the network data unit as a series of bits transmitted over a transmission medium.

[0063] In step 206, the method includes a network device along the data flow path of the network data unit performing a matching operation using any information included in the network data unit. In one or more embodiments, a network device similar to the NIC described above may be configured with one or more matching action rules and may perform a matching operation according to such rules to determine whether the network data unit has successfully matched information that the network device is configured to perform a comparison. The network device performing such a matching action rule may be a first network device on the data flow path configured to perform the operation according to the embodiments disclosed herein, which may be the first network device that the network data unit arrives at after leaving the application transmitter node, or may be any subsequent device along the data flow path. In one or more embodiments, the execution of the matching action rule may be similar to the execution discussed above with respect to the NIC of the application transmitter node. Therefore, in one or more embodiments, the network device may match any information in any field of any header of the network data unit, and any telemetry metadata previously added to the network data unit according to the embodiments disclosed herein, or any combination of such information.

[0064] In step 208, the method includes inserting additional telemetry metadata into the network data unit based on the successful matching operation performed in step 206. In one or more embodiments, the additional telemetry metadata may include all or any portion of the information type inserted as initial telemetry metadata (as discussed above), but is specific to the network device performing the insertion of the additional telemetry metadata. As an example, the additional telemetry metadata may include any standard in-band network telemetry data (e.g., ingress / egress interface, latency information, routing and forwarding decision information, buffer and queue depths, etc.) and any optional information specific to the network device when receiving, processing, and / or transmitting the network data unit (e.g., device register status, counter information, etc.). In one or more embodiments, the set of information included in the additional telemetry metadata is configured via matching action rules implemented on the network device.

[0065] In one or more embodiments, additional telemetry metadata is inserted into a network data unit in a location similar to the initial telemetry metadata and / or any other telemetry metadata added by other network devices before the network data unit arrives at the network device. As an example, the network device may insert additional telemetry metadata after the initial telemetry metadata in the network data unit, where "after" refers to the location of the telemetry metadata in the telemetry metadata stack built when each hop adds its telemetry metadata to the network data unit.

[0066] In one or more embodiments, any number of network devices may exist along the data flow path of the network data unit, and each device configured according to the embodiments disclosed herein, having a matching action rule that causes telemetry metadata to be added based on a successful match, may similarly insert additional telemetry metadata associated with the network device into which the telemetry metadata is inserted. This may result in a per-device stack of telemetry metadata within the network data unit. As an example, starting with the telemetry header and telemetry metadata as part of the initial telemetry metadata, a first device (such as a NIC on a node in which an application transmitter performs) may insert the initial telemetry data at a specific location in the network data unit (e.g., between the L3 and L4 headers). The telemetry header in the initial telemetry metadata may signal the start of the telemetry metadata of the network data unit. At each subsequent hop where the telemetry metadata is added here, additional telemetry metadata may be appended to the previous telemetry metadata added by one or more previous hops. Thus, each unit of telemetry metadata added by hops along the data path may be a discrete unit of telemetry metadata added sequentially as part of the initial telemetry data to the network data unit after the telemetry header added by the first device.

[0067] In step 210, the method includes sending a network data unit (which includes initial telemetry metadata and additional telemetry metadata added by any number of other network devices) from the network device to the last-hop device along the data flow path of the network data unit. In one or more embodiments, any device that is a hop along the data flow path of the network data unit can be configured as the last hop on which telemetry metadata is added to the network data unit. As an example, the NIC of the node on which the application receiver is executed can be the last hop for adding telemetry metadata. As another example, any network device along the data flow path before the network data unit reaches the node of the application receiver can be the last hop on which telemetry metadata is added. In one or more embodiments, although in Figure 2 Not shown, but the last-hop device also adds additional telemetry metadata to the network data unit, similar to the insertion of additional telemetry metadata discussed in the description of step 208 above (e.g., performing a matching operation and, based on a successful match, performing an action to insert last-hop device-specific additional telemetry metadata).

[0068] In step 212, the method includes stripping initial telemetry metadata and all additional telemetry metadata from the network data unit by the last-hop device. In one or more embodiments, although in Figure 2Not shown, but network data units without stripped telemetry metadata can be provided to an application receiver intended to be a network data unit. As an example, when the NIC of the application receiver node is the last hop, the NIC can strip the telemetry metadata and pass the network data unit up the network stack to the application receiver. As another example, when the last hop is a network device along the data flow path, the network device can strip the telemetry metadata and send the network data unit to the application receiver node.

[0069] In step 214, the method includes generating a telemetry metadata report by the last-hop device, which includes initial telemetry metadata and all additional telemetry metadata. The telemetry metadata report can be in any suitable form (e.g., raw data formatted in a particular way, etc.). Therefore, the telemetry metadata report includes, but is not limited to, application-specific information (e.g., information identifying the application sending the network data unit or any part thereof), and network telemetry information from each hop along the data flow path to which telemetry metadata is added to the network data unit. Various examples of the types of information that can be included in the telemetry metadata report (e.g., initial telemetry metadata and additional telemetry metadata) can be found, for example, in the descriptions of steps 202 and 208 above.

[0070] In step 216, the method includes providing a telemetry metadata report from the last-hop device to the application receiver to be executed in the network data unit (e.g., Figure 1 The application receiver 118 is located on the same node (e.g., Figure 1An application collector (e.g., application collector 120 of 1) is executed on node B (102). The application collector can then store telemetry metadata from the reports (e.g., in memory) for any purpose. As an example, the telemetry metadata can be used to render real-time information about application performance in a user interface for the user of the application to view. As another example, the telemetry metadata can be stored in a data structure in persistent storage (e.g., a database) for later consumption and / or analysis. As yet another example, the telemetry metadata can be provided to a remote collector configured to receive application-specific telemetry metadata, which can also be configured to perform analysis on aggregated telemetry metadata of the application. Application-specific telemetry metadata can be used for any other purpose without departing from the scope of the embodiments disclosed herein. In one or more embodiments, as discussed above and below, providing telemetry metadata reports to an application collector that runs on the node where the application receiver executes, rather than to a centralized collector, allows the collection of application-specific telemetry metadata to be scalable because it is collected by an application collector that runs at each node of the application, rather than by a centralized collector that receives network telemetry data from the entire network, regardless of the application that caused the data stream, and avoids the need for time-consuming and resource-intensive post-processing of network telemetry data by a collector that attempts to associate data streams with specific applications.

[0071] Figure 3 An example system for obtaining application-specific telemetry metadata is shown, according to one or more embodiments disclosed herein. Figure 4A An example network data unit without telemetry metadata is shown according to one or more embodiments disclosed herein. Figure 4B An example network data cell with telemetry metadata inserted according to one or more embodiments disclosed herein is shown. Figure 3 The examples shown and described below are simplified examples intended for illustrative purposes only and are not intended to limit the scope of the embodiments described herein. Furthermore, while this example illustrates certain aspects of the embodiments described herein, not all possible aspects of such embodiments may be shown in this particular example.

[0072] Consider a scenario where an HPC application runs on one thousand nodes, two of which are node A 300 and node B 302. An application transmitter 322 running on node A 300 is configured to send network data units to an application receiver 324 running on node B 302. Users of the HPC application have noticed that the application's performance is intermittently lower than expected and suspect that the problem may be related to the network data units sent from the application's application transmitter to the application's application receiver. Therefore, according to the embodiments disclosed herein, the user uses various matching action rules to configure NIC 304 of node A 300, NIC B 306 of node B 302, and each network device (e.g., 308, 310, 312, 316, 318, and 320) in the network between these two nodes. These matching action rules allow data flows through the NICs and network devices to be associated with the application and its various components (e.g., different phases of the application, different application functions, etc.).

[0073] In such a scenario, application transmitter 322 can generate a network data unit, which may include, for example, a data payload, information from various upper-layer protocols (e.g., the sending layer), the destination IP address of node B 302 executed therein by application receiver 324, etc. The network data unit can be provided to NIC A 304. Without the embodiments disclosed herein, the NIC may add additional headers (e.g., IP header, MAC header) to send the network data unit to the application receiver. Examples of network data units prepared in this manner can be found in... Figure 4A As can be seen, the network data unit 400 includes an upper-layer payload 402 (including data payload and any upper-layer header), an IP header 404, and a MAC header 406.

[0074] However, in this case, NIC A 304 is configured with matching action rules related to the application sender 322. Therefore, NIC A 304 performs a matching operation that identifies the network data unit as originating from the application sender 322 at a specific stage of the application execution, in order to send a specific category of network data units to a specific TCP port (e.g., at least four fields of the network data unit match). Based on a successful match, NIC A 304 identifies that initial telemetry metadata should be inserted into the network data unit. Therefore, NIC A 304 inserts the initial telemetry metadata between the upper-layer payload 402 and the IP header 404 of the network data unit 400 and sends the network data unit to network device A 308.

[0075] Network device A 308 is also configured with application-specific matching action rules and similarly performs a successful matching operation, inserting additional telemetry metadata A into the network data unit. Based on the routing and forwarding configuration of network device A 308, the network data unit is forwarded to the next hop along the data path. Network device C 312 also performs a successful pairing operation and inserts additional telemetry metadata B into the network data unit. A similar process occurs along the data flow path via network device D 316 (which inserts additional telemetry metadata C into the network data unit) and network device F 320 (which inserts additional telemetry metadata D into the network data unit).

[0076] Next, the network data unit is sent to NIC B306 at node B 302, where application receiver 324 executes the operation. NIC B is the last hop along the data flow path and also executes the matching action rules to perform the matching operation and inserts additional telemetry metadata into the network data unit. Figure 4B The result network data unit 400, containing initial telemetry metadata and various additional telemetry metadata, can be seen. For example... Figure 4B As shown, at this time, the network data unit 400 includes initial telemetry metadata 408 added by NIC A 304, additional telemetry metadata A 410 added by network device A 308, additional telemetry metadata B 412 added by network device C 312, additional telemetry metadata C 414 added by network device D 316, and additional telemetry metadata E 418 added by NIC B 306 between the upper-layer payloads.

[0077] As the last-hop device, the NIC B 306 strips the initial telemetry metadata 408 and all additional telemetry metadata (410, 412, 414, 416, and 418) from the network data unit to obtain... Figure 4A The network data unit 400 is shown. Then it can be processed normally. Figure 4A The network data unit 400 is provided to the application receiver 324 of node B 302.

[0078] As the last-hop device, NIC B is also configured to generate a telemetry metadata report using the initial telemetry metadata and all additional telemetry metadata, and to provide this report to the application collector 326, which also runs on Node B 302. Thus, network information (e.g., network telemetry data) from each hop along the data flow path, along with application-specific information, is provided to the application collector 326, which can store or use this information for any purpose, such as allowing users of the application to evaluate potential causes of application performance degradation due to problems observed by network data units while traversing the network's data flow path. Since the telemetry metadata includes both network information and application-specific information, it is already application-related. Furthermore, the telemetry metadata for such network data units sent from the application sender to the application receiver is checked at the application collector to allow such analysis to be scaled to the size of the application (e.g., the number of nodes), thereby avoiding bottlenecks and post-processing issues that can occur when network telemetry data not specifically associated with the application is aggregated and sent to a remote collector configured to receive all such data from the network.

[0079] Figure 5 A block diagram of a computing device according to one or more embodiments of the present disclosure is shown. As discussed above, the embodiments described herein can be implemented using a computing device. For example, Figure 1 All or any part of the components shown can be implemented using at least one or more computing devices, and Figure 2 All or any part of the methods shown can be performed using one or more computing devices, such as computing device 500. Computing device 500 may include one or more computer processors 502, non-persistent storage 504 (e.g., volatile memory such as random access memory (RAM), cache memory, etc.), persistent storage 506 (e.g., hard disk, optical drive such as optical disc (CD) drive or digital versatile disc (DVD) drive, flash memory, etc.), communication interface 512 (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device 510, output device 508, and many other elements (not shown) and functions. Each of these components will be described below.

[0080] In one or more embodiments, the computer processor(s) 502 may be an integrated circuit for processing instructions. For example, the computer processor(s) may be one or more cores or microcores of a processor. Processor 502 may be a general-purpose processor configured to execute program code included in software executed on computing device 500. Processor 502 may be a special-purpose processor, in which certain instructions are incorporated into the processor design. Processor 502 may be an application-specific integrated circuit (ASIC), graphics processing unit (GPU), data processing unit (DPU), tensor processing unit (TPU), associative processing unit (APU), vision processing unit (VPU), quantum processing unit (QPU), and / or various other processing units using special-purpose hardware (e.g., field-programmable gate array (FPGA), system-on-a-chip (SOC), digital signal processor (DSP), etc.). Figure 5 Only one processor 502 is shown, but computing device 500 may include any number of processors without departing from the scope of the embodiments disclosed herein.

[0081] The computing device 500 may also include one or more input devices 510, such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, motion sensor, or any other type of input device. Input devices 510 allow users to interact with the computing device 500. In one or more embodiments, the computing device 500 may include one or more output devices 508, such as a screen (e.g., a liquid crystal display (LCD), plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), printer, external storage, or any other output device. One or more of the output devices may be the same as or different from the input devices(s). The input and output devices(s) may be locally or remotely connected to the computer processor(s) 502, non-persistent storage(s) 504, and persistent storage(s) 506. Many different types of computing devices exist, and the aforementioned input and output devices(s) may take other forms. In some instances, a multi-mode system allows users to provide multiple types of input / output to communicate with the computing device 500.

[0082] Furthermore, communication interface 512 can facilitate the connection of computing device 500 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device such as another computing device. Communication interface 512 can use wired and / or wireless transceivers of any type and / or technology to perform or facilitate the reception and / or transmission of wired or wireless communications. Examples include, but are not limited to, those using audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, BLE wireless signal transmission, IBEACON® wireless signal transmission, RFID wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, Visible Light Communication (VLC), Global Microwave Access Interoperability (WiMAX), IR communication wireless signal transmission and Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 512 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing device 500 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS includes, but is not limited to, US-based GPS, Russian-based GLONASS, Chinese-based BeiDou Navigation Satellite System (BDS), and European-based Galileo GNSS. There are no limitations on operation of any particular hardware arrangement, and therefore the basic features herein can be easily replaced by developed, improved hardware or firmware arrangements.

[0083] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored, but does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or storage devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0084] All or any part of the components of computing device 500 may be implemented in a circuit system. For example, components may include electronic circuitry or other electronic hardware (including one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits)) and / or may be implemented therein, and / or may include computer software, firmware, or any combination thereof and / or may be implemented therein to perform the various operations described herein. In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0085] In the foregoing description, many details have been set forth as examples of the embodiments described herein. Those skilled in the art (who also benefit from this disclosure) will understand that one or more embodiments described herein can be practiced without these specific details and can be varied or modified in many ways without departing from the scope of the embodiments described herein. Certain details known to those skilled in the art may have been omitted to avoid obscuring the description.

[0086] Specific details are provided in the above description to provide a full understanding of the aspects and examples presented herein. However, those skilled in the art will understand that these aspects can be practiced without these specific details. For clarity, in some instances, the technology may be presented as including functional blocks that may include devices, device components, steps, or routines in a method embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure aspects with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring aspects of the embodiments disclosed herein.

[0087] The above can describe each aspect as a process or method, which can be depicted as a flowchart, diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart can describe operations as a sequential process, many operations can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning from the calling function or the main function.

[0088] The processes and methods described in the examples above can be implemented using computer-executable instructions stored in or otherwise accessible from a computer-readable medium. For example, such instructions may include instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Some of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary or intermediate format instructions (such as assembly language, firmware, source code, etc.). Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0089] In the above description of the accompanying drawings, any component described with respect to the drawings in the various embodiments described herein is equivalent to one or more components with the same or similar names and / or numbers described with respect to any other drawing. For brevity, the description of these components may not be repeated for each drawing. Therefore, each embodiment of a component in each drawing is incorporated by reference and is assumed to be optionally present in each other drawing having one or more components with the same or similar names and / or numbers. Furthermore, any description of components in the figures according to the various embodiments described herein should be interpreted as an optional embodiment that can be implemented as a supplement, combination, or substitution to embodiments described with respect to one or more corresponding components with the same or similar names and / or numbers in any other drawing.

[0090] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in the application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to a single element, unless explicitly stated otherwise, such as through the use of terms like “before,” “after,” “single,” and others. Rather, the use of ordinal numbers is for distinguishing between elements. By way of example, a first element is distinct from a second element, and a first element may include more than one element, and is ordered as follows after (or before) the second element.

[0091] As used herein, the phrase “operationally connected” or “operationally linked” means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact in a certain way. For example, the phrase “operationally linked” can refer to any direct (e.g., a direct wire between two devices or components) or indirect (e.g., a wired and / or wireless connection between any number of devices or components that are operably linked) connection. Therefore, any path through which information can pass can be considered an operational link.

[0092] While the embodiments discussed herein have been described with respect to a limited number of embodiments, those skilled in the art who benefit from this disclosure will understand that other embodiments can be conceived without departing from the scope of the embodiments disclosed herein. Therefore, the scope of the embodiments described herein should be limited only by the appended claims.

Claims

1. A computer-implemented method, comprising: The network interface card (NIC) of the first node, executed by the application transmitter of the application, identifies a network data unit for telemetry metadata insertion. The identification is based on performing a first matching operation on information associated with the network data unit, wherein the network data unit is sent from the application transmitter of the first node to the application receiver of the application executed on the second node. The NIC inserts initial telemetry metadata into the network data unit based on the first success of the first matching operation. This initial telemetry metadata includes network information and application-specific information corresponding to the application. The application-specific information identifies the application; The second matching operation is performed by a network device hopping along the data flow path of the network data unit using at least one of the information associated with the network data unit and the initial telemetry metadata; The network device inserts additional telemetry metadata into the network data unit based on the second success of the second matching operation; The initial telemetry metadata and the additional telemetry metadata are stripped from the network data unit by the last-hop device along the last hop of the data flow path to obtain the stripped network data unit. The last-hop device uses the initial telemetry metadata stripped from the network data unit and the additional telemetry metadata to generate a telemetry metadata report; as well as The telemetry metadata report is provided to the application collector running on the second node.

2. The computer-implemented method of claim 1, wherein the last-hop device includes the second NIC of the second node.

3. The computer-implemented method of claim 1, wherein the last-hop device comprises a second network device along the data flow path.

4. The computer-implemented method of claim 1, wherein the application-specific information further identifies a portion of the application.

5. The computer-implemented method according to claim 1, further comprising: The stripped network data unit is sent from the last hop device along the data flow path toward the application receiver.

6. The computer-implemented method according to claim 1, wherein: The network data unit is identified as being executed as part of the matching action rules, and The matching action rule specifies that only a portion of the network data units that have successfully completed the first matching operation are inserted with the corresponding initial telemetry metadata.

7. The computer-implemented method according to claim 1, wherein: The network data unit is identified as being executed as part of the matching action rules, and The matching action rule specifies that no network data unit below the size threshold has been inserted with the initial telemetry metadata.

8. A non-transitory computer-readable medium containing a stored program, the stored program being executed by one or more processors, the program comprising instructions for: The network interface card (NIC) of the first node, executed by the application transmitter of the application, identifies a network data unit for telemetry metadata insertion. The identification is based on performing a first matching operation on information associated with the network data unit, wherein the network data unit is sent from the application transmitter of the first node to the application receiver of the application executed on the second node. The NIC inserts initial telemetry metadata into the network data unit based on the first success of the first matching operation. This initial telemetry metadata includes network information and application-specific information corresponding to the application. The application-specific information identifies the application; The initial telemetry metadata is stripped from the network data unit by the last hop device of the last hop along the data flow path of the network data unit to obtain the stripped network data unit; The last-hop device uses the initial telemetry metadata stripped from the network data unit to generate a telemetry metadata report; as well as The telemetry metadata report is provided to the application collector running on the second node.

9. The non-transitory computer-readable medium of claim 8, wherein the last-hop device comprises the second NIC of the second node.

10. The non-transitory computer-readable medium of claim 8, wherein the last-hop device comprises a second network device along the data flow path.

11. The non-transitory computer-readable medium according to claim 8, wherein: The program includes additional instructions for the following: The second matching operation is performed by a network device that hops along the data flow path of the network data unit using at least one of the information associated with the network data unit and the initial telemetry metadata; as well as The network device inserts additional telemetry metadata into the network data unit based on the second success of the second matching operation. The last-hop device also strips the additional telemetry metadata from the network data unit, and The additional telemetry metadata is also used to generate the telemetry metadata report.

12. The non-transitory computer-readable medium of claim 8, wherein the application-specific information further identifies a portion of the application.

13. The non-transitory computer-readable medium according to claim 8, further comprising: The stripped network data unit is sent from the last hop device along the data flow path toward the application receiver.

14. The non-transitory computer-readable medium according to claim 8, wherein: The network data unit is identified as being executed as part of the matching action rules, and The matching action rule specifies that only a portion of the network data units that have successfully completed the first matching operation are inserted with the corresponding initial telemetry metadata.

15. The non-transitory computer-readable medium according to claim 8, wherein: The network data unit is identified as being executed as part of the matching action rules, and The matching action rule specifies that no network data unit below the size threshold has been inserted with the initial telemetry metadata.

16. A network device, comprising: One or more processors; as well as One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media storing instructions, the instructions causing the one or more processors to: Receive network data unit, the network data unit including initial telemetry metadata inserted into the network data unit by the network interface card (NIC) of the first node executed on it by the application transmitter of the application, wherein: The initial telemetry metadata includes network information and application-specific information corresponding to the application. The application-specific information identifies the application. The network device is a device that follows the data flow path of the network data unit, and The network data unit is intended for use as an application receiver executed on the second node; The matching operation is performed using at least one of the information associated with the network data unit and the initial telemetry metadata. Based on the second success of the matching operation, additional telemetry metadata is inserted into the network data unit; and The network data unit is sent to the last-hop device along the data flow path of the network data unit.

17. The network device of claim 16, wherein the application-specific information further identifies a portion of the application.

18. A system comprising: The network device according to claim 16; The NIC of the first node; as well as The last hop device. The last-hop device is configured as follows: The initial telemetry metadata and the additional telemetry metadata are stripped from the network data unit to obtain the stripped network data unit; A telemetry metadata report is generated using the initial telemetry metadata stripped from the network data unit and the additional telemetry metadata. as well as The telemetry metadata report is provided to the application collector running on the second node.

19. The system of claim 18, wherein the last-hop device is one of the following: a second NIC on the second node, or a second network device along the data flow path.

20. The system of claim 18, wherein the last-hop device is further configured to: The stripped network data unit is sent to the application receiver along the data flow path.

Images