A method and apparatus for network path tracing

Abstract

Techniques for optimizing techniques related to network path tracing and network delay measurement are described herein. Some techniques can include using IPv6 header options and / or the Segment Identifier field of the Segment List or TLVs of the Segment Routing Header as telemetry data carriers. These techniques can also include using SRv6 micro-segmentation (uSID) instructions to indicate to a node of a network that the node is to perform one or more path tracing actions and encapsulate and forward a packet. Additionally, these techniques can include using short interface identifiers corresponding to node interfaces to trace packet paths through a network. Moreover, these techniques can include using short time stamps to determine delay measurements associated with sending packets through a network. In various examples, the above and herein described techniques can be used together with one another to optimize network path tracing and delay measurement techniques.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 344,443, filed June 10, 2021, which claims priority to U.S. Provisional Patent Applications Nos. 63 / 119,938, 63 / 119,964, 63 / 119,992, and 63 / 120,002, filed December 1, 2020. The entire contents of each of the above-listed provisional applications are incorporated herein by reference and used for all purposes. Technical Field

[0003] This disclosure generally relates to improved network path tracing and latency measurement techniques. Background Technology

[0004] Path tracing solutions and data plane monitoring technologies can improve network operators' visibility into their underlying networks. These solutions collect various information associated with nodes as packets traverse them, such as device identifiers, port identifiers, etc., from each node in the traffic flow. As packets cross the network, the collected information can be propagated along with them as telemetry data and can be used to determine the actual path the packets take through the network. Conventional network telemetry solutions typically use any number of bits, from 24 to 128, to represent device or node identifiers (e.g., IP addresses, segment identifiers, etc.), and 16 bits to represent the ingress / egress ports of devices or nodes.

[0005] Similarly, delay measurement solutions allow network operators to characterize the latency experienced by packets traversing the network. These delay measurement solutions can provide end-to-end and per-hop delay measurements. Traditional delay measurement solutions that provide end-to-end and per-hop delay measurements typically require collecting timestamps from each node that receives and / or forwards packets through the network. As above, the collected timestamps can also be transmitted with the packets as telemetry data as they traverse the network. However, these traditional delay measurement solutions typically use a standard timestamp format with a 64-bit structure, where the 32 most significant bytes represent the seconds since the epoch, and the 32 least significant bits represent the fraction of a second accurate to nanoseconds or picoseconds.

[0006] However, these traditional network telemetry and delay measurement solutions use far more bits than actually required. Furthermore, telemetry data is typically collected from each node receiving and / or forwarding packets, leading to numerous problems such as significant overhead, exceeding device or node hardware limitations, increased network utilization, packet loss at least in part due to large Maximum Transmission Unit (MTU) sizes, and so on. Moreover, the traditional solutions for carrying the aforementioned collected information and timestamps as telemetry data within packets may differ across different types of network equipment. Attached Figure Description

[0007] Detailed description is given below with reference to the accompanying drawings. In the drawings, the leftmost number(s) of the reference numerals indicate the drawing in which the reference numeral first appears. The same reference numerals are used in different drawings to denote similar or identical items. The systems depicted in the drawings are not drawn to scale, and the components in the drawings may not be drawn to scale relative to each other.

[0008] Figure 1 A schematic diagram of an example system architecture for the network is shown, in which the techniques described in this paper can be implemented.

[0009] Figure 2A and Figure 2B This is a block diagram illustrating example grouping structures that can be used to carry telemetry data and / or program nodes to perform path tracking actions.

[0010] Figure 3A This is a block diagram illustrating an example telemetry data carrier that is being used to carry compressed midpoint data (CMD) packets.

[0011] Figure 3B An example 64-bit full timestamp is shown. The exemplary short timestamps described herein may include one or more bits of the full timestamp, and Figure 3B The short timestamp shown is an 8-bit timestamp.

[0012] Figure 4A and 4B Example traffic flows between network nodes and / or devices are shown together to perform some of the techniques described in this paper for optimizing telemetry data collection.

[0013] Figure 5 This is a logic flowchart illustrating an example method that can be performed at least partially by network nodes to append telemetry data to packets.

[0014] Figure 6 This is a logic flowchart illustrating an example method that can be at least partially executed by a network node to perform one or more path tracing actions.

[0015] Figure 7This is a logic flowchart illustrating an example method that can be executed, at least in part, by the network's controller or collector to determine the path for a group to traverse the network.

[0016] Figure 8 This is a logic flowchart illustrating an exemplary method that can be executed, at least in part, by a network controller or collector to determine latency measurements associated with sending packets through the network.

[0017] Figure 9 This is a system architecture diagram of an example data center configuration that can be used to implement various aspects of the technologies disclosed herein.

[0018] Figure 10 This is a computer architecture diagram, illustrating example computer hardware architectures of nodes and / or computing devices that can be used to implement various aspects of the technologies proposed in this paper. Detailed Implementation

[0019] Overview

[0020] Various aspects of the invention are set forth in the independent claims, and preferred features are set forth in the dependent claims. A feature of one aspect may be applied to each aspect alone or in combination with other aspects.

[0021] The systems and methods described in this disclosure, among others, improve upon techniques related to network path tracing and network latency measurement. By way of example and not limitation, methods according to the various techniques described in this disclosure may include receiving, at a first node in the network, a packet to be sent to at least a second node in the network. In various examples, the packet may include a first indication indicating a specific type of telemetry data (e.g., interface identification data, timestamp data, node identification data, etc.) to which the first node will append (e.g., insert, update, etc.) to the packet. The packet may also include a second indication indicating a first position within the Segment Identifier (SID) list of the packet's Segmented Routing Header, at which the first node will append a specific type of telemetry data. Additionally or alternatively, the packet may include an indication of a position within the Hop-by-Hop (HbH) Options header of the packet, at which the first node will append a specific type of telemetry data. Additionally or alternatively, the packet may include an indication of a position within the TLV of the Segmented Routing Header (SRH) of the packet, at which the first node will append a specific type of telemetry data. The method may further include determining, by the first node and at least in part, based on a first indication, first telemetry data of a specific type associated with the first node. Furthermore, the method may include, by the first node and at least in part, appending the first telemetry data to a first location within the SID list or IPv6 hop-by-hop options and / or TLV of the packet's SRH, at a location. In some examples, the first node may modify the second indication of the packet to indicate a second location within the SID list, whereby the second node will append, at that location, the second telemetry data of a specific type associated with the second node or the hop-by-hop options and / or TLV of the packet's SRH. Additionally, the first node may send a packet to the second node comprising the first telemetry data and a second indication indicating the second location. In some examples, the first node may also send an indication of the format of the telemetry data.

[0022] In some examples, the method may include receiving a packet at a second node in the network, and the second node determining that the packet includes a segmented route containing an IPv6 micro-segmentation (uSID) instruction configured to prompt the second node to perform a path tracing action. Therefore, the method may also include the second node performing the path tracing action at least in part based on reading the SRv6 uSID instruction. In some examples, the path tracing action includes determining a specific type of second telemetry data associated with the second node and appending (inserting or updating) the second telemetry data at a second location within the packet's SRH's SID list or hop-by-hop option and / or TLV. For example, the path tracing action may include determining timestamp data or interface identification data and / or appending timestamp data or interface identification data to the packet, adding a new encapsulation to the packet, and sending the packet to a network controller or collector configured to perform Operations, Administration, and Maintenance (OAM) actions.

[0023] In some examples, the method may include receiving telemetry data at a controller or collector. In some examples, the telemetry data may include a first interface identifier associated with a first node in the network. The first interface identifier may indicate a first interface of the first node used to receive packets and / or send packets to a second node. In some examples, the telemetry data may also include a second interface identifier associated with a second node in the network. The second interface identifier may indicate a second interface of the second node used to receive packets from and / or send packets from the first node. The method may also include determining, at least in part, a path through the network on which packets were sent, based on the telemetry data, the path including at least the first node and the second node. Based at least in part on the path, the method may include determining whether an anomaly exists in the network.

[0024] In some examples, the telemetry data may additionally or alternatively include a first full timestamp indicating the first time the packet was processed by a first node (e.g., received, transmitted, modified, etc.). Furthermore, the telemetry data may also include a short timestamp representing a portion of a second full timestamp, indicating a second time the packet was processed by a second node. The method may also include determining the second full timestamp based at least in part on the first full timestamp and the short timestamp, and calculating the delay associated with the transmission of the packet through the network based at least in part on the first and second full timestamps.

[0025] Additionally, the techniques described herein can be implemented as a method and / or by a system having a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, perform the techniques described herein.

[0026] Example Implementation

[0027] As mentioned above, path tracing solutions and data plane monitoring techniques can improve network operators' visibility into their underlying networks. However, the number of bits used to represent critical path tracing and latency measurement data (e.g., timestamps, device / node identifiers, etc.) can be much higher than may actually be needed. Because telemetry data is typically collected from each node receiving and / or forwarding packets, existing techniques are often associated with a number of problems, such as significant overhead, exceeding device or node hardware limitations, increased network utilization, packet loss at least in part based on large maximum transmission unit (MTU) sizes, and so on. Furthermore, conventional solutions for carrying the aforementioned collected information and timestamps as telemetry data within packets may differ across different types of network devices, thus requiring special programming in certain networks to implement the desired techniques.

[0028] Accordingly, this disclosure relates to various techniques for improving path tracing and delay measurement solutions. One aspect of the various techniques disclosed herein includes using short (e.g., 8-bit, 12-bit, etc.) interface identifiers (IDs) to trace packet paths through a network. Using short interface IDs addresses many root causes related to excessive telemetry data overhead and processing efficiency. Compared to the conventional path tracing and data plane monitoring techniques described herein, the short interface ID solution allows packet paths to be traced by collecting only short IDs, for example, the input interfaces that receive packets at each node or the output interfaces that send packets at each node. Therefore, using this technique may eliminate the need to collect conventional long (e.g., 32-bit, 128-bit, etc.) node / switch identifiers. Furthermore, the overhead required for collecting telemetry data is less, and hardware complexity can be simplified in terms of the depth of the packet header (where telemetry data will be read and written by hardware).

[0029] In some examples, the network's centralized controller may have the network's topology information, and each node in the network may output its configuration to the centralized controller. The controller, with a complete view of the network topology, can then assign an identifier (interface ID) to each node, corresponding to each interface of the node. For example, if a single node in the network has 10 different communication interfaces, that node could receive 10 interface IDs; that is, one interface ID is used to identify each interface. This could also include assigning interface IDs to nodes capable of path tracing, which would allow the collector / analyzer / controller to identify hidden nodes (e.g., in brownfield networks).

[0030] In various examples, the number of bits required to represent a single interface ID can be capped at the maximum number of interfaces on the router. This ensures that the interface ID assigned to an interface is unique within its own router / node. In some examples, the maximum number of interfaces for any given router is equal to 2.n For this network, the controller will need at most n bits to represent the interface ID. In at least one example, a 3-tuple (iPE_OUT, ePE_IN, ...) is used.<ID_seq> The path tracking packet is uniquely identified by the following: iPE_OUT is the interface ID of the outgoing interface of the ingress provider edge node (source node) that sends the path tracking packet, ePE_IN is the interface ID of the incoming interface of the egress provider edge node (aggregation node) that receives the path tracking packet, and ID_seq is a sequence of interface IDs collected from any intermediate node and representing any intermediate node that forwards the packet from the ingress provider edge node to the egress provider edge node.

[0031] Another aspect of this disclosure includes a technique for determining network latency measurements using short (e.g., 8 bits, etc.) hardware timestamps. Similar to using short interface identifiers, short timestamps can be collected at each node to address many root causes associated with excessive telemetry data overhead and processing efficiency. Compared to existing latency measurement techniques, this solution provides a mechanism for recording short timestamps at each intermediate node forwarding packets. Furthermore, this mechanism is universal and can be adapted based on network type, required latency accuracy, or the maximum number of bits available for recording timestamps. This solution results in smaller telemetry data (e.g., 8-bit timestamps versus, for example, 64-bit timestamps) while still maintaining the timestamp accuracy requirements of various network types.

[0032] Traditional telemetry solutions typically record a full 64-bit timestamp as telemetry data appended to the packet at each hop. However, given the short lifespan of packets in the network, the 64 bits recorded at each node contain many redundant bits. Furthermore, considering the required time precision, the 64-bit timestamp also contains unnecessary bits. For example, in a network where the packet lifespan is less than 1 second and the required time precision is in microseconds, the 32-bit seconds of the timestamp encoded by each hop are identical and redundant, and the bits encoding the microsecond fraction are also unnecessary. In other words, in the example above, more than two-thirds of the 64 bits are unnecessary.

[0033] However, in some examples, according to the short timestamp technique described herein, in networks where the packet lifetime is greater than one second, the short timestamp may include several bits from the least significant bits of the second portion of the full timestamp, with the remaining bits obtained from the fractional portion of the full timestamp. On the other hand, in networks where the packet lifetime is less than one second, the short timestamp may include bits only from the fractional portion. When determining which bits of the full timestamp should be selected for the short timestamp, the number of bits and their chosen positions give rise to the flip and precision. For example, the choice of bit positions must be made such that the flip is greater than the one-way delay between the source and sink nodes, and / or that the flip is greater than the one-way delay between any two timestamp nodes in the network.

[0034] When a packet enters the network at the source node, the source node can record a full timestamp as the telemetry data to be carried in the packet. As the packet traverses the network, each intermediate (e.g., relay) node, and in some cases, the sink node, can record a short timestamp as telemetry data. The sink node can also record a full timestamp (instead of a short timestamp) as the telemetry data to be carried in the packet. Thus, the controller or collector can calculate the full timestamp for each intermediate node. This can be done by the controller or collector by combining (e.g., adding) the recorded short timestamps with the most significant byte of the full timestamp determined for the previous node. For the first intermediate node, the most significant byte of the full timestamp can be taken from the timestamp of the source node.

[0035] Another aspect of this disclosure includes techniques for packet telemetry carriers used in different networks and / or different hardware. A first solution involves using an IPv6 hop-by-hop (HBH) header, which immediately follows the IPv6 header and precedes the fragmentation routing header (SRH). These techniques define new HBH option types for carrying telemetry data. However, some hardware cannot handle HBH+SRH, and such hardware assumes the SRH is the next header after the IPv6 header. A second solution involves using an SRH TLV, which is located at the end of the SRH. These techniques define new SRH TLVs with space for carrying telemetry data.

[0036] Another technique disclosed herein includes a telemetry data carrier for SRv6 networks, wherein the telemetry carrier can be added as a new SID immediately following the basic SRH header. The new SID can be used solely for telemetry collection purposes and can not affect any SRv6 deployment services. Compared to the existing solutions described above, the technique described herein for using a list of SIDs as a telemetry carrier allows packets to carry telemetry data as part of the SRH, rather than being very deep within the packet due to its earlier position. Furthermore, the disclosed technique avoids having an HBH header between the IPv6 header and the SRH header, thus addressing the issue of hardware inability to handle packets with HBH+SRH headers. Moreover, since the telemetry carrier is encoded immediately following the SRH-based HDR, telemetry data collection has no impact on hardware (which is limited by the depth they can write to packets) and is transparent to SRv6 traffic engineering (TE) / endpoint nodes. In summary, the disclosed technique provides an improved method for collecting telemetry data that works on hardware with limited write depth in packets and hardware that cannot handle HBH. The telemetry data carrier described in this article can be used to carry the short timestamps and short interface IDs described in this article, as well as other types of telemetry data.

[0037] Another aspect of this disclosure includes techniques for defining SRv6 micro-segment identifier (uSID) network programming instructions for path tracing actions on a router. In some examples, a node can trigger a path tracing action by specifying a particular path tracing action in the next SRv6 uSID network programming path tracing instruction. This SRv6 uSID path tracing instruction can be a known value, globally defined within the network across a domain, and / or programmed on each node. This SRv6 uSID instruction can be a local instruction on the node and advertised to the network. For example, a first SRv6 uSID path tracing instruction may prompt the node to insert a short or full timestamp, a short interface ID, and encapsulate the packet with an SR policy / IPv6 header, including IPv6 hop-by-hop options and / or SRH, where the TLV contains additional telemetry information on the node. Additionally, a second SRv6 uSID path tracing instruction may prompt the node to clone the packet, insert a short or full timestamp, a short interface ID, and encapsulate the packet with an SR policy / IPv6 header, including IPv6 hop-by-hop options and / or SRH, where the TLV contains additional telemetry information on the node. These SRv6 uSID path tracing instructions can be enabled on any intermediate node and the sink node. These behaviors defined by the SRv6 uSID instructions can also be triggered on the node by specific hop-by-hop option types, SRH TLV types, or other indications in the header, or by using local policies on the node.

[0038] As an example of using these SRv6 uSID path tracing instructions, the controller can request the source node of the network to generate a path tracing probe message, and the controller can also provide the probe endpoint address in the form of an SRv6 uSID carrier. Additionally, the controller can provide the source address, DSCP, packet size, and flow label value range for ECMP tracing of the customer data traffic. The source node of the network can then create one or more IPv6 path tracing probe messages containing hop-by-hop options carrying a short interface ID and / or a short timestamp, as well as short timestamp information, and then insert its transmitted short timestamp or full timestamp and short outgoing interface identifier SRH metadata when sending the message. The source node can then send (one or more) path tracing probe messages and scan the flow label range in the IPv6 header to traverse all ECMPs to reach the sink node. It can also scan a range of DSCP values. In some examples, each intermediate node can insert its incoming or outgoing short interface ID and its received or transmitted short timestamp into the hop-by-hop options of the path tracing message, or insert a segmentation identifier into the SRH of the message or the SRH TLV of the message, and then forward the message downstream. Then, the aggregation node can add its receiving and / or sending short timestamps or full timestamps, short interface identifiers, add new encapsulation with SRv6 / IPv6 headers, optionally add new hop-by-hop options and / or new SRH TLVs, and forward path tracing messages to the controller based on SRv6 uSID network programming instructions.

[0039] Improvements to computer-related technologies can be achieved according to the techniques described herein. For example, less overhead may be required by using short interface identifiers and short timestamps for packet path tracing and determining network latency measurements. Furthermore, hardware complexity can be simplified in terms of the depth of the packet header (in which telemetry data is written by hardware), allowing different types of hardware devices in SRv6 networks to perform path tracing and network latency measurements that would otherwise be incapable of doing so. The above discussion is merely one example of the many improvements that can be achieved according to the techniques described in this disclosure. These and other improvements will be readily understood and recognized by those skilled in the art.

[0040] By way of example and not limitation, methods according to the various techniques described in this disclosure may include receiving packets at a first node of the network to be sent to at least a second node of the network. In various examples, the packet may include a first indication indicating that the first node is to append (insert or update) a specific type of telemetry data (e.g., short interface ID, short timestamp, full timestamp, node identification data, etc.) to the packet. In at least one example, the packet may also include a second indication indicating a first position in the fragmentation identifier (SID) list of the packet's fragmentation routing header at which the first node is to append (insert or update) a specific type of telemetry data. In another example, the node may append a specific type of telemetry data to the packet's hop-by-hop options. In another example, the node may append a specific type of telemetry data to the packet's SRH TLV. In some examples, the first and / or second indications may be included within a first field (e.g., a tagging field) of the packet's fragmentation routing header. In some examples, the first and / or second indications may be included in the packet's IPv6 hop-by-hop options. In some examples, the first and / or second indications may be included in the TLV of the packet's SRH. In at least one example, the first node is the source node, while in another example, the first node is an intermediate node.

[0041] In some examples, the method may also include determining, by the first node, a first type of first telemetry data associated with the first node. In some cases, the specific type of telemetry data may include one or more of the following: a short interface ID indicating the interface of the first node used to send or receive packets; timestamp data indicating the time when the packet was processed by the first node (e.g., sent, received, modified, etc.); or short timestamp data indicating a portion of a complete timestamp associated with the time the packet was processed by the first node. In some examples, the first node may determine the first telemetry data based at least in part on a first indication. For example, the first node may determine that a flag field in a segmented routing header is enabled, and based at least in part on the enabled flag field, the first node may read a first indication and a second indication. In at least one example, the first and second indications may be included in the same field of the packet's fragmented routing header (e.g., included in a 16-bit tag field, where the first 8 bits of the 16-bit tag field represent the first indication and the last 8 bits represent the second indication), or indicated in IPv6 hop-by-hop options (e.g., the first 16 bits of the options data), or indicated in an SRH TLV (e.g., the first 16 bits of the TLV data) field. The indication may include the location, format, and type of the telemetry data to be appended.

[0042] Additionally, in some examples, the method may include the first node appending first telemetry data at a first position within the SID list. For example, based at least in part on a second indication, the first node may determine the first position in the SID list where the first telemetry data is to be appended. In some examples, the first position may correspond to an 8-bit position within the first SID field of the SID list. Additionally or alternatively, the second indication of the position may instruct the first node to write the telemetry data to an n-bit offset in the first SID field of the SID list, where "n" represents the number of bits the telemetry data will be offset by. In some examples, for these purposes, the IPv6 hop-by-hop option is used instead of the SID list. Moreover, the hop-by-hop option is provided due to its flexibility. In some other examples, for these purposes, SRH TLV is used instead of the SID list.

[0043] In some examples, the first node can modify the second indication of a packet to indicate a second location within the SID list where the second node will append a second type of telemetry data associated with the second node. That is, for example, after appending the first telemetry data, the first node can indicate the position in the SID list where the second node will begin appending telemetry data. As mentioned above, the second location can be within the first SID field of the SID list. Alternatively, the first node can send a packet to the second node containing the first telemetry data and the second indication of the second location. In some examples, for these purposes, the IPv6 hop-by-hop option is used instead of the SID list. Moreover, the hop-by-hop option is provided due to its flexibility. In some other examples, for these purposes, SRH TLV is used instead of the SID list.

[0044] In various examples, the second node may receive packets at least partially based on a first indication that identifies a specific type of second telemetry data associated with the second node. Additionally, the second node may append second telemetry data at a second location within the SID list, at least partially based on a second indication pointing to a second location. In at least one example, the second node may be an intermediate node, and the second node may modify the second indication of the packet to indicate a third location within the SID list, where the third node will append a specific type of third telemetry data associated with the third node at the third location. Furthermore, the second node may send packets to the third node, including the first telemetry data, the second telemetry data, and the second indication pointing to the third location. In some examples, for these purposes, the IPv6 hop-by-hop option is used instead of the SID list. Moreover, the hop-by-hop option is provided due to its flexibility. In some other examples, for these purposes, SRH TLV is used instead of the SID list.

[0045] In some examples, the second node may be a network aggregation node (e.g., an egress provider edge node), and the method may include the second node determining that the packet includes an SRv6 micro-segmentation (uSID) instruction. The SRv6 uSID instruction may be associated with one or more path tracing instructions. In some examples, the SRv6 uSID instruction may be included within the packet's destination address field (e.g., within the fragmentation list of the packet's fragmentation routing header, within the packet's IPv6 destination address field, etc.). In some cases, the second node may determine that the packet includes an SRv6 uSID instruction by shifting the packet's destination address field to determine the next SRv6 uSID instruction. That is, the destination address field may be used to carry the SRv6 uSID instruction (associated with a path tracing action) and one or more other SRv6 uSID instructions (indicating the next segment to which the packet should be sent). Therefore, the second node can by default shift the destination address field to determine the next SRv6 uSID instruction (e.g., the next SRv6 uSID indicates the next segment to which the packet should be sent), but the next uSID instruction can be associated with a path tracking action rather than with the packet's next hop.

[0046] In some examples, the SRv6 uSID instruction can be configured to prompt a second node to perform one or more path tracing actions. Therefore, the method may also include the second node performing path tracing actions at least in part based on reading the SRv6 uSID instruction. In at least one example, a path tracing action may include determining a specific type of second telemetry data associated with the second node and appending the second telemetry data at a second position within the SID list. For example, one path tracing action may include determining and / or appending timestamp data (e.g., short timestamp, full timestamp, etc.) indicating the time when the second node received the packet. Additionally or alternatively, a path tracing action may include appending interface identification data to the packet, indicating the interface of the second node used to receive the packet. Another path tracing action may include adding a new encapsulation on top of an existing header and its TLV extensions and sending the packet to a network controller or collector configured to perform Operation, Administration, and Maintenance (OAM) actions (e.g., tracing the path the packet takes to traverse the network, determining the latency associated with sending the packet through the network, etc.).

[0047] In some examples, the SRv6 uSID instruction can instruct a second node to clone packets and send the cloned packets to the network's controller or collector. The cloned packets may include telemetry data appended by both the first and second nodes. In some examples, the second node may add new encapsulation on top of an existing header including its TLV to send packets containing telemetry data to the network's controller or collector.

[0048] In various examples, the method may include receiving telemetry data at a controller or collector. In some examples, the telemetry data may include multiple interface identifiers (e.g., short interface IDs) that indicate the interfaces of network nodes that send or receive packets. For example, the telemetry data may include a first interface identifier associated with a first node of the network. The first interface identifier may indicate a first interface of the first node used to receive packets and / or send packets to a second node. In some examples, the telemetry data may also include a second interface identifier associated with a second node of the network. The second interface identifier may indicate a second interface of the second node used to receive packets from and / or send packets to the first node.

[0049] In some examples, the controller or collector may determine a path through the network, at least in part, based on telemetry data, according to which packets are sent, and the path includes at least a first node and a second node. For example, the path may include an indication of each node in the network that sends or receives packets. In some cases, the controller or collector may receive topology data associated with the network and determine the path, at least in part, based on the topology data. Alternatively, the controller or collector may determine the path, at least in part, based on a 3-tuple including a first interface identifier, a second interface identifier, and one or more interface identifiers indicating a series of intermediate nodes between the first and second nodes that forward packets.

[0050] At least in part based on paths, the method may include determining whether anomalies exist in the network. For example, a controller or collector may determine whether anomalies exist in the network. In some cases, anomalies may include: the path is an incorrect shortest path through the network, the path is an incorrect equal-cost multipath (ECMP) route, the network includes faulty links, faulty nodes, etc., and / or similar content.

[0051] In some examples, the telemetry data may additionally or alternatively include a first full timestamp indicating a first time when the packet was processed by the first node (e.g., received, transmitted, modified, etc.). Additionally, the telemetry data may include a short timestamp representing a portion of a second full timestamp, which indicates a second time when the packet was processed by the second node. The method may also include determining the second full timestamp based at least in part on the first full timestamp and the short timestamp, and calculating the delay associated with the transmission of the packet through the network based at least in part on the first and second full timestamps.

[0052] In some examples, telemetry data may include a first full timestamp representing the time when a packet is processed by the source node, and the telemetry data may also include multiple short timestamps representing the corresponding times when the packet is processed by other nodes in the network. The controller or collector may determine the full timestamp of each of the multiple short timestamps based at least in part on the first full timestamp and the other full timestamps determined from the short timestamps. In some cases, latency may include one or more of the following: link (or hop) delay between individual nodes in the network, end-to-end delay between the source and sink nodes of the network, etc.

[0053] Certain embodiments and implementations of this disclosure will now be described more fully below with reference to the accompanying drawings, in which various aspects are illustrated. However, these aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. As stated herein, this disclosure covers variations of the embodiments. The same reference numerals consistently denote similar elements.

[0054] Figure 1A schematic diagram of an example system architecture 100 for network 102 is shown, in which the technologies described herein can be implemented. Generally, network 102 may include devices housed or located in one or more data centers 104, which may be located in different physical locations. For example, network 102 may be supported by a network of devices from public cloud computing platforms, private / enterprise computing platforms, and / or any combination thereof. One or more data centers 104 may be physical facilities or buildings located in a geographic area, designated to store networked devices as part of network 102. Data centers 104 may include: various networked devices, and redundant or backup components and infrastructure for power supply, data communication connectivity, environmental control, and various security devices. In some examples, data center 104 may include one or more virtual data centers, which are pools or collections of cloud infrastructure resources specifically designed for enterprise needs and / or the needs of cloud-based service providers. Generally, (physical and / or virtual) data centers 104 can provide basic resources such as processors (CPU), memory (RAM), storage devices (disks), and networking (bandwidth). However, in some examples, the devices in network 102 may not be located in the explicitly defined data center 104, but may be located in other locations or buildings.

[0055] Network 102 may include one or more networks implemented using any feasible communication technology such as wired and / or wireless means and / or technologies. Network 102 may include any combination of centralized and / or distributed networks such as Personal Area Networks (PANs), Local Area Networks (LANs), Campus Area Networks (CANs), Metropolitan Area Networks (MANs), Extranets, Intranets, the Internet, Short-Range Wireless Communication Networks (e.g., ZigBee, Bluetooth, etc.), Virtual Private Networks (VPNs), Wide Area Networks (WANs), and / or may include any combination, arrangement, and / or aggregation of these. Network 102 may include devices, virtual resources, or other nodes that relay packets from one network segment to another.

[0056] Network 102 may include a control plane 106 and a data plane 108 (e.g., a network structure), or may otherwise be distributed (physically or logically) across the control plane 106 and the data plane 108 (e.g., a network architecture). The control plane 106 may include a controller 110, an application programming interface 112, one or more databases 114, and a collector 116. The data plane 108 may include one or more nodes (nodes 118(1) to 118(4), hereinafter collectively referred to as "node 118").

[0057] In some examples, controller 110 may assign different interface identifiers (IDs) to the corresponding interfaces of node 118. For example, controller 110 may assign a first interface ID to a first interface of node 118(1), a second interface ID to a second interface of node 118(1), a third interface ID to an interface of node 118(2), and so on. Controller 110 may send path tracing instructions 120 to node 118(1), which may be configured as a source node (e.g., a source provider edge node)), to instruct node 118(1) to initiate a telemetry collection session for path tracing and / or delay measurement using injected probes, in-band data traffic flow, or by sampling the traffic flow. In some examples, controller 110 may configure intermediate nodes 118(2) and 118(3) with some bits and specific bits of the full timestamp, which will be used to encode short timestamps in received path tracing packets 122. Controller 110 can also configure node 118 (4) (which can be configured as a convergence node (e.g., an egress provider edge node)) to report the collected telemetry data 124 to collector 116. In some examples, controller 110 can access the topology information of network 102, and node 118 can configure its output to controller 110.

[0058] exist Figure 1In this configuration, node 118(1) is configured as a source node (or headend node), such as an ingress provider edge router, a rack-top switch, a SmartNIC, etc. Source node 118(1) can receive path tracing instruction 120 from controller 110 and initiate a telemetry collection session. For example, source node 118(1) can allocate space in the telemetry collection packet / header of path tracing packet 122 for the number of hops specified in path tracing instruction 120. As part of allocating space in the telemetry collection packet / header of path tracing packet 122, source node 118(1) can include in the packet an indication of the type of telemetry data that downstream nodes 118(2)-118(4) should add to the packet, and a specific location in the packet where downstream nodes 118(2)-118(4) should attach the telemetry data. This minimizes the workload that downstream nodes 118(2)-118(4) need to perform, resulting in minimal latency and / or more accurate latency measurements. In some examples, source node 118(1) may record the short interface ID of its outgoing interface used to send path tracing packets 122. Alternatively or additionally, source node 118(1) may record its full timestamp, which can then be used by collector 116 to determine the full timestamps of downstream nodes 118(2)-118(4), which may be appended with their short timestamps. In some cases, source node 118(1) may record the interface ID and / or timestamp at the end of the telemetry data carrier, so that downstream nodes 118(2)-118(4) do not have to write such deep content in the packets to record their telemetry data.

[0059] In some examples, source node 118(1) can encapsulate path tracking packet 122 with a fragmented routing header, allowing telemetry data to be carried in the last fragment identifier field of the fragment list in the fragmented routing header. For example, source node 118(1) can encapsulate path tracking packet 122 with a fragmented routing header and set the value of the "T" flag in the fragmented routing header so that downstream nodes 118(2)-118(4) can read the tag field of the fragmented routing header. Additionally, source node 118(1) can encode the tag field of the fragmented routing header to indicate what type of telemetry data downstream nodes 118(2)-118(4) will encode, and the offset (fragment identifier field of the fragment list) that downstream nodes 118(2)-118(4) should write into the telemetry carrier of their telemetry data. The source node can encode the telemetry data in the IPv6 hop-by-hop option instead of in the fragment list in the SRH. In another example, the source node can encode telemetry data in a TLV within the SRH instead of in a segment list within the SRH.

[0060] Intermediate nodes 118(2) and 118(3) may receive path tracking packet 122. Furthermore, intermediate nodes 118(2) and 118(3) may record their telemetry data (e.g., short timestamps, short interface IDs, etc.) within path tracking packet 122. In some examples, intermediate nodes 118(2) and 118(3) may receive path tracking packet 122, read the “T” flag value of the segmented routing header, and, if “T” is enabled, read the tag field of the segmented routing header to determine what type of telemetry data to record in path tracking packet 122, and the location at which the telemetry data is recorded, indicated in the tag field. For example, the tag field may include a first indication (e.g., an “action” bit) indicating the type of telemetry data and a second indication (e.g., an “offset” bit) indicating the offset at which the telemetry data will be recorded within the telemetry carrier of path tracking packet 122. Intermediate nodes 118(2) and 118(3) may forward path tracking packet 122 downstream sequentially. Intermediate nodes can use indications (action / offset bits) from the IPv6 hop-by-hop options to encode telemetry data instead of encoding it in the segment list within the SRH. In another example, intermediate nodes can use indications (action / offset bits) from the TLV within the SRH to encode telemetry data within the TLV within the SRH, instead of encoding it in the segment list within the SRH.

[0061] In some examples, the sink node 118(4) (or the tail node) may receive path tracing packets 122 from intermediate node 118(3), record its own telemetry data (e.g., short timestamp, full timestamp, interface ID, etc.), and forward all telemetry data 124 (e.g., all telemetry data added by nodes 118(1)-118(4)) to collector 116. In some examples, if the sink node 118(4) determines upon receiving path tracing packets 122 that the “T” flag is enabled and the remaining segments are equal to 1, the sink node 118(4) may export the telemetry data 124 to collector 116, add a new segment routing header and IPv6 encapsulation, and / or forward the entire packet, including its header, to its destination (e.g., the collector). In some examples, path tracing packets 122 may include uSID instructions prompting the sink node 118(4) to perform some or all of these path tracing actions. In some cases, path tracing actions associated with a specific uSID instruction can be defined globally within network 102 and / or locally on individual nodes 118. The aggregation node 118 (4) can shift (e.g., shift by 16 bits) the destination address field of path tracing packet 122 to determine that path tracing packet 122 includes an SRv6 uSID instruction. For example, the SRv6 uSID instruction can be included in a uSID block of path tracing packet 122, the uSID block indicating the appropriate segment to which the packet will be sent.

[0062] In some examples, collector 116 of network 102 may receive telemetry data 124 and use the short interface IDs attached to path tracking packets 122 by each node 118 to determine the path taken by path tracking packets 122 through network 102. In some cases, in addition to short interface IDs, collector 116 may also use network topology information to determine the path. In some examples, collector 116 may also store the determined paths in one or more databases 114 for analysis by other components such as controller 110. For example, controller 110 may analyze the stored paths to detect anomalies in network 102, such as faulty short paths, faulty ECMPs, faulty links and / or nodes, etc.

[0063] Additionally or alternatively, collector 116 may determine the complete timestamp of each of intermediate nodes 118(2) and 118(3). The complete timestamp of each of intermediate nodes 118(2) and 118(3) may be determined by combining the recorded short timestamp with the most significant byte of the complete timestamp constructed at the previous node. For example, for the first intermediate node 118(2), the complete timestamp may be determined by combining (e.g., adding) the short timestamp recorded by the first intermediate node 118(2) with the most significant byte of the complete timestamp recorded by the source node 118(1).

[0064] An example algorithm for calculating a complete timestamp can be defined as follows: (1) where TS(k) is the N-bit complete timestamp of node k; (2) where TS_short(k) is the short timestamp collected at node k; and (3) for each node k in the recorded group path: (a) retrieve the position (P) and number of bits (M) of the recorded short timestamp at the configuration of node k, which is derived from the node configuration output to the controller; (b) TS(k)[N:P] = TS(k-1[N:P]; (c) TS(k)[P:PM] = TS_short(k); (d) TS(k)[PM:0] = 0. In the above algorithm, for the first node k in the recorded group path, node k-1 is the source node; in operation (b), the most significant byte is copied from the complete timestamp of the previous node; in operation (c), the short timestamp is copied from the path tracking group; and in operation (d), the least significant bit is set to zero.

[0065] In some examples, the collector 116, controller 110, or analyzer of network 102 can analyze the collected timestamps to determine various use cases. Some of these use cases may include calculating the latency of each link and using the latency of each link to calculate a path with a given latency SLA. Additionally or alternatively, use cases may include calculating edge-to-edge latency (e.g., the latency between source node 118(1) and sink node 118(4)) accurate to nanoseconds if both source node 118(1) and sink node 118(4) add their full timestamps to path tracking packet 122. Additionally or alternatively, the latency of a given path may be monitored based on the collected timestamps.

[0066] Figure 2A and 2BThis is a block diagram illustrating example packet structures that can be used to carry telemetry data and / or program nodes to perform path tracing actions. Packet 200 includes: an IPv6 header 204, a hop-by-hop (HBH) header 206 (which can be used to carry telemetry data), a fragmentation routing header 208, a fragmentation list (including one or more fragmentation list fields 210(1) to 210(N) capable of carrying a fragmentation identifier or uSID (where N represents any number greater than or equal to 1)), and a payload 212. The HBH header 206 may also include “Actions” and “Offsets”, as described in further detail below.

[0067] Packet 202 also includes: an IPv6 header 204, a segment routing header 208, a segment list (including one or more segment list fields 210(1) to 210(N)), and a payload 212. However, compared to packet 200, packet 202 includes a segment routing header (SRH) type length value (TLV) field 214 (which can be used as a telemetry carrier).

[0068] exist Figure 2BIn this packet 216, the packet also includes: an IPv6 header 204, a fragmentation routing header 208, a fragmentation list (including one or more fragmentation list fields 210(1) to 210(N)), and a payload 212. However, in packet 216, the last fragmentation list field 210(1) is used as a telemetry data carrier, while the other fragmentation list fields 210(2) to 210(N) are used as regular SID or SRv6 uSID carriers. The fragmentation routing header 208 may include one or more flag fields 218 that can be used to enable path tracing behavior. For example, by enabling the flag “T”, the node receiving packet 216 can recognize that telemetry data will be added to the packet and read the flag field 220 accordingly. The flag field 220 may be a 16-bit field, where 8 of the 16 bits are used to indicate “action” and the remaining 8 bits are used to indicate “offset”. As described herein, the action bit can indicate the type of telemetry data (e.g., short timestamp data, short interface ID, etc.) to which a node will append telemetry data to packet 216. The offset bit can indicate a specific position (e.g., the position of a bit) within the segment list [0] field 210 (1) of packet 216, where a node will append (insert or update) telemetry data. In some cases, after the first node appends telemetry data at the offset position within the segment list [0] field 210 (1), the first node can modify the offset bit to point to the next available position within the segment list [0] field 210 (1) (e.g., the position of the next bit), where the second node will begin appending telemetry data. When using the hop-by-hop option, the offset and telemetry data are carried in the hop-by-hop option. The offset field identifies the position where the second node appends telemetry data in the option.

[0069] Figure 3AThis is a block diagram illustrating an example telemetry data carrier portion 300 used to carry compressed midpoint data (CMD) packets. The compressed midpoint data may include a short interface identifier, a short timestamp, etc. In some examples, the corresponding rows of the telemetry data carrier 300 (e.g., the first row including CMD 302(1) through CMD 302(4)) may correspond to HBH fields, SRH TLV fields, segment list fields, etc. The telemetry data carrier 300 may carry CMD 302(1) through CMD 302(N) (where N represents any number greater than or equal to 1). CMD 302(1)–302(N) may correspond to interfaces of intermediate nodes, such as intermediate nodes 118(2) and 118(3). Additionally, the telemetry data carrier 300 may also carry a source node complete timestamp 304 and a sink node complete timestamp 306. These timestamps may be associated with the time when the packets were processed by the source and sink nodes (e.g., sent, received, etc.). In some examples, if CMD 302 represents a short interface ID, it means that the maximum number of bits required for the interface ID can be the maximum number of interfaces on the node / router. In this case, the interface ID assigned to an interface is unique within its own node / router. For example, the maximum number of interfaces on any given router is equal to 2. n In a network, the interface ID requires a maximum length of n bits to represent the interface ID. The interface ID may not need to be unique within the network or on a node.

[0070] Figure 3B An example 64-bit full timestamp 320 is shown. An exemplary short timestamp 322 described herein may include one or more bits 324 of the full timestamp 320, and Figure 3B The short timestamp 322 shown is an 8-bit timestamp. The 64-bit full timestamp 320 may include 32 bits representing the seconds portion 326 of the full timestamp 320 and 32 bits representing the nanosecond portion 328 of the full timestamp 320. The positioning of the short timestamp 322 relative to bit 324 can represent a short timestamp for 5G deployment. In some examples, the short timestamp 322 may be associated with a precision of 2.097 microseconds (as shown by the least significant bit of the short timestamp 322 with a value of 2.097 milliseconds) and a flip of 536.86 milliseconds (as shown by doubling the value of the most significant byte with a value of 268.43 milliseconds). In other words, the short timestamp 322 is likely optimal for use in networks where the latency between the source and sink nodes is less than 536.86 milliseconds and the latency between individual nodes is greater than 2.097 microseconds. In at least one example, different nodes in the network can choose different bits 324 of the full timestamp 320 for the short timestamp 322. In other words, the bit selection does not need to be the same for each node along a given path.

[0071] Figure 4A and 4B Example traffic flows between network nodes and / or devices are illustrated to demonstrate some of the techniques described herein for optimizing telemetry data collection. Figure 4A In the diagram, at "1", the network controller 110 can assign the interface identifier 402 to the corresponding interface of node 118. For example, controller 110 can assign a first interface ID to the first interface of node 118 (1), a second interface ID to the second interface of node 118 (1), a third interface ID to the interface of node 118 (2), and so on. In some examples, controller 110 can send the indication of the interface identifier 402 to node 118 at different times or substantially simultaneously. Controller 110 can assign the interface identifier 402 to node 118 based on the configuration data received associated with each node in node 118.

[0072] about Figure 4BAt “2”, controller 110 may send path tracing instruction 404 to source node 118(1) to instruct node 118(1) to initiate a telemetry collection session (for path tracing and / or delay measurement) using injected probes, striped communication streams, or by sampling the traffic stream. Source node 118(1) may receive path tracing instruction 404 from controller 110 and initiate a telemetry collection session. For example, source node 118(1) may allocate space in the telemetry collection packet / header of path tracing enable packet 406 for the number of hops specified in path tracing instruction 404. As part of allocating space in the telemetry collection packet / header of path tracing enable packet 404, source node 118(1) may include in the packet an indication of the type of telemetry data that downstream nodes 118(2)-118(4) should add to the packet, and the specific location in the packet where downstream nodes 118(2)-118(4) should attach telemetry data. For example, source node 118(1) may specify that intermediate node 118(2) shall append its telemetry data at a first location within telemetry data carrier 410(1), intermediate node 118(3) shall append its telemetry data at a second location within telemetry data carrier 410(2), and sink node 118(4) shall append its telemetry data at a third location within telemetry data carrier 410(3). This minimizes the workload of downstream nodes 118(2)-118(4), resulting in minimal latency and / or more accurate latency measurements. In some examples, source node 118(1) may record the short interface ID of its outgoing interface used to send path tracing enable packet 406 as telemetry data 408(1) within path tracing enable packet 406. Additionally or alternatively, source node 118(1) may record its full timestamp, which may then be used to determine the full timestamps of downstream nodes 118(2)-118(4), which may append their short timestamps. In some cases, the source node 118(1) can record the interface ID and / or timestamp at the end of the telemetry data carrier, so that the downstream nodes 118(2)-118(4) do not have to write so much content in the packet to record their telemetry data.

[0073] At position “3”, source node 118(1) may send a path tracking enable packet 406, including node 118(1) telemetry data 408(1), to intermediate node 118(2). Intermediate node 118(2) may receive the path tracking enable packet 406 and append its own telemetry data 408(2) to the telemetry data carrier field 410(1) of the path tracking enable packet 406. The telemetry data 408(2) of node 118(2) may include a short timestamp and / or a short interface identifier used by intermediate node 118(2) to receive or send the path tracking enable packet 406.

[0074] At position “4”, intermediate node 118(2) may forward a path tracking enable packet 406, which includes telemetry data 408(1) of node 118(1) and telemetry data 408(2) of node 118(2), to intermediate node 118(3). Intermediate node 118(3) may receive the path tracking enable packet 406 and append its own telemetry data 408(3) to the telemetry data carrier field 410(2) of the path tracking enable packet 406. The telemetry data 408(3) of node 118(3) may include a short timestamp and / or a short interface identifier used by intermediate node 118(3) to receive or send the path tracking enable packet 406.

[0075] At “5”, intermediate node 118(3) can forward a path tracking enable packet 406, which includes telemetry data 408(1) of node 118(1), telemetry data 408(2) of node 118(2), and telemetry data 408(3) of node 118(3), to sink node 118(4). Sink node 118(4) can receive path tracking enable packet 406 and append its own telemetry data 408(4) to the telemetry data carrier field 410(3) of path tracking enable packet 406. The telemetry data 408(4) of node 118(4) may include a full timestamp and / or short interface identifier used by sink node 118(4) to receive path tracking enable packet 406.

[0076] At position “6”, the aggregation node 118(4) can send all telemetry data 410 to the controller 110. Telemetry data 410 may include telemetry data 408(1) of node 118(1), telemetry data 408(2) of node 118(2), telemetry data 408(3) of node 118(3), and telemetry data 408(4) of node 118(4). In some cases, the aggregation node 118(4) may send telemetry data 410 to the controller 110 when it detects that a path tracing enable packet 406 includes an SRv6 uSID instruction associated with one or more path tracing actions. One or more path tracing actions may be defined globally within the network or locally on each node 118. A path tracing action may include sending telemetry data 410 to the controller 110. Alternatively, a path tracing action may include copying the path tracing enable packet 406 and sending the copied packet, which includes telemetry data 410, to the controller 110. In some examples, node 118 (4) can encapsulate telemetry data 410 to send it to controller 110.

[0077] Figures 5 to 8Logic flowcharts are shown for various example methods associated with the techniques presented herein for optimizing telemetry data collection for path tracking and delay measurement techniques. This document references... Figures 5 to 8 The described logical operations can be implemented as (1) a series of computer-implemented actions or program modules running on a computing system and / or (2) interconnected machine logic circuits or circuit modules within a computing system.

[0078] The implementation of the various components described herein depends on the performance of the computing system and other requirements. Accordingly, the logical operations described herein are referred to as operations, structured devices, actions, or modules. These operations, structured devices, actions, and modules can be implemented in software, firmware, dedicated digital logic, and any combination thereof. It should also be understood that more advanced technologies can be implemented... Figures 5-8 The operations shown and described herein may be more or fewer. These operations may also be performed in parallel or in a different order than those described herein. Some or all of these operations may also be performed by components other than those specifically identified. While the techniques described in this disclosure refer to specific components, in other examples, these techniques may be implemented by fewer components, more components, different components, or any configuration of components.

[0079] Figure 5 This is a logical flowchart illustrating an example method 500 that can be performed at least partially by a network node to append telemetry data to a packet. Method 500 begins at operation 502, which includes receiving, at a first node in the network, a packet to be sent to at least a second node in the network. The packet includes (i) a first indication specifying a particular type of telemetry data to be appended to the packet by the first node, and (ii) a second indication specifying a first position within the Segment Identifier (SID) list of the segmented routing header of the packet to which the first node intends to append the particular type of telemetry data. For example, the first and second indications may be included within a tag field 220 of the segmented routing header 208. The first indication may indicate an "action," and the second indication may indicate an "offset."

[0080] At operation 504, method 500 includes determining, by the first node and at least in part based on a first indication, first type of first telemetry data associated with the first node. For example, the first node may determine a short timestamp that includes a portion (e.g., one or more individual bits) of a full timestamp representing the time when the packet was processed by the first node. Additionally or alternatively, the first node may determine a short interface ID that indicates the interface of the first node used to receive the packet.

[0081] At operation 506, method 500 includes appending (e.g., inserting, updating, etc.) first telemetry data at a first position within the SID list by the first node and at least in part based on a second indication. For example, the first node may append a short interface ID and / or a short timestamp at the first position within the SID list.

[0082] At operation 508, method 500 includes modifying a second indication of a packet by the first node to indicate a second location within a SID list, at which the second node will append second telemetry data of a specific type associated with the second node. At operation 510, method 500 may include sending a packet to the second node, the packet including the first telemetry data and the second indication indicating the second location.

[0083] Figure 6 This is a logical flowchart illustrating an example method 600 that can be at least partially executed by network nodes to perform one or more path tracing actions. Method 600 begins with operation 602, which includes receiving a packet at a first node in the network. For example, the packet may be received by the aggregation node 118(4) of network 102.

[0084] At operation 604, method 600 includes determining by the first node that the packet includes an SRv6 micro-segment (uSID) instruction associated with one or more path tracing actions. For example, the first node may shift the destination address field of the packet to determine that the packet includes an SRv6 uSID instruction.

[0085] At operation 606, method 600 includes performing a first path tracing action by a first node based at least in part on a read SRv6 uSID instruction, the first path tracing action including determining timestamp data indicating when the first node received a packet. In some cases, the timestamp data may include a full timestamp, while in other cases, the timestamp data may include a short timestamp that includes a portion of the full timestamp.

[0086] At operation 608, method 600 includes performing a second path tracing action by the first node, at least in part, based on a read SRv6 uSID instruction. The second path tracing action includes determining an interface identifier associated with the interface of the first node used to receive packets. For example, the first node may determine a short interface ID corresponding to the interface of the first node and append that short interface ID to the packet.

[0087] At operation 610, method 600 includes performing a third path tracing action by the first node at least in part based on a read SRv6 uSID instruction, the third path tracing action including sending a packet to a network controller configured to perform operation, administration and maintenance (OAM) actions, the packet including timestamp data and an interface identifier.

[0088] Figure 7 This is a logic flowchart illustrating example method 700, which can be executed at least in part by a network controller or collector to determine the path of packets traversing the network. At operation 702, method 700 begins by receiving telemetry data associated with sending packets through the network. For example, the telemetry data can be received by controller 110 or collector 116 of network 102.

[0089] At operation 704, method 700 includes determining that the telemetry data includes a first interface identifier associated with a first node of the network, the first interface identifier indicating a first interface of the first node used to send packets to a second node. For example, the telemetry data may include a first short interface identifier indicating the interface of the source node 118(1) used to send packets.

[0090] At operation 706, method 700 includes determining that the telemetry data includes a second interface identifier associated with a second node in the network, the second interface identifier indicating a second interface of the second node used to receive packets from the first node. For example, the telemetry data may include a second short interface identifier indicating the interface of the aggregation node 118(4) used to receive packets.

[0091] At operation 708, method 700 includes determining a path through the network based at least in part on telemetry data, and sending packets according to that path, which includes at least a first node and a second node. For example, controller 110 or collector 116 may determine, at least in part, that packets were sent through at least a first node and a second node based on a first interface identifier and a second interface identifier.

[0092] At operation 710, method 700 includes determining, at least in part, the presence of anomalies in the network based on paths. For example, controller 110 may determine that a packet is traveling through the network on an incorrect shortest path, an incorrect ECMP path, a faulty link or node in network 102, and / or similar issues.

[0093] Figure 8This is a logic flowchart illustrating an exemplary method 800, which can be performed at least in part by a network controller or collector to determine a latency measurement associated with transmitting packets through the network. Method 800 begins at operation 802, which includes receiving telemetry data associated with transmitting packets through the network. For example, the telemetry data may be received by a controller 110 or collector 116 of network 102.

[0094] At operation 804, method 800 includes determining that the telemetry data includes a first complete timestamp indicating a first time when the packet was processed by a first node of the network. For example, the telemetry data may include a first complete timestamp corresponding to the time when the source node 118(1) of network 102 receives or sends the packet.

[0095] At operation 806, method 800 includes determining that the telemetry data includes a short timestamp representing a portion of a second full timestamp indicating a second time at which the packet was received by a second node of the network. For example, the telemetry data may include a short timestamp representing a portion of a second full timestamp indicating a second time at which intermediate node 118(2) of network 102 received the packet.

[0096] At operation 808, method 800 includes determining a second complete timestamp based at least in part on a first complete timestamp and a short timestamp. For example, controller 110 or collector 116 may determine the second complete timestamp by combining the short timestamp and the most significant byte of the first complete timestamp.

[0097] At operation 810, method 800 includes calculating the delay associated with the transmission of packets through the network, based at least in part on a first complete timestamp and a second complete timestamp. For example, the delay may correspond to a link delay measurement between the first node and the second node.

[0098] Figure 9 This is a system architecture diagram illustrating the configuration of data center 900, which can be used to implement various aspects of the technologies disclosed herein. Figure 9 The example data center 900 shown includes several server computers 902A to 902F (which may be referred to herein as "one server computer 902" in the singular or "a plurality of server computers 902") for providing computing resources. In some examples, the resources and / or server computers 902 may include or correspond to any type of computing device described herein. Although described as a server, server computer 902 may include any type of networking device, such as a server, switch, router, hub, bridge, gateway, modem, repeater, access point, etc.

[0099] Server computer 902 may be a standard tower, rack, or blade server computer appropriately configured to provide computing resources. In some examples, server computer 902 may provide computing resources 904, including data processing resources such as VM instances or hardware computing systems, database clusters, compute clusters, storage clusters, data storage resources, database resources, networking resources, VPNs, etc. Some servers 902 may also be configured to execute resource manager 906, which is capable of instantiating and / or managing computing resources. For example, in the case of VM instances, resource manager 906 may be a hypervisor or another type of program configured to run multiple VM instances on a single server computer 902. Server computer 902 in data center 900 may also be configured to provide network services and other types of services.

[0100] exist Figure 9 In the example data center 900 shown, server computers 902A to 902F are also interconnected using appropriate LAN 908. It should be understood that the configuration and network topology described herein have been greatly simplified, and more computing systems, software components, networks, and networking devices can be used to interconnect the various computing systems disclosed herein and provide the aforementioned functionality. Appropriate load balancing devices or other types of network infrastructure components can also be used to balance the load between data centers 900, between each server computer 902A to 902F in each data center 900, and potentially among the computing resources in each server computer 902. It should be understood that references... Figure 9 The configuration of the data center 900 described is merely illustrative and other implementations may be used.

[0101] In some cases, data center 900 can provide computing resources, such as applications, VM instances, and storage, on a permanent or on-demand basis. In other types of functions, computing resources provided by the cloud computing network can be used to implement the various services and technologies mentioned above. Computing resources 904 provided by the cloud computing network can include various types of computing resources, such as data processing resources (e.g., tenant containers and VM instances), data storage resources, networking resources, data communication resources, network services, VPN instances, etc.

[0102] Each type of computing resource 904 provided by a cloud computing network can be general-purpose or available in several specific configurations. For example, data processing resources can be used as physical computers or VM instances in a variety of different configurations. VM instances can be configured to run applications, including web servers, application servers, media servers, database servers, some or all of the aforementioned network services, and / or other types of programs. Data storage resources can include file storage devices, block storage devices, etc. Cloud computing networks can also be configured to provide other types of computing resources 904 not specifically mentioned herein.

[0103] In one embodiment, computing resources 904 provided by a cloud computing network can be enabled by one or more data centers 900 (which may be referred to herein as "one data center 900" in the singular or "multiple data centers 900" in the plural). A data center 900 is a facility for housing and operating computer systems and related components. A data center 900 typically includes redundant and backup power, communication, cooling, and security systems. Data centers 900 may also be located in geographically distinct locations. The following will combine... Figure 10 An illustrative implementation of a data center 900 that can be used to implement the technology disclosed herein is described.

[0104] Figure 10 This is a computer architecture diagram, illustrating example computer hardware architectures of nodes and / or computing devices that can be used to implement various aspects of the technologies proposed in this paper. Figure 10 The illustrated computer architecture diagram shows traditional server computers, network devices, workstations, desktop computers, laptops, tablet computers, web applications, e-readers, smartphones, and / or other computing devices. Computer 1000 may include networked devices such as servers, switches, routers, hubs, bridges, gateways, modems, repeaters, access points, etc.

[0105] Computer 1000 includes a baseboard 1002, or "motherboard," which is a printed circuit board to which many components or devices can be connected via a system bus or other electrical communication paths. In an illustrative configuration, one or more central processing units (CPUs) 1004 operate in conjunction with a chipset 1006. The CPU 1004 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of computer 1000.

[0106] The CPU 1004 performs operations by transitioning from one discrete physical state to the next, where state transitions are achieved by manipulating switching elements that distinguish and change these states. Switching elements typically include electronic circuitry, such as flip-flops, that maintains one of two binary states, and electronic circuitry, such as logic gates, that provides the output state based on a logical combination of the states of one or more other switching elements. These basic switching elements can be combined to create more complex logic circuits, including registers, adders / subtractors, arithmetic logic units, floating-point units, and so on.

[0107] Chipset 1006 provides an interface between CPU 1004 and the remaining components and devices on substrate 1002. Chipset 1006 may provide an interface with RAM 1008, which is used as main memory in computer 1000. Chipset 1006 may further provide an interface with a computer-readable storage medium, such as read-only memory (ROM) 1010 or non-volatile RAM (NVRAM), for storing basic routines that facilitate booting computer 1000 and transferring information between various components and devices. ROM 1010 or NVRAM may also store other software components necessary for the operation of computer 1000 according to the configuration described herein.

[0108] Computer 1000 can operate in a networked environment, using logical connections to remote computing devices and computer systems via a network (e.g., network 102 and / or (one or more) networks 1024). Chipset 1006 may include functionality for providing network connectivity via NIC 1012 (e.g., a Gigabit Ethernet adapter). NIC 1012 enables computer 1000 to connect to other computing devices via a network. It should be understood that multiple NICs 1012 may be present in computer 1000 to connect the computer to other types of networks and remote computer systems. In some examples, NIC 1012 may be configured to perform at least some of the techniques described herein and may include components for performing the techniques described herein.

[0109] Computer 1000 can be connected to storage device 1018, which provides non-volatile storage for the computer. Storage device 1018 can store operating system 1020, programs 1022, and data, which have been described in more detail herein. Storage device 1018 can be connected to computer 1000 via storage controller 1014, which is connected to chipset 1006. Storage device 1018 can consist of one or more physical storage units. Storage controller 1014 can be connected to physical storage units via a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other types of interfaces used for physical connection and data transfer between the computer and physical storage units.

[0110] Computer 1000 can store data on storage device 1018 by changing the physical state of physical storage units to reflect the stored information. In different embodiments of this specification, the specific changes in physical state can depend on various factors. Examples of such factors include, but are not limited to, the technology used to implement the physical storage units, whether storage device 1018 is characterized as primary or secondary storage, etc.

[0111] For example, computer 1000 can store information in storage device 1018 by issuing instructions through storage controller 1014 to change the magnetic properties of a specific location within a disk drive unit, the reflection or refraction properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of this specification; the foregoing examples are provided merely for convenience. Computer 1000 can further read information from storage device 1018 by detecting the physical state or characteristics of one or more specific locations within the physical storage unit.

[0112] In addition to the aforementioned high-capacity storage device 1018, computer 1000 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will understand that a computer-readable storage medium is any available medium that provides non-transitory storage of data and is accessible by computer 1000. In some examples, operations performed by system architecture 100 and / or any components included therein may be supported by one or more devices similar to computer 1000. In other words, some or all of the operations performed by system architecture 100 and / or any components included therein may be performed by one or more computer devices 1000 operating in a cloud-based configuration.

[0113] By way of example, and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM (EPROM), electrically-erasable programmable ROM (EEPROM), flash memory or other solid-state storage technologies, compact disc ROM (CD-ROM), digital versatile disk (DVD), high-definition DVD (HD-DVD), Blu-ray or other optical storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information in a non-transitory manner.

[0114] As briefly mentioned above, storage device 1018 may store operating system 1020 used to control the operation of computer 1000. According to one embodiment, the operating system includes a LINUX operating system. According to another embodiment, the operating system includes a WINDOWS® SERVER operating system from MICROSOFT Corporation of Redmond, Washington. According to yet another embodiment, the operating system may include a UNIX operating system or one of its variants. It should be understood that other operating systems may also be used. Storage device 1018 may store other systems, applications, and data used by computer 1000.

[0115] In one embodiment, storage device 1018 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into computer 1000, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions transform computer 1000 by specifying how CPU 1004 transitions between states. According to one embodiment, computer 1000 has access to a computer-readable storage medium storing computer-executable instructions that, when executed by computer 1000, perform the above-described... Figures 1-9 The various processes described herein. The computer 1000 may also include a computer-readable storage medium storing instructions for performing any other computer-implemented operations described herein.

[0116] Computer 1000 may also include one or more input / output controllers 1016 for receiving and processing input from a number of input devices, such as a keyboard, mouse, touchpad, touchscreen, electronic stylus, or other types of input devices. Similarly, input / output controllers 1016 may provide output to a display, such as a computer monitor, flat panel display, digital projector, printer, or other types of output devices. It will be understood that computer 1000 may not include... Figure 10 All components shown may include Figure 10 Other components not explicitly shown, or those that may utilize [the technology / method] Figure 10 The architecture shown is completely different.

[0117] As described herein, computer 1000 may include one or more of a data node, control node, firewall node, edge router, and / or key-value store. Computer 1000 may include one or more hardware processors 1004 (processors) configured to execute instructions stored in one or more stores. The processors 1004 may include one or more cores. Additionally, computer 1000 may include one or more network interfaces (e.g., NIC 1012) configured to provide communication between computer 1000 and other devices via networks such as network 102 and / or network 1024. Network interfaces may include devices configured to couple to personal area networks (PANs), wired and wireless local area networks (LANs), wired and wireless wide area networks (WANs), etc. For example, network interfaces may include devices compatible with Ethernet, Wi-Fi™, etc.

[0118] Program 1022 may include any type of program or process to perform the techniques described in this disclosure for optimizing telemetry data for group path tracking and / or delay measurement techniques.

[0119] In summary, this paper describes techniques for optimizing network path tracing and network latency measurement. Some techniques may include using the fragmentation identifier field of the IPv6 header options and / or fragmentation list, or the TLV of the fragmented routing header, as a telemetry data carrier. These techniques may also include using SRv6 micro-fragmentation (uSID) instructions to instruct network nodes that the node will perform one or more path tracing actions, as well as packet encapsulation and forwarding. Additionally, these techniques may include using short interface identifiers corresponding to node interfaces to trace packet paths through the network. Furthermore, these techniques may include using short timestamps to determine latency measurements associated with packets transmitted through the network. In various examples, the techniques described above and herein can be used together to optimize network path tracing and latency measurement techniques.

[0120] Although the invention has been described with reference to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. For example, while many examples are described with respect to the IPsec protocol, it should be understood that the described techniques can be applied to other protocols. Since other modifications and variations made to adapt to specific operational requirements and environments will be apparent to those skilled in the art, the invention is not to be considered limited to the examples chosen for disclosure purposes, and covers all changes and modifications that do not constitute a departure from the true spirit and scope of the invention.

[0121] While this application describes embodiments with specific structural features and / or methodological actions, it should be understood that the claims are not necessarily limited to the specific features or actions described. Rather, the specific features and actions merely illustrate some embodiments that fall within the scope of the claims of this application.

Claims

1. A method for network path tracking, comprising: receiving telemetry data associated with transmission of a packet through a network, the telemetry data comprising at least: a first short interface identifier associated with a first node of the network, the first short interface identifier indicating a first interface of the first node used to transmit the packet to a second node, the first short interface identifier comprising fewer bits than a node identifier associated with the first node; and a second short interface identifier associated with a second node of the network, the second short interface identifier indicating a second interface of the second node used to receive the packet from the first node, the second short interface identifier comprising fewer bits than a node identifier associated with the second node; wherein a number of bits included in each of the first short interface identifier and the second short interface identifier is determined based on a maximum number of interfaces on any node of the network; determining a path through the network based at least in part on the telemetry data, wherein the packet was transmitted according to the path, the path comprising at least the first node and the second node; and determining whether an anomaly exists in the network based at least in part on the path.

2. The method of claim 1, further comprising receiving topology data associated with the network, wherein, Determining the path is further based at least in part on the topology data.

3. The method of claim 1 or 2, wherein, The telemetry data further comprises one or more short intermediate interface identifiers associated with one or more intermediate nodes of the network disposed along the path between the first node and the second node, the one or more short intermediate interface identifiers comprising at least a first short intermediate interface identifier corresponding to a third interface of a third node that forwarded the packet from the first node to the second node.

4. The method of claim 3, wherein, The path is determined based at least in part on a 3-tuple comprising the first short interface identifier, the second short interface identifier, and a sequence of the one or more short intermediate interface identifiers.

5. The method of claim 1 or 2, wherein, The anomaly comprises one or more of: the path comprising an incorrect shortest path through the network, the path comprising incorrect equal cost multi-path (ECMP) routing, or the network comprising at least one failed link or failed node.

6. The method of claim 1 or 2, wherein, The telemetry data further comprises: a first full timestamp indicating a first time at which the packet was processed by the first node; and a first short timestamp representing a portion of a second full timestamp indicating a second time at which the packet was received by the second node.

7. The method of claim 6, further comprising: determining the second full timestamp based at least in part on the first full timestamp and the first short timestamp; and calculating a latency associated with transmitting the packet along the path based at least in part on the first full timestamp and the second full timestamp.

8. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any one of claims 1 to 7. ​ 9. One or more computer-readable media comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method of any of claims 1-7.

10. An apparatus for network path tracking, comprising: means for receiving telemetry data associated with transmission of a packet through a network, the telemetry data comprising at least: a first short interface identifier associated with a first node of the network, the first short interface identifier indicating a first interface of the first node used to transmit the packet to a second node, the first short interface identifier comprising fewer bits than a node identifier associated with the first node; and a second short interface identifier associated with a second node of the network, the second short interface identifier indicating a second interface of the second node used to receive the packet from the first node, the second short interface identifier comprising fewer bits than a node identifier associated with the second node; wherein a number of bits included in each of the first short interface identifier and the second short interface identifier is determined based on a maximum number of interfaces on any node of the network; means for determining a path through the network based at least in part on the telemetry data, wherein the packet was transmitted according to the path, the path comprising at least the first node and the second node; and means for determining whether an anomaly exists in the network based at least in part on the path.

11. The apparatus of claim 10, further comprising: means for receiving topology data associated with the network, wherein determining the path is further based at least in part on the topology data.

12. The apparatus of claim 10 or 11, wherein, the telemetry data further comprises one or more short intermediate interface identifiers associated with one or more intermediate nodes of the network disposed along the path between the first node and the second node, the one or more short intermediate interface identifiers comprising at least a first short intermediate interface identifier corresponding to a third interface of a third node that forwarded the packet from the first node to the second node.

13. The apparatus of claim 12, wherein, the path is determined based at least in part on a 3-tuple comprising the first short interface identifier, the second short interface identifier, and a sequence of the one or more short intermediate interface identifiers.

14. The apparatus of claim 10 or 11, wherein, the anomaly comprises one or more of: the path comprises an incorrect shortest path through the network, the path comprises incorrect equal cost multi-path (ECMP) routing, or the network comprises at least one failed link or failed node.

15. The apparatus of claim 10 or 11, wherein, the telemetry data further comprises: a first full timestamp indicating a first time at which the packet was processed by the first node; and a first short timestamp representing a portion of a second full timestamp indicating a second time at which the packet was received by the second node.

16. The apparatus of claim 15, further comprising: means for determining the second full timestamp based at least in part on the first full timestamp and the first short timestamp; and ​ a module for calculating a latency associated with sending the packet along the path based at least in part on the first complete timestamp and the second complete timestamp.

Images