Accurate timestamping of outgoing packets

Abstract

The present invention relates to accurately timestamping outgoing packets. A network device includes a port, a transmission pipeline, and a timestamping circuit. The port is configured to connect to a network. The transmission pipeline includes a plurality of pipeline stages and is configured to process packets and send the packets to the network via the port. The timestamping circuit is configured to: temporarily suspend at least some processing of at least a given packet in the transmission pipeline; verify whether a pipeline stage having a variable processing delay downstream of the timestamping circuit satisfies an idle condition; and only when the pipeline stage satisfies the idle condition, timestamp the given packet and resume processing of the given packet.

Description

Technical Field

[0001] The present invention generally relates to packet communication and, more particularly, to methods and systems for accurately timestamping outbound packets in network devices. Background Art

[0002] In various applications, network devices timestamp the packets they receive, send, or forward. For example, timestamping is used in time synchronization protocols that synchronize network nodes to a common timebase. Common examples of such protocols are the Precision Time Protocol (PTP) defined in IEEE standards 1588-2002 and 1588-2008 and their subsequent versions. Summary of the Invention

[0003] Embodiments of the invention described herein provide a network device that includes a port, a transmission pipeline, and a timestamping circuit. The port is configured to connect to a network. The transmission pipeline includes a plurality of pipeline stages and is configured to process packets and send the packets to the network via the port. The timestamping circuit is configured to: temporarily suspend at least some processing of at least a given packet in the transmission pipeline; verify whether a pipeline stage having a variable processing delay downstream of the timestamping circuit satisfies an idle condition; and timestamp the given packet and resume processing of the given packet only if the pipeline stage satisfies the idle condition.

[0004] In some embodiments, the pipeline stage having a variable processing delay is a buffer having a variable occupancy level.

[0005] In some embodiments, the timestamping circuit is configured to support: (i) a normal mode in which packets are timestamped regardless of whether the pipeline stage satisfies the idle condition; and (ii) a high accuracy mode in which packets are timestamped only if the pipeline stage satisfies the idle condition. In an exemplary embodiment, the timestamping circuit is configured to apply the high accuracy mode only to selected packets. In another embodiment, the timestamping circuit is configured to apply the high accuracy mode only to time synchronization packets. In yet another embodiment, the timestamping circuit is configured to apply the high accuracy mode only when the port is not under backpressure from the network.

[0006] In an embodiment, the timestamping circuit is configured to adaptively modify the stringency of the idle condition. In the disclosed embodiments, the timestamping circuit is configured to adjust the timestamping accuracy of the timestamping circuit by adaptively modifying the idle condition until a target accuracy is achieved. In an example embodiment, the timestamping circuit is configured to: apply a first idle condition having a first stringency when a given packet is associated with a first application; and apply a second idle condition having a second stringency when the given packet is associated with a second application.

[0007] In another embodiment, the timestamping circuit is configured to apply first and second idle conditions having different first and second degrees of stringency to different first and second given packets of the same application. In yet another embodiment, the timestamping circuit is configured to apply a modified idle condition in response to detecting that a port is under backpressure from the network. In yet another embodiment, the timestamping circuit is configured to set the idle condition based on the link speed of the network link connected to the port.

[0008] In an embodiment, the timestamping circuit is configured to verify whether the pipeline stage meets the idle condition by receiving a status indication from the pipeline stage. In an example embodiment, the timestamping circuit is configured to verify whether the pipeline stage meets the idle condition by waiting for a duration when processing in the transmission pipeline is aborted.

[0009] In the disclosed embodiments, the timestamping circuit is configured to report the degree of idleness of the pipeline stage to the processor of the network device. In an embodiment, the timestamping circuit is configured to apply a fail-safe mechanism that limits the maximum degree of aborting processing in the transmission pipeline.

[0010] In some embodiments, the transmission pipeline and the timestamping circuit are configured to process packets for multiple hosts. In an example embodiment, the timestamping circuit is configured to apply a first setting of the idle condition to a first host among the hosts, and apply a second setting of the idle condition different from the first setting to a second host among the hosts. In another embodiment, the timestamping circuit is configured to perform a first type of verification of the idle condition for the first host among the hosts, and perform a second type of verification of the idle condition different from the first type for the second host among the hosts.

[0011] In yet another embodiment, a given packet originates from a given host among the hosts, and the timestamping circuit is configured to temporarily pause processing of packets originating from hosts other than the given host so that the pipeline stage can meet the idle condition for timestamping the given packet. In yet another embodiment, the timestamping circuit is configured to prohibit aborting at least some packets originating from a given host among the hosts.

[0012] In an embodiment, a timestamping circuit is configured to insert a timestamp into a given packet, and a pipeline stage having a variable processing delay is an encryption engine configured to encrypt the given packet including the inserted timestamp.

[0013] According to an embodiment of the present invention, a method in a network device is further provided. The method includes: using a transmission pipeline in the network device to process a packet and send the packet to the network, the transmission pipeline including a plurality of processing stages. Using a timestamping circuit in the network device, (i) temporarily suspend at least some processing of at least a given packet in the transmission pipeline, verify whether a pipeline stage having a variable processing delay downstream of the timestamping circuit satisfies a vacancy condition, and (iii) timestamp the given packet and resume processing of the given packet only when the pipeline stage satisfies the vacancy condition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be more fully understood from the following detailed description of embodiments of the invention in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a block diagram schematically showing a network device that timestampes an egress packet with high precision according to an embodiment of the present invention;

[0016] Figure 2 is a flowchart schematically showing a method for accurately timestamping an egress packet according to an embodiment of the present invention; and

[0017] Figure 3 is a graph showing an example relationship between timestamping accuracy and vacancy condition strictness according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] OVERVIEW

[0019] Embodiments of the present invention described herein provide methods and apparatuses for accurately timestamping outbound (egress) packets in a network device. Embodiments described herein mainly relate to implementations in network adapters, but the disclosed techniques are also applicable to network switches and other types of network devices that require accurate timestamping.

[0020] Generally, a network device includes a transmission (TX) pipeline that processes packets and transmits them to the network via an egress port. To timestamp packets accurately with the egress time, it is desirable to timestamp the packets at a point in the pipeline as close as possible to the egress port. However, in many practical implementations, one or more processing stages still remain downstream of the timestamping point, i.e., between the timestamping point and the egress port.

[0021] If the latency of these intermediate processing stages is fixed, these intermediate processing stages are less of a problem because this latency can be calibrated and compensated for in the timestamp. On the other hand, intermediate processing stages with variable latency are problematic because they add variable jitter to the timestamp and degrade their accuracy. For example, aspects related to the estimation of time jitter in timestamps are addressed in U.S. Patent Application 17 / 315,396, filed May 10, 2021, the disclosure of which is incorporated herein by reference.

[0022] For example, any buffer or queue located between the timestamping point and the egress port will add variable latency to the timestamp because the latency of the buffer or queue changes according to its occupancy level. For clarity and conciseness, the following description refers to a "buffer". However, in the context of the present disclosure and in the claims, the term "buffer" broadly refers to various types of memories, registers, or data structures that temporarily store packets, parts of packets, or descriptors, pointers, or metadata associated with packets. For example, individual queues are also considered buffers herein. The use of a buffer can have any suitable producer / consumer semantics (e.g., the entity written to the buffer can be the same entity read from the buffer, or a different entity; the buffer can have multiple producers and / or multiple consumers; the order in which items are written to the buffer can be the same as, or different from, the order in which they are read from the buffer; etc.).

[0023] Other types of processing stages in the pipeline can also have variable processing delays and can thus affect timestamping accuracy. Examples of such processing stages include, but are not limited to, cryptographic engines such as IPsec engines and MACsec engines. The following description refers mainly to buffers by way of example, but the disclosed techniques can be applied to various other types of processing stages. The terms "pipeline stage" and "processing stage" are used interchangeably herein.

[0024] In some embodiments of the present invention, the network device includes a timestamping circuit that timestampes egress packets with high accuracy even if the TX pipeline includes a buffer located downstream of the timestamping point. (The terms "downstream of the timestamping point" and "downstream of the timestamping circuit" are used interchangeably herein.)

[0025] In some embodiments, to accurately timestamp a given packet, the timestamping circuit ensures that at the time of timestamping, the buffer downstream of the timestamping point is sufficiently empty. In this way, the timestamping circuit substantially clears the low-jitter path in the pipeline for the given packet. The timestamping circuit can verify various empty conditions regarding the buffer, e.g., the buffer is completely empty, the occupancy level of the buffer is below a threshold, etc. The empty condition is generally such that when satisfied, the delay jitter of the buffer is small enough to achieve the required timestamping accuracy.

[0026] In some embodiments, the timestamping circuit actively empties and clears the buffer by temporarily aborting at least a portion of the processing in the pipeline (e.g., aborting the processing upstream of the buffer, at least for the given packet and possibly all packets). Then, the timestamping circuit verifies whether the buffer meets the empty condition. Only when the buffer meets the empty condition does the timestamping circuit timestamp the given packet and resume pipeline processing.

[0027] The timestamping circuit can use various techniques (directly or indirectly) to evaluate whether the buffer meets the empty condition. In some embodiments, the timestamping circuit receives an explicit fill status indication from the buffer. In other embodiments, the timestamping circuit waits for a defined duration when the pipeline processing is stalled. A combination of these two techniques can also be used.

[0028] As can be appreciated, aborting the processing in the pipeline can result in some degradation of performance, such as average transmission delay and throughput. Thus, in some embodiments, the timestamping circuit selectively applies the disclosed techniques, i.e., only to selected packets. For example, high-accuracy timestamping may be applied only to time synchronization packets, e.g., PTP packets.

[0029] Another way to reduce the possible performance degradation that may be caused by the disclosed techniques is to adaptively modify the strictness of the empty condition to match specific requirements. For example, some packet types or applications may require higher-accuracy timestamping than other packet types or applications, and thus require more stringent empty conditions. For other packet types or applications, more relaxed empty conditions may be sufficient. Since performance degradation generally increases with the strictness of the empty condition, this adaptive approach can achieve a better trade-off between timestamping accuracy and communication performance.

[0030] Even in the presence of buffers with large delay jitter (e.g., on the order of hundreds of nanoseconds or a few microseconds), the methods and apparatuses described herein achieve highly accurate timestamping (e.g., on the order of a few nanoseconds).

[0031] This document describes various implementation options and usage scenarios of the disclosed technology. Aspects of multi-host configurations in which network devices serve multiple hosts are also addressed.

[0032] System Description

[0033] Figure 1 is a block diagram schematically showing a network device 20 that timestamp-exports packets with high accuracy according to an embodiment of the present invention. In this example, the network device 20 is a network adapter, for example, an Ethernet NIC, Infiniband TM host channel adapter (HCA), data processing unit (DPU - also known as "intelligent NIC"), network-enabled graphics processing unit (GPU), field programmable gate array (FPGA), etc. Alternatively, the network device 20 may include a network switch or router, or any other suitable type of device capable of network communication.

[0034] The network device 20 (a network adapter in this example) includes a host interface 22 for connecting to a host 23, at least one port 24 for connecting to a network 28, a transmit (TX) pipeline 32 for processing the transmission of egress (outbound packets), a processor 36 for running software that manages the network device 20, and a timestamping circuit 48. The pipeline 32 includes a plurality of processing stages 40 and buffers 44. For clarity, other possible elements of the network device 20 that are not relevant to the disclosed technology, such as a receive (RX) pipeline for receiving packets from the network, have been omitted from the figure.

[0035] The TX pipeline 32 receives packets via the host interface 22 for transmission to the network 28. The pipeline 32 uses the processing stages 40 to process the packets, four stages in this example denoted as STAGE #1 - STAGE #4. Operations that may be applied by the stages 40 may include, for example, encapsulating the packet in a VLAN tunnel header, encrypting the packet using IPSec, and calculating the Ethernet frame check sequence (FCS) of the packet and appending the FCS to the packet, to name just a few. Between stage #2 and stage #3, the packet is buffered in the buffer 44. After stage #4, the packet is transmitted to the network 28 via the port 24 (also known as the egress port).

[0036] In some embodiments, the timestamping circuit 48 timestamps at least some of the packets in the pipeline 32 using the current time (e.g., the time of day, network time for synchronizing various elements of the network 28 to a common time base, or any other suitable time). The timestamp applied to a given packet is intended to represent the egress time of the packet, i.e., the exact time the packet leaves the network device. The timestamping circuit 48 may obtain the current time from an integrated clock (e.g., a Precision Time Protocol hardware clock (PHC)) or from any other suitable source.

[0037] In this example, the timestamping circuit 48 timestamps a packet as it traverses stage #2 of the pipeline 32. This stage of the pipeline is referred to as the "timestamping point". Note that the buffer 44 is located downstream of the timestamping point, i.e., between the timestamping point and the egress port 24. Thus, the occupancy level of the buffer 44 affects the latency between the timestamping point and the egress port 24, which in turn affects the accuracy of the timestamp. Techniques for mitigating this undesirable effect are described in detail below.

[0038] Figure 1 The configuration of the network device 20 shown is an example configuration described purely for conceptual clarity. Any other suitable configuration may be used in alternative embodiments. For example, the pipeline 32 may include any other suitable number of processing stages of any suitable type. The pipeline 32 may include more than one buffer 44, either upstream or downstream of the timestamping point. Such buffers may have any suitable type and size. The timestamping point may be located at any other suitable stage of the pipeline. When the network device is a switch or router, as opposed to a network adapter, the host interface 22 may be absent. Instead, the network device may receive packets for transmission from another source (e.g., from an ingress port).

[0039] The various elements of the network device 20 may be implemented using suitable hardware (such as in one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs)), using software, or using a combination of hardware and software elements. In some embodiments, the processor 36 includes a general purpose processor that is programmed with software to perform the functions described herein. The software may, for example, be downloaded electronically to the processor over a network, or alternatively or additionally, it may be provided and / or stored on a non-transitory tangible medium, such as a magnetic, optical, or electronic memory.

[0040] Techniques for Accurate Timestamping

[0041] As described above, since the buffer 44 is located between the timestamping point (stage #2) and the egress port 24, the latency between the timestamping point and the actual packet egress depends on the occupancy level of the buffer. In an example practical implementation, this latency can vary between 100 nS (for an empty buffer) and 2 μS (for a full buffer). In some applications, such as for timestamping PTP packets, this magnitude of jitter is unacceptable.

[0042] In some embodiments, the timestamping circuit 48 overcomes the latency jitter of the buffer 44 by ensuring that for a given packet that needs to be accurately timestamped, the buffer 44 is sufficiently empty when the packet is timestamped and sent downstream in the pipeline 32.

[0043] In some embodiments, when preparing to timestamp a given packet, the timestamping circuit 48 temporarily aborts at least some processing in pipeline 32 to empty buffer 44. Typically, the processing is aborted in processing stage 40 upstream of buffer 44, and the processing continues normally in processing stage 40 downstream of buffer 44. In an exemplary implementation, the timestamping circuit 48 aborts the operation of the stage immediately preceding it in the pipeline ( Figure 1 the stage #2 in the example). As a result, the output of the previous stage accumulates in the pipeline (e.g., in the buffer between the stages). In extreme cases, the buffer space may be exhausted, and the entire portion of the pipeline upstream of buffer 44 may stall. However, such events are typically very rare. However, more generally, circuit 48 may use any suitable form of abort that causes buffer 44 to empty.

[0044] In one embodiment, circuit 48 aborts only the processing of a given packet to be timestamped. In other embodiments, the processing of all outstanding packets is aborted. In other embodiments, the processing of a selected subset of packets is aborted. For example, in the sense that they should not be aborted, some packet types may be defined as critical. A non-limiting example is a "mission-critical" packet in an autonomous vehicle network. In such a case, circuit 48 may abort only the processing of packets not defined as critical.

[0045] After aborting packet processing, the timestamping circuit 48 verifies whether the buffer meets a "vacancy condition" defined for buffer 44. The vacancy condition is typically defined such that when met, the delay jitter of buffer 44 is small enough to achieve the required timestamping accuracy.

[0046] In various embodiments, the timestamping circuit 48 may verify various vacancy conditions. In one embodiment, the vacancy condition requires buffer 44 to be completely empty. Other possible vacancy conditions may require the occupancy level of buffer 44 to be below a certain threshold, buffer 44 to hold fewer than a threshold number of packets, buffer 44 to hold less than a threshold data size, etc. The thresholds for the various vacancy conditions may be fixed (e.g., predefined) or adaptive.

[0047] As described above, in some embodiments, the idle condition can be adaptive, for example, by adjusting a threshold or in any other suitable manner. An example scenario of an adaptive idle condition involves the use of PAUSE frames (sometimes also referred to as the XON / XOFF mechanism) in Ethernet. Typically, in response to receiving a PAUSE frame from a link partner, it is expected that network device 20 delays its transmission, temporarily increasing the delay that packets experience between the time-stamping point and the egress on the line. In some embodiments, the time-stamping circuit 48 can react to the increased delay by relaxing the idle condition to ensure that the time-stamped packets are not affected by a large downstream delay.

[0048] Another example for an adaptive idle condition involves runtime tuning. In some embodiments, the typical or maximum amount of time that one or more measurable packets in the pipeline stage downstream from the time-stamping point spend in the pipeline from the time-stamping point to their egress. In an embodiment, the time-stamping circuit 48 can tune (e.g., continuously at runtime) the idle condition to ensure that the pipeline is blocked for a necessary minimum amount of time.

[0049] In other embodiments, the time-stamping circuit 48 can apply different idle conditions with different levels of strictness to different packets that require different levels of time-stamping accuracy. This type of embodiment is further elaborated below with reference to Figure 3 Further elaborate on this type of embodiment.

[0050] In other embodiments, depending on the link speed of the network link connected to port 24, the time-stamping circuit 48 can apply different idle conditions. For example, the time-stamping circuit 48 can use a certain idle condition on a 10 GbE link, a different idle condition on a 25 GbE link, and yet another idle condition on a 200 GbE link.

[0051] In various embodiments, the time-stamping circuit 48 can use various techniques to evaluate whether buffer 44 meets the idle condition. In some embodiments, the time-stamping circuit 48 receives an explicit fill status indication from the buffer. In other embodiments, the time-stamping circuit 48 waits for a certain duration when the processing in the pipeline is aborted. This duration can be fixed (e.g., predefined) or adaptive. A combination of these two techniques (explicit buffer status indication plus waiting time) can also be used. All of these techniques are considered ways of an adaptive idle condition herein.

[0052] When buffer 44 meets the idle condition, it can be safely assumed that the delay jitter between the time-stamping point and the egress port is small. Therefore, when verifying that buffer 44 meets the idle condition, the time-stamping circuit 48 timestamps a given packet and resumes normal processing of pipeline 32.

[0053] In some embodiments, in addition to verifying the empty condition, the timestamping circuit 48 also reports the degree of emptiness of the buffer 44 to the processor 36 and / or the host 23.

[0054] In some embodiments, the timestamping circuit 48 selectively applies the disclosed techniques, i.e., only to selected packets. In other words, the timestamping circuit 48 can support two modes: a normal mode and a high-accuracy mode. In the normal mode, packets are timestamped regardless of whether the buffer 44 meets the empty condition, while in the high-accuracy mode, packets are timestamped only when the buffer 44 meets the empty condition. The timestamping circuit can apply the high-accuracy mode only to selected packets, e.g., only to PTP packets or other time synchronization packets.

[0055] In some embodiments, the timestamping circuit 48 can select between modes based on an external indication (e.g., from trusted software running in the processor 36) that identifies which packets require high-accuracy timestamping and which packets do not. For example, the timestamping circuit can receive an indication that differentiates between PTP packets and other packets. For example, the processor 36 can provide an indication for a given packet by setting appropriate bits in the metadata associated with the packet. Such metadata can travel through the pipeline 32 with the packet. Alternatively, any other suitable form of indication in metadata or other forms can be used. Further alternatively, the timestamping circuit can autonomously identify packets that require high-accuracy timestamping.

[0056] Another factor that can affect the timestamping accuracy is backpressure from the network 28. In other words, even if the buffer 44 meets the empty condition, if the network imposes backpressure on the network device 20 due to congestion, packets may still stall or experience high latency. Thus, in some embodiments, the timestamping circuit 48 applies the high-accuracy mode only when the port 44 is not under backpressure from the network. Another possible way to account for backpressure is to modify the empty condition if there is backpressure. For example, in the presence of backpressure from the network, the circuit 48 can wait for an extended period before timestamping high-accuracy packets. Waiting for an extended period increases the likelihood that the buffer 44 is empty enough despite the backpressure.

[0057] In some embodiments, when aborting packet processing to accurately timestamp a given packet, the timestamping circuit 48 applies a fail-safe mechanism that limits the extent of the temporary abort of processing in the restriction pipeline 32. Such a mechanism prevents the temporary abort from causing excessive performance degradation. The circuit 48 can limit the extent of the abort in various ways and using various criteria. In an example embodiment, the timestamping circuit 48 is configured with a (fixed or adaptive) timeout, i.e., the maximum length of time that packet processing is allowed to be aborted. In another embodiment, the circuit 48 is configured with a (fixed or adaptive) maximum data size whose processing is allowed to be aborted. The maximum data size can be expressed, for example, in terms of the number of packets or the number of bits. A hybrid criterion that takes into account both time and data size can also be used. In some embodiments, the fail-safe mechanism is configurable. Configuring the maximum timeout or data size to infinity is equivalent to disabling the fail-safe mechanism.

[0058] If the fail-safe mechanism is triggered when preparing to timestamp a given packet (e.g., exceeding the maximum timeout and / or maximum data size), the timestamping circuit 48 can take any suitable action. For example, the circuit 48 can resume timestamping with normal accuracy, i.e., timestamp the given packet regardless of whether the idle condition is satisfied. As another example, the circuit 48 can abort the transmission of the given packet and send an error message to the processor 36 and / or the host 23.

[0059] Figure 2 is a flowchart schematically showing a method for accurately timestamping egress packets in the network device 20 according to an embodiment of the present invention. The method begins with the TX pipeline 32 of the network device 20 receiving and processing an egress packet in a TX processing step 50. As part of packet processing in the pipeline, the egress packet is buffered in a buffer 44.

[0060] In a packet selection step 54, the timestamping circuit 48 selects a packet to be timestamped with high accuracy. The selected packet can include, for example, a PTP packet. In an abort step 58, the timestamping circuit 48 instructs the pipeline 32 to abort at least some of the processing performed upstream of the buffer 44 for at least the selected packet. Any kind of pipeline operation performed by any pipeline stage 40 can be aborted, e.g., encapsulation, encryption, FCS calculation, etc.

[0061] In an idle verification step 62, the timestamping circuit 48 verifies whether the buffer 44 satisfies a specific idle condition. The idle condition can be predefined or can be changed dynamically. Once it is found that the buffer 44 satisfies the idle condition, in a timestamping step 66, the timestamping circuit 48 timestamps the selected packet. Then, in a resume step 70, the timestamping circuit 48 resumes normal operation of the pipeline 32.

[0062] Figure 2The process is an example process described purely for conceptual clarity. In alternative embodiments, any other suitable process may be used to implement the disclosed technology.

[0063] Figure 3 is a qualitative graph showing an example relationship between timestamp addition accuracy and idle condition strictness according to an embodiment of the present invention. Figure 3 The horizontal axis of represents the strictness of the idle condition, where larger values correspond to more strict conditions (e.g., a requirement for the buffer to be completely empty, or a longer wait time), and smaller values correspond to more lenient conditions. The vertical axis represents timestamp addition accuracy, where larger values correspond to worse accuracy (and thus higher time jitter in the timestamp), and smaller values correspond to better accuracy (and thus lower time jitter).

[0064] Curve 80 shows the qualitative correlation of timestamp addition accuracy with the strictness of the idle condition. As can be seen, relaxing the idle condition results in higher jitter and vice versa. However, typically, there are constraints on the achievable accuracy, even when the idle condition becomes very strict. For example, once the idle condition is strict enough to empty the buffer with a very high probability, further tightening of the condition will provide little or no additional improvement in accuracy.

[0065] In various embodiments, network device 20 may use the above correlation to reduce performance degradation. In some embodiments, the timestamp addition circuit 48 may adapt the idle condition to match the timestamp addition accuracy actually required for a certain packet or application. In this way, the network device enforces strict idle conditions only when actually needed and not always, thereby minimizing performance degradation.

[0066] For example, in some embodiments, when timestamping packets associated with different applications, the timestamp addition circuit 48 may apply different idle conditions with different degrees of strictness. Additionally or alternatively, when timestamping different packets associated with a given application, the timestamp addition circuit 48 may apply different idle conditions with different degrees of strictness. In some embodiments, the timestamp addition circuit may receive an indication (e.g., in metadata traveling with the packet) indicating the required strictness of the idle condition. In other embodiments, the timestamp addition circuit may autonomously determine the required strictness, e.g., by autonomously identifying the application associated with a given packet.

[0067] In some embodiments, by adaptively modifying the idle condition, the timestamp addition circuit 48 adjusts the timestamp addition accuracy until the desired target accuracy is achieved.

[0068] Accurate Timestamp Addition for Processing Stages with Variable Delay

[0069] The above description mainly relates to buffers, which have variable occupancy levels and thus variable latencies. However, the disclosed techniques are not limited to buffers and can be used with various other processing levels having variable processing latencies. A non-limiting example of a variable latency processing level is an encryption engine. Generally, although not necessarily, processing levels that operate on packet data (as opposed to only operating on packet headers) are more likely to have large latency variations.

[0070] When the variable latency processing level is downstream of the timestamping point in the TX pipeline 32, the timestamping circuit 48 can apply the disclosed techniques in a manner similar to how it applies them to buffers.

[0071] In an example embodiment, when preparing to accurately timestamp a given packet, the timestamping circuit 48 temporarily aborts the processing of at least some packets in the TX pipeline upstream of the variable latency level. The circuit 48 then checks whether the variable latency processing level satisfies an idle condition. The idle condition is typically set such that the variable latency processing level has completed the processing of all previous packets and will thus delay the given packet with a minimum known latency.

[0072] Any of the above idle conditions can be used, as well as any of the above techniques for verifying whether the idle condition is satisfied.

[0073] In an example embodiment, each variable latency processing level in the pipeline 32 is configured to output a "done" signal when it has completed processing and becomes idle. One possible way for the timestamping circuit 48 to verify the idle condition is to wait until all "done" outputs of the downstream variable latency levels are set. In other embodiments, the circuit 48 can check only a subset of the "done" signals, or use the "done" signals in any other way, possibly in combination with other information.

[0074] Another advantage of the disclosed techniques is the ability to insert a timestamp into a packet and then encrypt the packet including the timestamp, and still achieve a high timestamping accuracy. Since the encryption level tends to have large latency variations, it is impossible or at least highly challenging to achieve this goal without the disclosed techniques.

[0075] Usage of multiple hosts

[0076] In Figure 1 the embodiment, the network device 20 serves a single host 23. However, in an alternative embodiment, the network device 20 (a network adapter in this example) can serve multiple hosts 23 that can operate independently of each other. Specifically, the network device can apply the disclosed timestamping techniques to egress packets originating from multiple different hosts.

[0077] In some embodiments, the timestamping circuit 48 in the network device 20 operates differently for different hosts. For example, the timestamping circuit 48 may evaluate different idle conditions for different hosts (e.g., each host may have its own idle condition), and / or verify the idle condition differently for different hosts. In an example embodiment, the timestamping circuit 48 may verify that the buffer 44 is empty for one host and also wait for a defined duration. For a second host, the timestamping circuit 48 may verify that the buffer 44 is 90% empty without waiting for any additional duration. For a third host, the timestamping circuit 48 may completely deactivate high-accuracy timestamping. Alternatively, any other suitable operating mode may be used.

[0078] In some embodiments, the timestamping circuit 48 may assign a priority to selected hosts over other hosts in order to ensure high-accuracy timestamping of packets originating from that host. For example, the timestamping circuit 48 may temporarily pause transmissions from all hosts except the selected host and timestamp packets from the selected host during the pause. As another example, the circuit 48 may prohibit aborting (some or all) packets of the selected host due to accurate timestamping of other hosts. Such selective operation may be transparent to other hosts. For example, this feature is useful when one of the hosts in a multi-host configuration acts as a PTP master or otherwise plays some critical role in the network.

[0079] It should be understood that the above embodiments are cited by way of example, and the present invention is not limited to what is specifically shown and described above. Instead, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications thereof that would occur to those skilled in the art upon reading the above description and that are not disclosed in the prior art. Documents incorporated by reference into this patent application are considered to be part of this application, except to the extent that any term is defined in those incorporated documents in a manner that conflicts with the definitions expressly or implicitly set forth in this specification, in which case only the definitions in this specification shall be considered.

Claims

1. A network device, comprising: a port for connecting to a network; a transmission pipeline that includes a plurality of pipeline stages and is configured to process packets and send the packets to the network via the port; and a timestamping circuit configured to: apply different idle conditions with different strictness levels to different packets that require different timestamping accuracy levels; temporarily suspend at least some processing of at least a given packet in the transmission pipeline; verify whether a pipeline stage with a variable processing delay located downstream of the timestamping circuit meets the idle condition applied to the given packet; and timestamp the given packet and resume processing of the given packet only if the pipeline stage meets the idle condition applied to the given packet.

2. The network device according to claim 1, wherein the pipeline stage with the variable processing delay is a buffer with a variable occupancy level.

3. The network device according to claim 1, wherein the timestamping circuit is configured to support: (i) a normal mode in which packets are timestamped regardless of whether the pipeline stage meets the idle condition; and (ii) a high-accuracy mode in which packets are timestamped only if the pipeline stage meets the idle condition.

4. The network device according to claim 3, wherein the timestamping circuit is configured to apply the high-accuracy mode only to selected packets.

5. The network device according to claim 3, wherein the timestamping circuit is configured to apply the high-accuracy mode only to time synchronization packets.

6. The network device according to claim 3, wherein the timestamping circuit is configured to apply the high-accuracy mode only when the port is not under backpressure from the network.

7. The network device according to claim 1, wherein the timestamping circuit is configured to adaptively modify the strictness of the idle condition.

8. The network device according to claim 1, wherein the timestamping circuit is configured to adjust the timestamping accuracy of the timestamping circuit by adaptively modifying the idle condition until a target accuracy is achieved.

9. The network device according to claim 1, wherein the timestamping circuit is configured to: apply a first idle condition with a first strictness level when the given packet is associated with a first application; and apply a second idle condition with a second strictness level when the given packet is associated with a second application.

10. The network device according to claim 1, wherein the timestamping circuit is configured to apply a first idle condition and a second idle condition with different first and second strictness levels to different first and second given packets of the same application.

11. The network device according to claim 1, wherein the timestamping circuit is configured to apply a modified idle condition in response to detecting that the port is under backpressure from the network.

12. The network device according to claim 1, wherein the timestamping circuit is configured to set the idle condition according to the link speed of the network link connected to the port.

13. The network device according to claim 1, wherein the timestamping circuit is configured to verify whether the pipeline stage meets the idle condition by receiving a status indication from the pipeline stage.

14. The network device according to claim 1, wherein the timestamping circuit is configured to verify whether the pipeline stage meets the idle condition by waiting for a duration when the processing in the transmission pipeline is aborted.

15. The network device according to claim 1, wherein the timestamping circuit is configured to report the degree of idleness of the pipeline stage to the processor of the network device.

16. The network device according to claim 1, wherein the timestamping circuit is configured to apply a fail-safe mechanism that limits the maximum degree of aborting the processing in the transmission pipeline.

17. The network device according to claim 1, wherein the transmission pipeline and the timestamping circuit are configured to process packets for multiple hosts.

18. The network device according to claim 17, wherein the timestamping circuit is configured to apply a first setting of the idle condition to a first host among the hosts, and apply a second setting different from the first setting of the idle condition to a second host among the hosts.

19. The network device according to claim 17, wherein the timestamping circuit is configured to perform a first type of verification of the idle condition for a first host among the hosts, and perform a second type of verification different from the first type of the idle condition for a second host among the hosts.

20. The network device according to claim 17, wherein the given packet originates from a given host among the hosts, and wherein the timestamping circuit is configured to temporarily pause the processing of packets originating from hosts other than the given host so that the pipeline stage can meet the idle condition for timestamping the given packet.

21. The network device according to claim 17, wherein the timestamping circuit is configured to prohibit aborting at least some of the packets originating from a given host among the hosts.

22. The network device according to claim 1, wherein the timestamping circuit is configured to insert a timestamp into the given packet, and wherein the pipeline stage having the variable processing delay is an encryption engine configured to encrypt the given packet including the inserted timestamp.

23. A method in a network device, the method comprising: processing a packet using a transmission pipeline in the network device and sending the packet to a network, the transmission pipeline including a plurality of processing stages; and using a timestamping circuit in the network device: applying different idle conditions with different levels of strictness to different packets requiring different levels of timestamp accuracy; temporarily aborting at least some of the processing of at least a given packet in the transmission pipeline; Verify whether a pipeline stage with variable processing delay located downstream of the timestamping circuit meets an idle condition applied to the given packet; and Timestamp the given packet and resume processing of the given packet only if the pipeline stage meets the idle condition applied to the given packet.

24. The method according to claim 23, wherein the pipeline stage with the variable processing delay is a buffer with variable occupancy levels.

25. The method according to claim 23, comprising operating the timestamping circuit in (i) a normal mode and (ii) a high accuracy mode, wherein in the normal mode, packets are timestamped regardless of whether the pipeline stage meets the idle condition, and in the high accuracy mode, packets are timestamped only if the pipeline stage meets the idle condition.

26. The method according to claim 23, comprising adaptively modifying the strictness of the idle condition.

27. The method according to claim 23, comprising adjusting the timestamping accuracy of the timestamping circuit by adaptively modifying the idle condition until a target accuracy is achieved.

28. The method according to claim 23, wherein verifying whether the pipeline level meets the empty condition includes: Wait for a duration when processing in the transmission pipeline is aborted.

Images