Receiver timestamp accuracy

Abstract

In one embodiment, a network device includes a network interface port configured to receive data symbols from a network node over a packet data network, at least some of the symbols being included in a data packet; and a controller circuit including a physical layer (PHY) circuit including a receive PHY pipeline circuit configured to process the received data symbols, and a counter configured to maintain a counter value indicating a number of data symbols in the receive PHY pipeline circuit.

Description

Technical Field

[0001] The present invention relates to network equipment and, in particular but not exclusively, to receive-end processing. Background Art

[0002] In a computer network, each node (e.g., a switch or an endpoint) typically has its own real-time clock. In many applications, it is desirable that the real-time clocks of different nodes be precisely synchronized. However, such synchronization can be difficult to achieve due to the delays and jitter involved in distributing clock synchronization messages between nodes.

[0003] The Precision Time Protocol (PTP) is considered a solution to this problem. The protocol enables network nodes to determine the offset of their respective clocks using messaging between nodes and master devices to achieve nanosecond accuracy. In order to measure the maximum accuracy when clock offsets, hardware-based timestamps are usually used, for example, as described in "Implementation and Performance of Time Stamping Techniques" by Weibel and Bechaz at the 2004 IEEE1588 Conference (September 28, 2004), which is incorporated herein by reference. A PTP Hardware Clock (PHC) is a hardware clock (e.g., including an oscillator and a counter) that runs according to the PTP format. The PHC is synchronized to the host in the network.

[0004] Successful operation of the PTP protocol generally requires accurate timestamping of received packets so that the timestamped point is not affected by the network load or the CPU load of the receiving node. Therefore, hardware timestamping in a network interface controller (NIC) or other network node is generally better than software timestamping, which may suffer from CPU load errors, while hardware timestamping is more accurate and stable.

[0005] Now let's describe an example, synchronization. In the first step, the frequency is synchronized. The master sends a synchronization message at time t1, which arrives at the slave at time t2. The master sends a synchronization message at time t3, which arrives at the slave at time t4. The slave calculates the drift according to:

[0006] [t4-t2–(t3-t1)] / (t3-t1).

[0007] The slave device can then adjust its clock based on the drift to synchronize the frequency of its clock.

[0008] The master device then sends another synchronization message at time t5, which is received by the slave device at time t6. The slave device then sends a delay request message at time t7. The master device then sends another message at time t8, which includes the time at which the master device received the delay request. The slave device can calculate the round trip time as equal to:

[0009] t8-t5–(t7-t6).

[0010] Therefore, the one-way delay from the master to the slave is given by:

[0011] [t8-t5–(t7-t6)] / 2.

[0012] The master then sends another message at time t9. When the slave receives the message at t10, the slave knows that the phase is t10 plus the one-way delay.

[0013] U.S. Patent 8,370,675 to Kagan describes a method of clock synchronization that includes calculating an offset value between a local clock time of a real-time clock circuit and a reference clock time, and loading the offset value into a register associated with the real-time clock circuit. The local clock time is then added to the value in the register to give an adjusted value of the local clock time that is synchronized with the reference clock. Summary of the invention

[0014] According to an embodiment of the present disclosure, a network device is provided, comprising: a network interface port, which is configured to receive data symbols from a network node through a packet data network, at least some of the symbols being included in data packets; and a controller circuit, including a physical layer (PHY) circuit, the physical layer circuit including a receive PHY pipeline circuit, which is configured to process received data symbols, and a counter, which is configured to maintain a counter value indicating the number of data symbols in the receive PHY pipeline circuit.

[0015] Further in accordance with an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to identify a start-of-frame delimiter of one of the data packets; and calculate a timestamp of the identified start-of-frame delimiter.

[0016] Further according to an embodiment of the present disclosure, the data packet of the identified frame start delimiter includes a cyclic redundancy check (CRC) field filled with a first CRC code, wherein the controller circuit is configured to check the first CRC code, calculate a second CRC code in response to the data packet, the second CRC code being shorter than the first CRC code, and insert the second CRC code and the calculated timestamp into the CRC field to replace the first CRC code.

[0017] Additionally, according to an embodiment of the present disclosure, the data packet of the identified start-of-frame delimiter includes a time synchronization message received from a master clock in the packet data network.

[0018] In addition, according to an embodiment of the present disclosure, the device includes a hardware clock configured to maintain a clock value, wherein the receiving PHY pipeline circuit is configured to process the arrival of the received data symbols according to a given symbol timing, and in response to the clock value being reduced based on the counter value and the adjustment of the symbol timing, calculate the timestamp of when the identified frame start delimiter enters the receiving PHY pipeline circuit, and the controller circuit is configured to assign the calculated timestamp to one of the data packets of the identified frame start delimiter.

[0019] Further in accordance with an embodiment of the present disclosure, the controller circuit is configured to calculate an extended timestamp in response to the calculated timestamp and a current clock value of the hardware clock; and generate a control message including the extended timestamp.

[0020] Still further in accordance with an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to calculate the adjustment in response to the counter value divided by a symbol frequency of the symbol.

[0021] Additionally, according to an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to calculate the adjustment in response to the counter value multiplied by a symbol period of the symbol.

[0022] Furthermore, according to an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to process the symbols with a non-constant delay such that different corresponding ones of the symbols reside in the receive PHY pipeline circuit for respective different lengths of time.

[0023] Further according to an embodiment of the present disclosure, the counter is configured to: increment the counter value in response to some of the symbols entering the receive PHY pipeline circuit; and decrement the counter value in response to some of the symbols leaving the receive PHY pipeline circuit.

[0024] Still further in accordance with an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to set the counter value in response to an event and a given number of symbols, the event indicating the given number of symbols in the receive PHY pipeline circuit.

[0025] Additionally, according to an embodiment of the present disclosure, the event includes achieving block locking in the receive PHY pipeline circuit.

[0026] In addition, according to an embodiment of the present disclosure, the event includes: achieving locking of a forward error correction FEC block in the receiving PHY pipeline circuit; and the given number is the size of the FEC block.

[0027] Further in accordance with an embodiment of the present disclosure, the counter is configured to decrement the counter value in response to some of the symbols being consumed in the receive PHY pipeline circuit.

[0028] Still further in accordance with an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to decrement the counter value in response to when the consumed symbol reaches the counter along the receive PHY pipeline circuit if the consumed symbol is still in the receive PHY pipeline circuit.

[0029] In addition, according to an embodiment of the present disclosure, the receiving PHY pipeline circuit is configured to insert a corresponding control signal into the receiving PHY pipeline circuit at a corresponding position in the receiving PHY pipeline circuit, and remove a corresponding symbol of the consumed symbols from the receiving PHY pipeline circuit at the corresponding position.

[0030] According to another embodiment of the present disclosure, a computer system including a network device is also provided, the computer system including a network interface port, which is configured to receive data symbols from a network node through a packet data network, at least some of the symbols are included in a data packet; a hardware clock, which is configured to maintain a clock value; and a controller circuit, the controller circuit including a physical layer (PHY) circuit, the physical layer circuit including a receive PHY pipeline circuit, which is configured to: process the arrival of the received data symbols according to a given symbol timing; and identify a frame start delimiter of one of the data packets, and a counter for maintaining a counter value indicating the number of data symbols in the receive PHY pipeline circuit, wherein the receive PHY pipeline circuit is configured The controller circuit is configured to calculate a timestamp of when the identified frame start delimiter enters the receive PHY pipeline circuit in response to a clock value that is reduced based on the counter value and the first adjustment of the symbol timing, and the controller circuit is configured to assign the calculated timestamp to one of the data packets of the identified frame start delimiter, the data packet of the identified frame start delimiter includes a time synchronization message received from a master clock in the packet data network, and a host device including a processor, the processor being configured to execute software instructions to: receive the time synchronization message and calculate a second adjustment to the clock value of the hardware clock in response to the time synchronization message, wherein the hardware clock is configured to adjust the clock value in response to the calculated second adjustment.

[0031] In addition, according to an embodiment of the present disclosure, the counter is configured to increment the counter value in response to some of the symbols entering the receiving PHY pipeline circuit; and decrement the counter value in response to some of the symbols leaving the receiving PHY pipeline circuit.

[0032] Further in accordance with an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to set the counter value in response to an event and a given number of symbols, the event indicating the given number of symbols in the receive PHY pipeline circuit.

[0033] Still further in accordance with an embodiment of the present disclosure, the counter is configured to decrement the counter value in response to some of the symbols being consumed in the receive PHY pipeline circuit.

[0034] Additionally, according to an embodiment of the present disclosure, the receive PHY pipeline circuit is configured to: if the consumed symbol is still in the receive PHY pipeline circuit, decrement the counter value in response to when the consumed symbol will reach the counter along the receive PHY pipeline circuit.

[0035] According to another embodiment of the present disclosure, a networking method is also provided, comprising: receiving data symbols from a network node via a packet data network, at least some of the symbols being included in a data packet, and maintaining a clock value; processing the arrival of the received data symbols according to a given symbol timing, identifying a frame start delimiter of one of the data packets, maintaining a counter value indicating the number of data symbols in a receiving PHY pipeline circuit, calculating a timestamp of when the identified frame start delimiter enters the receiving PHY pipeline circuit in response to a clock value that decreases based on an adjustment of the counter value and the symbol timing, and assigning the calculated timestamp to one of the data packets with the identified frame start delimiter. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0037] Figure 1 is a block diagram of a computer system constructed and operative in accordance with an embodiment of the present invention;

[0038] Figure 2 yes Figure 1 A block diagram of a PHY receive pipeline circuit system processing symbols in a system;

[0039] Figure 3 yes Figure 1 A block diagram of a PHY receive pipeline circuit consuming symbols in a system;

[0040] Figure 4 Yes Figure 1 A flowchart of the steps in a counter resetting method in a system;

[0041] Figure 5A and Figure 5B Yes Figure 1 A flowchart of the steps in a method for calculating a timestamp in a system;

[0042] Figure 6 Yes Figure 1 A flowchart of steps in a method for assigning a calculated timestamp to a data packet in a system;

[0043] Figure 7 It is shown Figure 6 A block diagram of the method;

[0044] Figure 8 Yes Figure 1 A flowchart of steps in a method for extending a calculated timestamp in a system;

[0045] Fig. 9 It is shown Figure 8 A block diagram of the method;

[0046] Fig.10 yes Figure 1 A block diagram of a hardware clock in a system; and

[0047] Fig.11 Yes Figure 1 Flowchart of steps in a method for time synchronization in a system. DETAILED DESCRIPTION

[0048] As mentioned previously, successful operation of the PTP protocol typically requires accurate timestamping of received packets so that the timestamped point is not affected by network load or central processing unit (CPU) processing of the PTP packets. Therefore, hardware timestamping in a network interface controller (NIC) or other network node is typically better than software timestamping, which can suffer from CPU load errors.

[0049] In many applications, even hardware timestamping may not provide higher precision (e.g., up to picoseconds) because hardware processing does not process packets in a single step, but rather along a long and complex pipeline, which typically has a delay of about 1 microsecond.

[0050] Therefore, to eliminate errors caused by long and complex pipelines, it is desirable to calculate the timestamp as close to the beginning of the pipeline as possible before buffering and delays occur. At the receiving end of the device, the beginning of the pipeline is usually in the physical layer (PHY).

[0051] Timestamping early in the PHY is challenging because the PHY processes received symbols and not at the packet level. Only later in the PHY, after delays have occurred, can the frame delimiter (SFD) used to identify the start of a packet be identified. The IEEE 1588 standard defines the timestamp of a packet as the timestamp of the SRD of that packet. However, even if the start of a packet can be identified in the PHY, the nature of the packet, such as the relative nature of a PTP packet to other packets, is unknown. Furthermore, in the PHY layer, the arrival time of a symbol is not easily calculated because the raw bits are received before channel offset, scrambling, and forward error correction (FEC) encoding.

[0052] On the other hand, if timestamping is performed after the packets are framed, accuracy is affected because the latency of the pipeline may not be constant and may suffer from high jitter due to physical coding layer (PCS) marking, FEC parity, FEC store and forward, and any buffering in the pipeline.

[0053] Some network devices (e.g., NICs or switches) may include a constant-delay PHY pipeline. Therefore, a solution for devices with a constant-delay PHY pipeline is to calculate the SFD timestamp at the end of the PHY pipeline based on subtracting the constant delay of the PHY pipeline from the current time given by the local clock.

[0054] However, other network devices include variable latency PHY pipelines, which results in high performance and provides lower latency. However, for such variable latency devices, the above timestamp method may not provide sufficient accuracy when calculating the timestamp.

[0055] Embodiments of the present invention solve the above-mentioned problems by providing a network device that uses a counter to calculate the timestamp in the receiving PHY pipeline, and the counter maintains a counter value of the number of data symbols in the pipeline. After the SFD is identified, the timestamp of the SFD can be calculated toward the end of the PHY pipeline based on the current time (given by the local clock) and the time the data symbol of the SFD spends in the pipeline. Although the pipeline delay can be variable, the arrival of each data symbol is processed (e.g., input into the pipeline) according to a given symbol timing, for example, according to the symbol rate specified by the communication protocol operated by the network device. Therefore, the number of symbols in the pipeline given by the counter value and the given symbol timing can be used to calculate the time spent by the data symbol of the SFD in the pipeline. Then, the timestamp can be calculated based on the current time minus the time spent by the data symbol of the SFD in the pipeline. Therefore, the timestamp is calculated in the PHY layer, which is usually the last chance to obtain accurate delay information in the processing pipeline of the network device.

[0056] The PHY layer does not parse the packets, for example, does not recognize the PTP packets. Then after the PHY layer in the network device recognizes the packets, a timestamp can be added to the packets associated with the identified SFD. In some embodiments, a timestamp is added to each SFD so that each packet is accurately timestamped. Therefore, when the PTP packets are identified at a later stage of processing, each PTP packet has an accurate timestamp. In some embodiments, an existing packet field such as a cyclic redundancy check (CRC) field can be used to store the timestamp. The existing CRC code in the CRC field can be checked, and if the check is successful, a replacement shorter CRC code can be calculated. The replaced CRC code can then be stored in the CRC field together with the calculated timestamp.

[0057] In some embodiments, the calculated timestamp may include only the least significant bit (LSB) of the time associated with the SFD entering the pipeline. The most significant bit (MSB) of the time associated with the SFD entering the pipeline may be stored elsewhere. At a later stage, the stored value may be used to extend the calculated timestamp. The extended timestamp may be added to a control message to be forwarded to the host, for example, for accurate processing of PTP synchronization messages. In some embodiments, the MSB does not need to be explicitly saved, as the time of the local clock (when the timestamp is extended) based on the assumption that the delay between the calculated timestamp and the extended timestamp is short enough will include the MSB.

[0058] The counter is incremented for data symbols entering the PHY pipeline and decremented for data symbols leaving the PHY pipeline to the media access control (MAC) circuit. However, maintaining an accurate count of the number of data symbols in the pipeline is not an easy task. This is due to various factors, including data symbols such as service bits (e.g., tags, FEC parity, etc.) being consumed in the PHY pipeline and not exiting the PHY pipeline. The consumed data symbols are identified and accounted for by the counter. If the consumed data symbols are still in the PHY pipeline at this time, the counter is decremented for the consumed data symbols according to the time when the consumed data symbols leave the PHY pipeline. The corresponding control signal can be inserted at the corresponding position in the PHY pipeline, from which the corresponding consumed symbols are removed from the PHY pipeline.

[0059] Another factor is that before block lock is achieved, the PHY pipeline knows when and how to reset the counters before receiving symbols, but symbols are typically not released from the pipeline until after block lock is achieved. Block lock is defined as completing the link process, which may include locking to the incoming bit stream, aligning to block start and end positions, and ramping across multiple lanes. Therefore, running the counters before block lock can easily result in counter overruns and / or meaningless counter values. Therefore, the counters are typically not run from a start state when it may be known that the pipeline is empty. Instead, the counters are reset in response to identifying an event associated with a known value (or a good estimate of that value) for the number of symbols in the PHY pipeline. One such event is achieving FEC lock when the number of symbols in the PHY pipeline is known to be equal to or very close to the known size of the FEC block associated with FEC lock.

[0060] System Description

[0061] Documents incorporated herein by reference shall be considered to be an integral part of this application, and except where any term is defined in these incorporated documents in a way that contradicts an explicit or implicit definition in this specification, only the definitions in this specification shall apply.

[0062] Reference now Figure 1 , which is a block diagram of a computer system 10 constructed and operated in accordance with an embodiment of the present invention. The computer system 10 includes a network device 12 (e.g., a NIC) connected to a host computer 14. The host computer 14 includes a processor 16 to perform various processing tasks, optionally including running PTP software 18, which will be referred to as Fig.10 and Fig.11 Described in more detail. In practice, some or all of the functions of processor 16 may be combined in a single physical component, or alternatively, implemented using multiple physical components. These physical components may include hardwired or programmable devices, or a combination of both. In some embodiments, at least some of the functions of processor 16 may be performed by a programmable processor under the control of appropriate software. The software may be downloaded to the device in electronic form, for example, over a network. Alternatively or additionally, the software may be stored in a tangible, non-transitory computer-readable storage medium, such as an optical, magnetic, or electronic memory.

[0063] In some implementations, the functionality of network device 12 and host computer 14 may be combined in any suitable network device, such as a network switch including a CPU running PTP software.

[0064] The network device 12 includes a network interface port 24 , a hardware clock 26 , a buffer 28 , a controller circuit 30 , and a host interface 32 .

[0065] The network interface port 24 is configured to receive data symbols from the network node 20 via the packet data network 22. At least some of the symbols are included in data packets. In some embodiments, the network interface port 24 is configured to receive data symbols included in a PTP synchronization message from the master clock device 34.

[0066] Hardware clock 26 is configured to maintain a clock value according to any suitable time standard, such as Coordinated Universal Time (UTC). Fig.10 and Fig.11 Hardware clock 26 is described in greater detail. Buffer 28 is configured to store data used by various elements of network device 12. Host interface 32 is configured to transfer data between network device 12 and host computer 14. Host interface 32 may include any suitable interface, such as, but not limited to, a Peripheral Component Interconnect Express (PCIe) interface.

[0067] Controller circuit 30 includes physical layer (PHY) circuit 36 ​​and MAC circuit 38. PHY circuit 36 ​​is configured to process data symbols at the physical layer. MAC circuit 38 is configured to process packets at the data link layer in the receive and transmit pipelines (not shown). Controller circuit 30 may perform other forwarding and processing functions.

[0068] The PHY circuit 36 ​​includes a receive PHY pipeline circuit 40 and a transmit pipeline circuit 42 for processing received data symbols and data symbols to be transmitted, respectively. The PHY circuit 36 ​​also includes a counter 44 to maintain a counter value indicating the number of data symbols in the receive PHY pipeline circuit 40. Figures 2 to 5B The receive PHY pipeline circuit 40 is described in more detail.

[0069] Reference now Figure 2 , which is in Figure 1 1 is a block diagram of a receive PHY pipeline circuit 40 for processing data symbols 46 in the system 10. Figure 2 4. A data symbol 46 is shown entering the PHY pipeline circuit 40 on the left and exiting the pipeline on the right. A counter 44 is coupled to the receive PHY pipeline circuit 40 and is configured to receive data from the network interface port 24 in response to the symbol 46. Figure 1 )) enters the receive PHY pipeline circuit 40 to increment the counter value, and exits the receive PHY pipeline circuit 40 (towards the MAC circuit 38 (in response to the symbol 46) Figure 1 )) and decrement the counter value.

[0070] Figure 2One of the data symbols 46 identified as an SFD 48 is shown. The receive PHY pipeline circuitry 40 is typically configured to detect every SFD, and may even discard those SFDs in the MAC circuitry 38. Detection of the SFD may be performed at the last stage in the receive PHY pipeline circuitry 40. The symbol rate at which the data symbols 46 enter the receive PHY pipeline circuitry 40 is constant and known. Therefore, at the end of the receive PHY pipeline circuitry 40, the time t that the data symbol 46 of the SFD 48 has spent in the receive PHY pipeline circuitry 40 is proportional to the number of data symbols 46 currently in the pipeline as given by the counter value maintained by the counter 44. Because the symbol rate is known, the time t can be calculated in response to the current counter value and the symbol rate. The time t can be used to calculate the time at which the identified SFD 48 entered the receive PHY pipeline circuitry 40, as shown in reference to FIG. Figure 5A and Figure 5B Described in more detail.

[0071] Reference now Figure 3 , which is consumption Figure 1 10. Some data symbols 46 entering the receive PHY pipeline circuit 40 are processed by the receive PHY pipeline circuit system 40 and sent to the MAC circuit 38 ( Figure 1 ) exit the receive PHY pipeline circuitry 40. Other symbols 46 enter the receive PHY pipeline circuitry 40 and are consumed within the receive PHY pipeline circuitry 40 and do not exit the receive PHY pipeline circuitry 40 toward the MAC circuitry 38. Some of the consumed symbols 50 are Figure 3 5. As described in more detail below, the consumed symbols 50 are replaced by control signals 52. Examples of consumed symbols 50 are service bits (e.g., tags, FEC parity bits, etc.). The consumed symbols 50 may be detected based on the presence of other data. For example, when an FEC block is detected, the receive PHY pipeline circuitry 40 knows that there are parity symbols that are removed, so the receive PHY pipeline circuitry 40 will compensate for the removed FEC parity symbols, for example, by adding control signals 52 to the receive PHY pipeline circuitry 40. Tags may be physical layer symbols that are consumed by the receive PHY pipeline circuitry 40 and are not forwarded to the MAC circuitry 38.

[0072] When the control signal 52 reaches the end of the receive PHY pipeline circuit 40, the control signal 52 will be counted as a symbol leaving the receive PHY pipeline circuit 40, and the counter value will be decremented accordingly.

[0073] Thus, the receive PHY pipeline circuitry 40 is configured to decrement the counter value in response to when the consumed symbol 50 will reach the counter 44 along the receive PHY pipeline circuitry 40, if the consumed symbol 50 is still in the receive PHY pipeline circuitry 40 at this time. In some embodiments, the receive PHY pipeline circuitry 40 is configured to insert a corresponding control signal 52 into the receive PHY pipeline circuitry 40 at a corresponding position in the receive PHY pipeline circuitry 40 where the corresponding consumed symbol 50 is removed from the receive PHY pipeline circuitry 40. The counter 44 is configured to decrement the counter value in response to the symbol 50 being consumed in the receive PHY pipeline circuitry 40.

[0074] Reference now Figure 4 , which includes Figure 1 60 of the steps in a counter reset method in the system 10 of FIG. 6. Because the receive PHY pipeline circuit 40 receives the symbol 46 before block lock is achieved, but the symbol 46 is generally not released from the receive PHY pipeline circuit 40 until after block lock is achieved, it is known when and how to reset the counter 44 ( Figure 3 ) is meaningful. Therefore, running the counter 44 before block lock can easily result in counter overruns and / or meaningless counter values ​​because the counter is not decremented before block lock. Therefore, the counter 44 is typically not run from a start-up state when it is known that the receive PHY pipeline circuit 40 is empty. Instead, the counter 44 is reset in response to identifying an event associated with a known value (or a good estimate of the value) of the number of symbols 46 in the receive PHY pipeline circuit 40. One such event is when FEC lock is achieved when some markers are identified and the entire FEC block is accumulated to check the parity of the FEC block in order to achieve FEC lock. Therefore, when FEC lock is achieved, it is known that the number of symbols in the receive PHY pipeline circuit 40 is equal to or very close to the known size of the FEC block accumulated in the receive PHY pipeline circuit 40. Another such event is when block lock is achieved when it can be estimated how many data symbols are in each lane of the receive PHY pipeline circuit 40. The counter 44 is then reset for that event to the known or estimated number of symbols 46 in the receive PHY pipeline circuit 40. Typically, certain events provide a good estimate of the number of bits within the receive PHY pipeline circuit 40, depending on the protocol.

[0075] Thus, the receive PHY pipeline circuitry 40 is configured to detect (block 62) an event (e.g., achieving block lock in the receive PHY pipeline circuitry 40, or achieving lock of an FEC block in the receive PHY pipeline circuitry 40) indicating a given number of symbols 46 (e.g., the size of an FEC block) in the receive PHY pipeline circuitry 40, and to set (block 64) a counter value in response to detecting the event, and to set the counter value in response to a given (known or estimated) number of symbols 46 in the receive PHY pipeline circuitry 40 associated with the detected event).

[0076] Figure 5A and Figure 5B Yes Figure 1 Flowcharts 70, 81 of the steps in the time stamp calculation method in the system 10. Now refer to Figure 5A . Also refer to Figure 3 .

[0077] The receive PHY pipeline circuitry 40 is configured to process the arrival of received data symbols according to a given symbol timing. The receive PHY pipeline circuitry 40 is configured to process the data symbols 46 at the physical layer (block 72). The receive PHY pipeline circuitry 40 is configured to process the symbols 46 with a non-constant delay such that different respective symbols 46 stay in the receive PHY pipeline circuitry 40 for respective different lengths of time.

[0078] The sub-steps of the step of block 72 are now described in more detail. The receive PHY pipeline circuitry 40 is configured to receive a signal from the network interface port 24 ( Figure 1 ) receive data symbols 46 (block 74). The receive PHY pipeline circuitry 40 is configured to consume some of the data symbols 46 (block 76). The receive PHY pipeline circuitry 40 is configured to insert (block 78) a corresponding control signal 52 into the receive PHY pipeline circuitry 40 at a corresponding position in the receive PHY pipeline circuitry 40 from which the corresponding consumed symbol 50 is removed from the receive PHY pipeline circuitry 40. In this manner, the receive PHY pipeline circuitry 40 is configured to decrement the counter value in response to when the consumed symbol 50 reaches the counter 44 along the receive PHY pipeline circuitry 40 if the consumed symbol is still in the receive PHY pipeline circuitry 40. The receive PHY pipeline circuitry 40 is configured to forward (block 80) the data symbols 46 that are not consumed from the receive PHY pipeline circuitry 40 to the MAC circuitry 38.

[0079] The steps of blocks 74 to 80 are typically performed at and / or approximately at the same time. For example, when a data symbol 46 enters the receive PHY pipeline circuit 40, other data symbols 46 are being processed and / or consumed by the receive PHY pipeline circuit 40, and other data symbols 46 are being sent to the MAC circuit 38 ( Figure 1) leaves the receive PHY pipeline circuit 40.

[0080] Reference now Figure 5B . Also refer to Figure 3 The receive PHY pipeline circuitry 40 is configured to identify (block 82) an SFD of one of the data packets. The receive PHY pipeline circuitry 40 is configured to calculate (block 84) a timestamp of when the identified SFD enters the receive PHY pipeline circuitry 40 in response to a current clock value (maintained by the hardware clock 26) and decremented based on the counter value and the adjustment of the symbol timing. In some embodiments, the receive PHY pipeline circuitry 40 is configured to calculate the adjustment in response to the counter value divided by the symbol frequency of the symbols 46. In other embodiments, the receive PHY pipeline circuitry 40 is configured to calculate the adjustment in response to the counter value divided by the symbol period of the symbol.

[0081] The receive PHY pipeline circuit 40 is configured to save (block 86) the timestamp to the buffer 28 ( Figure 1 ). The timestamp is then retrieved from buffer 28 and added to the packet of the identified SFD, as shown in reference Figure 6 and Figure 7 The steps of blocks 82-86 are repeated for subsequent SFDs as described in more detail.

[0082] The timestamp may be represented by the LSB of the current time adjusted by a calculated adjustment (e.g., based on the counter value and the symbol rate). The MSB of the current time adjusted by the calculated adjustment may be added to the timestamp at a later stage, as described in reference to Figure 8 and Fig. 9 Described in more detail. The most significant bit (MSB) of the current time adjusted by the calculated adjustment can be stored in other locations. In some embodiments, the MSB does not need to be explicitly saved, because based on the assumption that the delay between the calculated timestamp and the extended timestamp is short enough, the current time of the hardware clock 26 (when the timestamp is extended in a later stage) will include the MSB.

[0083] Reference now Figure 6 and Figure 7 , Figure 6 Yes Figure 1 Flowchart 88 of steps in a method of assigning a calculated timestamp to a data packet 90 in the system 10 of FIG.

[0084] When the identified SFD exits the receive PHY pipeline circuit 40, and typically before entering the MAC circuit 38 ( Figure 1 ) before the controller circuit 30 ( Figure 1 ) is configured as a slave buffer 28 ( Figure 1) retrieves (block 94) the calculated timestamp 100. The controller circuit 30 is configured to assign (block 96) the calculated timestamp 100 to the data packets 90 of the identified SFD. One reason for adding timestamps to data packets associated with the identified SFD is to allow full line rate (wire speed) of packets with accurate timestamps.

[0085] In order to add the timestamp 100 to the data packet 90 without extending the data path, the calculated timestamp 100 can be added to an existing field of the data packet 90, such as the CRC field 92. For example, assume that the CRC field 92 has a length of 32 bits and the calculated timestamp 100 has a length of 26 bits. However, in some embodiments, the CRC field 92 and the calculated timestamp 100 can have any suitable length.

[0086] The CRC field 92 is filled with an existing CRC code (eg, 32 bits). Thus, the existing CRC code may be replaced by a shorter CRC code 98 (eg, 6 bits) and a calculated timestamp 100 (eg, 26 bits).

[0087] Thus, the controller circuit 30 is configured to: check (box 102) an existing CRC code; calculate (box 104) a shorter replacement CRC code 98 in response to the data packet (using any suitable CRC algorithm); and insert the shorter replacement CRC code 98 and the calculated timestamp 100 into the CRC field 92 to replace the existing CRC code.

[0088] Reference now Figure 8 and Fig. 9 , Figure 8 is included in Figure 1 Flowchart 106 of steps in a method of extending a calculated timestamp 100 in the system 10. Referring now to Fig. 9 , which is shown Figure 8 Block diagram of the method.

[0089] The controller circuit 30 is configured to receive (block 108) a data packet 90 having a calculated timestamp 100 (eg, 26 bits) ( Figure 7 ). The controller circuit 30 is configured to receive the hardware clock 26 ( Figure 1) retrieves (block 110) a current clock value 118. The MSBs (e.g., 64 bits) of the current clock value 118 are selected from the retrieved current clock value 118 (including the MSB of the time when the data packet 90 enters the receive PHY pipeline circuit 40). The controller circuit 30 is configured to calculate (block 112) an extended timestamp 120 (e.g., 80 bits) in response to the calculated timestamp 100 and the MSB of the current clock value 118 of the hardware clock 26. The controller circuit 30 is configured to generate (block 114) a control message 122 including the extended timestamp 120 and provide (block 116) the control message 122 to the host computer 14 ( Figure 1 ).

[0090] The above steps of blocks 108 - 116 may be performed at any suitable location in the controller circuit 30 .

[0091] Reference now Fig.10 and Fig.11 , Fig.10 yes Figure 1 1. Block diagram of the hardware clock 26 in the system 10. Fig.11 Yes Figure 1 Flowchart 130 of steps in a method of time synchronization in system 10 .

[0092] The hardware clock 26 may include an oscillator 124, a phase-locked loop 126, and a register 128. The oscillator 124 drives the phase-locked loop 126, which provides adjustments to the frequency provided by the oscillator 124. The frequency adjustment is provided by the phase-locked loop 126 to the register 128, which maintains the current time according to any suitable standard, such as UTC. The current time can then be provided from the register 128 to the receive PHY pipeline circuit 40 and other components of the network device 12 as needed.

[0093] Some packets processed by network device 12 are received from master clock device 34 ( Figure 1 ) received PTP synchronization message or other time synchronization message. The message is sent by PTP software 18 ( Figure 1 ) processing, as described in more detail below.

[0094] The processor 16 of the host computer 14 ( Figure 1 ) is configured to run software instructions of the PTP software 18 to: receive (block 132) a time synchronization message; in response to the time synchronization message, calculate (block 134) an adjustment 138 to the clock value (e.g., drift) of the hardware clock 26, and provide (block 136) the adjustment 138 to the hardware phase-locked loop 126. The phase-locked loop 126 of the hardware clock 26 is configured to adjust the clock value stored in the register 128 in response to the calculated adjustment 138.

[0095] For the sake of clarity, various features of the invention described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity, various features of the invention described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.

[0096] The above embodiments are cited by way of example, and the present invention is not limited to what has been specifically shown and described above. On the contrary, the scope of the present invention includes combinations and sub-combinations of the various features described above, as well as variations and modifications that will occur to those skilled in the art after reading the foregoing description and which are not disclosed in the prior art.

Claims

1. A network device, comprising: a network interface port configured to receive data symbols from a network node over a packet data network, at least some of the symbols being included in data packets; a hardware clock configured to maintain a clock value; and A controller circuit, the controller circuit comprising a physical layer (PHY) circuit, the physical layer circuit comprising: a receive PHY pipeline circuit configured to process received data symbols according to a given symbol timing; and a counter configured to maintain a counter value indicating the number of data symbols in the receive PHY pipeline circuit, wherein the receive PHY pipeline circuit is configured to: identify a frame start delimiter of one of the data packets; and in response to the clock value being reduced based on the counter value and the adjustment of the symbol timing, calculate a timestamp of when the identified frame start delimiter enters the receive PHY pipeline circuit, the controller circuit being configured to assign the calculated timestamp to one of the data packets of the identified frame start delimiter.

2. The apparatus of claim 1 , wherein the data packet of the identified start-of-frame delimiter comprises a cyclic redundancy check (CRC) field populated with a first CRC code, wherein the controller circuit is configured to: checking the first CRC code; responsive to the data packet, calculating a second CRC code, the second CRC code being shorter than the first CRC code; and The second CRC code and the calculated timestamp are inserted into the CRC field to replace the first CRC code.

3. The apparatus of claim 1, wherein the data packet of the identified start-of-frame delimiter comprises a time synchronization message received from a master clock in the packet data network.

4. The apparatus of claim 1 , wherein the controller circuit is configured to: calculating an extended timestamp in response to the calculated timestamp and a current clock value of the hardware clock; and A control message including the extended timestamp is generated.

5. The apparatus of claim 1, wherein the receive PHY pipeline circuit is configured to calculate the adjustment in response to the counter value divided by a symbol frequency of the symbol.

6. The apparatus of claim 1, wherein the receive PHY pipeline circuit is configured to calculate the adjustment in response to the counter value multiplied by a symbol period of the symbol.

7. The apparatus of claim 1, wherein the receive PHY pipeline circuit is configured to process the symbols with a non-constant delay such that different corresponding ones of the symbols reside in the receive PHY pipeline circuit for respective different lengths of time.

8. The apparatus of claim 1 , wherein the counter is configured to: increment the counter value in response to some of the symbols entering the receive PHY pipeline circuit; and decrement the counter value in response to some of the symbols leaving the receive PHY pipeline circuit.

9. The apparatus of claim 8, wherein the receive PHY pipeline circuit is configured to set the counter value in response to an event and the given number of symbols, the event indicating the given number of symbols in the receive PHY pipeline circuit.

10. The apparatus of claim 9, wherein the event comprises achieving block lock in the receive PHY pipeline circuitry.

11. The apparatus of claim 9, wherein: The event includes: achieving lock on a forward error correction (FEC) block in the receive PHY pipeline circuit; and the given number is a size of the FEC block.

12. The apparatus of claim 8, wherein the counter is configured to decrement the counter value in response to some of the symbols being consumed in the receive PHY pipeline circuit.

13. The apparatus of claim 12, wherein the receive PHY pipeline circuit is configured to decrement the counter value in response to when the consumed symbol will reach the counter along the receive PHY pipeline circuit if the consumed symbol is still in the receive PHY pipeline circuit.

14. The apparatus of claim 13, wherein the receive PHY pipeline circuit is configured to insert a corresponding control signal into the receive PHY pipeline circuit at a corresponding position in the receive PHY pipeline circuit, and to remove a corresponding symbol of the consumed symbols from the receive PHY pipeline circuit at the corresponding position.

15. A computer system comprising: A network device, comprising: a network interface port configured to receive data symbols from a network node over a packet data network, at least some of the symbols being included in a data packet; a hardware clock configured to maintain a clock value; and a controller circuit comprising a physical layer (PHY) circuit, the physical layer circuit comprising: a receive PHY pipeline circuit configured to: process the arrival of received data symbols according to a given symbol timing; and identify a start-of-frame delimiter of one of the data packets; and a counter for maintaining a counter value indicative of the number of data symbols in the receive PHY pipeline circuit, wherein: the receive PHY pipeline circuitry being configured to calculate a timestamp of when the identified start of frame delimiter enters the receive PHY pipeline circuitry in response to the clock value being decremented based on the counter value and the first adjustment of the symbol timing; and The controller circuit is configured to assign a calculated timestamp to one of the data packets of an identified start-of-frame delimiter, the data packet of the identified start-of-frame delimiter including a time synchronization message received from a master clock in the packet data network; and A host device includes a processor configured to execute software instructions to: receive the time synchronization message and calculate a second adjustment to the clock value of the hardware clock in response to the time synchronization message, wherein the hardware clock is configured to adjust the clock value in response to the calculated second adjustment.

16. The system of claim 15, wherein the counter is configured to: increment the counter value in response to some of the symbols entering the receive PHY pipeline circuit; and decrement the counter value in response to some of the symbols leaving the receive PHY pipeline circuit.

17. The system of claim 16, wherein the receive PHY pipeline circuit is configured to set the counter value in response to an event and the given number of symbols, the event indicating the given number of symbols in the receive PHY pipeline circuit.

18. The system of claim 16, wherein the counter is configured to decrement the counter value in response to some of the symbols being consumed in the receive PHY pipeline circuit.

19. The system of claim 18, wherein the receive PHY pipeline circuit is configured to decrement the counter value in response to when the consumed symbol will reach the counter along the receive PHY pipeline circuit if the consumed symbol is still in the receive PHY pipeline circuit.

20. A networking method, comprising: receiving data symbols from a network node over a packet data network, at least some of the symbols being included in data packets; Maintain the clock value; processing the arrival of received data symbols according to a given symbol timing; identifying a start of frame delimiter of one of said data packets; maintaining a counter value indicating a number of said data symbols in a receive PHY pipeline circuit; responsive to the clock value being decremented based on the counter value and the adjustment of the symbol timing, calculating a timestamp of when the identified start of frame delimiter enters the receive PHY pipeline circuit; as well as The calculated timestamp is assigned to one of the data packets of the identified start-of-frame delimiter.

Images