Time synchronization method and device of computing node, computing node and cluster system

Abstract

The invention provides a time synchronization method and device of a computing node, the computing node and a cluster system, and belongs to the technical field of computers, and the method comprises the following steps: recovering a frequency synchronization signal from a physical layer of a network interface on the computing node; analyzing the received network protocol message to obtain a phase synchronization signal; the frequency synchronization signal and the phase synchronization signal serve as input of a clock synthesis circuit at the same time, so that the clock synthesis circuit locks the frequency of an output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock of the computing node, and providing the reference clock as a unified time source to at least one component of the compute node. According to the invention, the fast frequency locking capability of the physical layer and the precise phase alignment capability of the protocol layer are combined to generate the reference clock, the time synchronization precision of the clock synthesis mechanism is relatively high, the error is relatively small, and a stable and reliable nanosecond time reference is provided for the computing node.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a time synchronization method, apparatus, computing node, and cluster system for computing nodes. Background Technology

[0002] In applications such as Artificial Intelligence (AI) training, distributed databases, and real-time data processing, numerous server nodes (such as servers equipped with Graphics Processing Units (GPUs)) need to work collaboratively. To ensure data consistency, the correctness of computation results, and the accuracy of event sequencing, the time base of all nodes within the cluster must be highly unified, and the synchronization accuracy requirements have increased from the traditional millisecond and microsecond levels to the stringent nanosecond level.

[0003] To achieve the aforementioned time synchronization, a widely used existing technology is the Network Time Protocol (NTP). In this scheme, the server node acts as a client, periodically sending time request messages to the NTP server in the network through its network interface. Upon receiving a response message from the NTP server, the NTP software in the client's operating system kernel or user-space application estimates network latency and local deviation by combining the timestamp information carried in the message with the local timestamp. Subsequently, the software algorithm gradually adjusts the local software clock or hardware clock to synchronize it with the NTP server.

[0004] The existing network time protocol-based solutions described above are inherently architecturally incapable of meeting nanosecond-level synchronization accuracy requirements. First, their timestamp generation and message processing rely entirely on general-purpose operating system software stacks. Software-level latency jitter fundamentally limits their synchronization accuracy, achieving only milliseconds or slightly higher microseconds at best. Second, these solutions lack a physical-level frequency synchronization mechanism. Clock frequency deviations can only be corrected through periodic, reactive adjustments in the protocol software, making it difficult for nodes to achieve stable clock frequency locking and resulting in significant accumulated errors. Therefore, at the architectural level, these solutions cannot provide sufficiently accurate and stable time synchronization support for high-performance computing scenarios such as GPU clusters that require nanosecond-level time bases. Summary of the Invention

[0005] This application provides a time synchronization method, apparatus, computing node, and cluster system for computing nodes, aiming to solve the problems of low time synchronization accuracy and large error in existing time synchronization schemes based on network time protocols, which cannot meet the requirements of nanosecond-level synchronization accuracy.

[0006] Firstly, this application provides a time synchronization method for computing nodes, including: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node; Parse the received network protocol messages to obtain the phase synchronization signal; The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, which locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node. The reference clock is then provided as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0007] In one embodiment, recovering the frequency synchronization signal from the physical layer of the network interface on the computing node includes: The Ethernet physical layer chip of the network interface recovers the clock frequency signal from the data stream of the physical link as the frequency synchronization signal.

[0008] In one embodiment, parsing the received network protocol messages to obtain a phase synchronization signal includes: The network protocol messages are parsed using a protocol stack based on a precise time protocol to generate a pulse signal per second, which serves as the phase synchronization signal.

[0009] In one embodiment, the network interface is a network interface that supports remote direct memory access.

[0010] In one embodiment, the time synchronization method for computing nodes further includes: If the frequency synchronization signal is detected to be lost, the frequency synchronization signal is obtained based on the frequency information extracted from the network protocol message.

[0011] In one embodiment, the time synchronization method for computing nodes further includes: In the event of loss of the phase synchronization signal, the computing node maintains frequency synchronization with the network based on the frequency synchronization signal.

[0012] Secondly, this application also provides a time synchronization device for computing nodes, comprising: A frequency synchronization module is used to recover a frequency synchronization signal from the physical layer of the network interface on a computing node; wherein the computing node includes one or more of a graphics processor, tensor processor, neural network processor, deep learning processor, accelerator processor, and general-purpose graphics processor. The phase synchronization module is used to parse received network protocol messages and obtain phase synchronization signals. A reference clock generation module is used to simultaneously use the frequency synchronization signal and the phase synchronization signal as inputs to a clock synthesis circuit, so that the clock synthesis circuit locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference, thereby generating a reference clock for the computing node, and providing the reference clock as a unified time source to at least one component of the computing node.

[0013] Thirdly, this application also provides a computing node, including a network interface and a clock synthesizer; The network interface is configured to recover the frequency synchronization signal from the physical layer of the network interface and use the protocol stack of the Precision Time Protocol to parse the received network protocol messages to obtain the phase synchronization signal. The clock synthesizer is configured to simultaneously take the frequency synchronization signal and the phase synchronization signal as inputs, lock the frequency of the output clock with the frequency synchronization signal, and calibrate the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node, and provide the reference clock as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0014] Fourthly, this application also provides a cluster system, the cluster system including multiple computing nodes, each computing node being used to execute the above-described computing node time synchronization method; or, The computing node is the computing node described above.

[0015] Fifthly, this application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned time synchronization methods for computing nodes. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the flowcharts illustrating the time synchronization method for computing nodes provided in this application; Figure 2 This is a schematic diagram of the network structure of the server cluster provided in this application; Figure 3 This is an application example of the reference clock provided in this application; Figure 4 This is the second flowchart illustrating the time synchronization method for computing nodes provided in this application; Figure 5 This is a schematic diagram of the time synchronization device for the computing node provided in this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0020] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0021] The following description, in conjunction with the accompanying drawings, describes the time synchronization method, apparatus, computing node, and cluster system for computing nodes provided in this application.

[0022] It should be noted that the time synchronization method for computing nodes provided in this application is implemented based on the time synchronization device of computing nodes. The time synchronization method of computing nodes combines the fast frequency locking capability of the physical layer with the precise phase alignment capability of the protocol layer. The reference clock is generated through the collaborative work of the two. Compared with the time synchronization scheme based on the network time protocol, this clock synthesis mechanism has higher time synchronization accuracy and smaller error. It can provide computing nodes with a stable and reliable nanosecond-level time reference, effectively solving the performance bottleneck and data consistency problem caused by time asynchrony in large-scale distributed computing.

[0023] This application describes the time synchronization method of a computing node using a time synchronization device of a computing node as an example.

[0024] Figure 1 This is one of the flowcharts illustrating the time synchronization method for computing nodes provided in this application. For example... Figure 1 As shown, the time synchronization method for computing nodes provided in this application includes: S110: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node.

[0025] A computing node is any server or processing unit in a cluster system that needs to perform collaborative computing. Specifically, a computing node includes any one or more of the following: Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Neural Network Processing Unit (NPU), Deep Learning Processing Unit (DPU), Accelerated Processing Unit (APU), and General-Purpose Graphics Processing Unit (GPGPU).

[0026] In one possible implementation, the computing node could be an AI server configured with a graphics processing unit (GPU), such as... Figure 2 In a server cluster, multiple computing nodes form a server cluster, interconnected through a multi-layered switching network (such as access switches (ASW), aggregation switches (PSW), and data center switches (DSW)) to jointly complete large-scale computing tasks.

[0027] The compute node is equipped with a network interface, which serves as the physical gateway for communication between the compute node and the external network. The physical layer of the network interface is the lowest layer of the network protocol stack, responsible for handling the transmission of the raw bit stream. The frequency synchronization signal recovered from the physical layer can accurately characterize the periodic electrical signal of the network master clock frequency. Therefore, frequency synchronization can be achieved directly by extracting time information at the physical layer of the communication link, without relying on the message content of higher-level protocols.

[0028] S120: Parse the received network protocol messages to obtain the phase synchronization signal.

[0029] The network interface of a compute node not only receives data from the physical layer, but also reassembles this data into network protocol messages and passes them up to the protocol stack. Network protocol messages are data packets that follow a specific time synchronization protocol (such as Precision Time Protocol (PTP) or NTP) format, and they usually carry precise timestamp information.

[0030] The process of parsing network protocol messages is typically performed by protocol stack software running on the processor of a compute node. This process includes identifying the message type, extracting the timestamp field, and performing calculations according to protocol rules to eliminate the impact of network transmission latency.

[0031] The phase synchronization signal obtained by parsing network protocol messages can accurately pinpoint a specific point in time (usually the start of a second), and it carries the phase information of absolute time.

[0032] S130: The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, so that the clock synthesis circuit locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node, and provides the reference clock as a unified time source to at least one component of the computing node.

[0033] The frequency synchronization signal and the phase synchronization signal are used as inputs to the clock synthesis process. The reference clock generated based on these two signals is subject to both frequency and phase constraints. Therefore, the reference clock is a high-quality clock signal with both high frequency stability and accurate phase alignment. This signal will serve as a unified time source for all components within the computing node that require a precise time reference.

[0034] In one possible implementation, the clock synthesis circuit can be a system-level phase-locked loop (PLL). The phase-frequency detector inside this PLL compares the input frequency synchronization signal with the output of the voltage-controlled oscillator to lock the frequency of the output clock; simultaneously, it uses the input phase synchronization signal as a phase reference to precisely calibrate the phase of the output clock. Through this dual-locking mechanism, the system-level PLL can generate a reference clock with extremely low jitter and a frequency and phase highly synchronized with the network master clock.

[0035] The phase-locked loop (PLL) is mounted on the universal baseboard (UBB) of the computing node. When the system is first powered on, because the network link is not yet established or stable, effective frequency and phase synchronization signals cannot be obtained. At this time, the PLL operates in free-running mode based on the clock signal CLK provided by the local crystal oscillator (OSC), providing a temporary reference clock signal. After the physical layer of the network interface is functioning normally, the system begins to recover frequency and phase synchronization signals from the network, and the PLL switches to locked mode, entering a high-precision synchronization state. At this point, please refer to... Figure 3The central processing unit (CPU) of the computing node sequentially obtains the data stream from the external network through the eight-channel Octal Small Form-factor Pluggable (OSFP) interface on the universal baseboard UBB, the Ethernet re-timer, the OCP Accelerator Module (OAM), the PCIe (Peripheral Component Interconnect Express) re-timer, and the PCIe bus and PCIe switches outside the universal baseboard UBB, in order to obtain the frequency synchronization signal (see...). Figure 3 Eth_refclk in the context of phase synchronization signals (see Eth_refclk in the context of phase synchronization signals) and phase synchronization signals (see Eth_refclk in the context of phase synchronization signals). Figure 3 The 1pps_out in the clock ultimately generates the reference clock (156.25MHz).

[0036] The phase-locked loop (PLL) then outputs the generated high-precision reference clock (156.25MHz) to the OAM and at least one component within the compute node. This at least one component is a computational core or functional module with high time precision requirements. In the context of AI servers, this at least one component typically includes any one or more of the following: a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processing unit (DPU), an accelerator processing unit (APU), and a general-purpose graphics processing unit (GPGPU).

[0037] Figure 3 An application example of a reference clock is shown. For example... Figure 3 As shown, the 156.25MHz reference clock generated by the phase-locked loop (PLL) on the general-purpose UBB substrate is distributed to the central processing unit (CPU) of multiple GPUs. By providing a unified and high-precision time base for the GPUs, it is ensured that the operations of multiple GPUs are strictly aligned when performing large-scale parallel computing tasks (such as gradient aggregation in data-parallel training), thereby avoiding computational errors and improving overall computing performance and efficiency.

[0038] The reference clock can also be provided to other components within the compute node, such as OAM that supports Remote Direct Memory Access (RDMA). Figure 3 As shown), to ensure the accuracy of timestamps for its network operations.

[0039] This application combines the fast frequency locking capability of the physical layer with the precise phase alignment capability of the protocol layer. By working together, a reference clock is generated. Compared with time synchronization schemes based on network time protocols, this clock synthesis mechanism has higher time synchronization accuracy and smaller error. It can provide computing nodes with a stable and reliable nanosecond-level time reference, effectively solving the performance bottleneck and data consistency problems caused by time asynchrony in large-scale distributed computing.

[0040] In one possible implementation, recovering the frequency synchronization signal from the physical layer of the network interface on the computing node includes: The Ethernet physical layer chip of the network interface recovers the clock frequency signal from the data stream of the physical link as the frequency synchronization signal.

[0041] Synchronous Ethernet (SyncE) is a standard technology for transmitting synchronous clock information at the Ethernet physical layer. The frequency information of the master clock in the network (e.g., located on the core switch) is passed cascadingly through physical links to every node in the network. The Ethernet physical layer chip on the network interface (see...) Figure 4 The EthPHY in the Ethernet module is the key hardware for implementing Synchronous Ethernet (SyncE) technology. It is responsible for processing electrical or optical signals on the Ethernet link. For example, when a switch supporting SyncE sends data to a GPU computing node via Ethernet, the bit rate of the transmitted data stream is controlled by its internal high-precision clock source, thus the data stream carries precise frequency information.

[0042] Please combine Figure 4 After receiving the data stream from the physical link, the Ethernet physical layer chip (Eth PHY) of the compute node uses a phase-locked loop (PLL) to lock the bit rate of the data stream, ensuring correct data decoding. During the locking process, the voltage-controlled oscillator (VCO) of the CDR circuit adjusts its output frequency to perfectly match the frequency of the received data stream. This recovered, stable clock signal, synchronized with the upstream device's clock, is the clock frequency signal. This clock frequency signal can be directly used as a frequency synchronization signal, and its frequency is typically a division of the line rate. For example, for some high-speed Ethernet interfaces, 25MHz or 156.25MHz can be recovered (e.g.,...). Figure 3 The clock frequency signal (as shown).

[0043] Since the frequency synchronization signal is extracted directly from the physical bitstream carrying the data, it is not affected by upper-layer factors such as network congestion and protocol stack processing delay, and therefore has high stability and accuracy.

[0044] This application employs synchronous Ethernet technology, utilizing the Ethernet physical layer chip to achieve fast and robust frequency synchronization. Since this process is completed entirely in a closed loop within the physical layer hardware, it is unaffected by factors such as upper-layer network traffic congestion, CPU load fluctuations, or operating system scheduling delays. Therefore, frequency locking can be completed instantaneously. Compared to schemes that rely solely on upper-layer protocols for frequency estimation and compensation, the frequency synchronization signal obtained in this way has higher stability and lower jitter, providing an extremely solid foundation for subsequent high-precision phase synchronization and reference clock generation.

[0045] In one possible implementation, parsing the received network protocol messages to obtain the phase synchronization signal includes: The network protocol messages are parsed using a protocol stack based on a precise time protocol to generate a pulse signal per second, which serves as the phase synchronization signal.

[0046] Precision Time Protocol (PTP) is a network protocol specifically designed for achieving microsecond-level or even nanosecond-level time synchronization in distributed systems. The PTP stack running on a compute node can be a kernel module, a driver, or a user-space application, responsible for handling all PTP-related communication and computation.

[0047] Combination Figure 4 The PTP protocol stack participates in a master-slave synchronization process: receiving a series of PTP messages from the PTP master clock in the network, such as Sync Message, Follow_Up Message, Delay_Req Message, and Delay_Resp Message. These messages precisely record the exact hardware timestamps at the time of transmission by the master clock and reception by the slave clock (i.e., the local clock of the compute node).

[0048] The PTP protocol stack parses these messages and extracts multiple timestamp information. Then, the PTP protocol stack executes the PTP algorithm to calculate the bidirectional delay of the network path using these timestamps, and accurately calculates the phase deviation between the local clock and the master clock of the local computing node based on the bidirectional delay. After calculating the phase deviation, the PTP protocol stack adjusts the local clock hardware to compensate for this phase deviation, generating a 1-pulse-per-second (1pps) signal. The 1pps signal is a digital square wave signal, whose rising edge of each pulse is precisely calibrated to align with the second boundary of Coordinated Universal Time (UTC) represented by the master clock. In this embodiment, the 1pps signal serves as a high-precision, physically accessible phase synchronization signal that can be directly used by other hardware circuits.

[0049] In one possible implementation, the Synchronous Ethernet (SyncE) and PTP protocol stacks can share a clock source or use separate clock sources.

[0050] This application embodiment achieves high-precision absolute time alignment by utilizing the timestamp exchange and delay compensation mechanism of the Precision Time Protocol (PTP). In addition, the frequency synchronization accuracy achieved through Synchronous Ethernet (SyncE) is higher than that achieved through PTP, thus improving the time synchronization accuracy (reaching the nanosecond level). Therefore, with Synchronous Ethernet providing a stable frequency reference, the burden on the PTP protocol stack is greatly reduced, allowing it to focus more on phase calibration. This together contributes to achieving the final accuracy of the entire synchronization system at the nanosecond level, resulting in excellent time synchronization performance.

[0051] In one possible implementation, the network interface is a network interface that supports remote direct memory access.

[0052] Remote Direct Memory Access (RDMA) technology allows the memory of a compute node to be directly read and written by other compute nodes on the network without occupying the processor resources of the respective operating systems of the two nodes. This technology achieves extremely low latency and extremely high bandwidth by bypassing the network protocol stack of the operating system kernel.

[0053] In this embodiment of the application, the network interface of the computing node is a network card that supports RDMA functionality. For example... Figure 2 As shown, the compute nodes are directly connected to the Master Clock (MC) switch via the RDMA 400G northbound network port. Figure 3 As shown, the OAM module in the computing node is an accelerator module configured with RDMA functionality. It connects to the external network through an eight-channel small pluggable OSFP optical module interface to pursue better communication performance.

[0054] like Figure 4 As shown, the switch uses Ethernet to send data streams to the GPU compute node via its Ethernet physical layer chip (Eth PHY). The GPU compute node recovers a 25MHz synchronization clock (i.e., frequency synchronization signal) from the switch through the Eth PHY and then sends the 25MHz synchronization clock to the System PLL on the UBB (ubiquitous base station). Simultaneously, the GPU compute node obtains 1pps (i.e., phase synchronization signal) from the network protocol message through the integrated PTP stack and sends it to the System PLL. Subsequently, the System PLL generates a reference clock Ref_clk based on the 25MHz synchronization clock and the 1pps signal, and returns it to the GPU, ensuring strict synchronization between the GPU clock and the network switch clock.

[0055] Furthermore, RDMA operations themselves also require high precision in time synchronization. Therefore, in this embodiment, the network interface supporting RDMA communication is also used as the physical basis for time synchronization. That is, the hardware channel for low-latency data transmission and the hardware channel for nanosecond-level time synchronization are the same physical entity. This deep functional integration ensures that the timestamp used during RDMA operations is completely consistent with the global high-precision time base of the entire cluster. For example, when one GPU writes data to another GPU via RDMA, it can include a nanosecond-level precise timestamp, which the receiver can use for precise data sorting or version control.

[0056] This application embodiment adopts a network interface that supports RDMA, and utilizes the high-speed and low-latency characteristics of the RDMA network to provide a high-quality physical channel for the transmission of time synchronization protocol messages such as PTP, thereby further improving the overall performance and reliability of time synchronization.

[0057] In one possible implementation, the time synchronization method for computing nodes also includes: If the frequency synchronization signal is detected to be lost, the frequency synchronization signal is obtained based on the frequency information extracted from the network protocol message.

[0058] Since both Synchronous Ethernet (SyncE) and Precision Time Protocol (PTP) have frequency synchronization capabilities, the system prioritizes using SyncE for frequency synchronization. However, if a clock source or link failure occurs in SyncE, resulting in the loss of the frequency synchronization signal, the system will switch to PTP for frequency synchronization. This involves using the frequency information extracted from the network protocol messages via the PTP protocol stack to obtain the frequency synchronization signal.

[0059] Specifically, by analyzing the timestamp sequence of a series of PTP network protocol messages, the PTP protocol stack can not only calculate the phase deviation but also estimate the frequency drift of the local clock relative to the master clock. Once the loss of the physical layer's frequency synchronization signal based on synchronous Ethernet technology is detected, the system immediately instructs the PTP protocol stack to prioritize its frequency synchronization function. The PTP protocol stack takes over the frequency synchronization task, adjusting the frequency control input of the clock synthesis circuit through software or a dedicated hardware interface to obtain a frequency synchronization signal. Although its stability and response speed may not be as good as the physical layer signal, it can still maintain the clock frequency of the computing node at a level roughly synchronized with the network master clock.

[0060] This application embodiment prioritizes the use of a high-quality frequency synchronization signal from the physical layer. When this signal fails due to a fault, it can seamlessly degrade to the software frequency synchronization mode provided by the PTP protocol, thereby avoiding the rapid accumulation of time errors caused by frequency lockout. This redundant setting of dual frequency synchronization ensures that even if some hardware links fail, the entire computing node can still maintain high time synchronization availability, greatly enhancing the stability and reliability of the cluster in complex production environments.

[0061] In one possible implementation, the time synchronization method for computing nodes also includes: In the event of loss of the phase synchronization signal, the computing node maintains frequency synchronization with the network based on the frequency synchronization signal.

[0062] Under normal operating conditions, the 1pps signal generated by the PTP protocol stack is stable and appears periodically. However, in certain situations, this signal may be lost or become unreliable. For example, severe network congestion or packet loss may prevent the PTP protocol stack from receiving enough valid PTP packets to complete accurate phase calculations; the PTP master clock itself may malfunction; or the PTP protocol stack software within the compute node may malfunction. These situations can all cause the 1pps signal output to be interrupted or exhibit significant jumps. This loss can be detected by monitoring the pulse count of the 1pps signal or querying the status of the PTP protocol stack.

[0063] In the event of a loss of the phase synchronization signal, the system enters a holdover mode. In this mode, although the system loses the precise reference for calibrating the absolute time phase, it does not completely lose synchronization. Since the frequency synchronization signal from the physical layer is unaffected by upper-layer protocol issues, it remains stable and valid. Therefore, even if the phase synchronization signal is lost, the clock synthesis circuit continues to use the frequency synchronization signal as the frequency reference, ensuring that the frequency of the compute node's local clock oscillator remains precisely locked to the same frequency as the network master clock, and the system's time deviation remains within an acceptable range.

[0064] This application embodiment separates the fault domains of frequency synchronization and phase synchronization. In the event of a phase synchronization failure, the frequency of the local clock oscillator of the computing node is still precisely locked to the same frequency as the network master clock, which will not cause the entire synchronization system to collapse immediately. This degradation capability ensures that even if there are problems at the network protocol level, the system can still maintain the most basic and important frequency synchronization, which greatly improves the resilience of the entire cluster time synchronization system and its survivability in harsh network environments.

[0065] Based on the above, this application also provides a time synchronization device for computing nodes. The time synchronization device for computing nodes and the aforementioned time synchronization method for computing nodes can be referred to in correspondence.

[0066] In one possible implementation, the time synchronization device for the computing node is a processor, a dedicated control system, or a controller with integrated control logic deployed within the computing node, such as the central processing unit (CPU) on the computing node.

[0067] As an example, such as Figure 5 As shown, the time synchronization device for the computing node includes: The frequency synchronization module 510 is used to recover the frequency synchronization signal from the physical layer of the network interface on the computing node; wherein the computing node includes one or more of a graphics processor, tensor processor, neural network processor, deep learning processor, accelerator processor and general graphics processor.

[0068] The phase synchronization module 520 is used to parse received network protocol messages and obtain phase synchronization signals. The reference clock generation module 530 is used to simultaneously use the frequency synchronization signal and the phase synchronization signal as inputs to the clock synthesis circuit, so that the clock synthesis circuit locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference, so as to generate a reference clock for the computing node, and provides the reference clock as a unified time source to at least one component of the computing node.

[0069] This application combines the fast frequency locking capability of the physical layer with the precise phase alignment capability of the protocol layer. By working together, a reference clock is generated. Compared with time synchronization schemes based on network time protocols, this clock synthesis mechanism has higher time synchronization accuracy and smaller error. It can provide computing nodes with a stable and reliable nanosecond-level time reference, effectively solving the performance bottleneck and data consistency problems caused by time asynchrony in large-scale distributed computing.

[0070] In one possible implementation, the frequency synchronization module 510 is specifically used for: The Ethernet physical layer chip of the network interface recovers the clock frequency signal from the data stream of the physical link as the frequency synchronization signal.

[0071] This application employs synchronous Ethernet technology, utilizing the Ethernet physical layer chip to achieve fast and robust frequency synchronization. Since this process is completed entirely in a closed loop within the physical layer hardware, it is unaffected by factors such as upper-layer network traffic congestion, CPU load fluctuations, or operating system scheduling delays. Therefore, frequency locking can be completed instantaneously. Compared to schemes that rely solely on upper-layer protocols for frequency estimation and compensation, the frequency synchronization signal obtained in this way has higher stability and lower jitter, providing an extremely solid foundation for subsequent high-precision phase synchronization and reference clock generation.

[0072] In one possible implementation, the phase synchronization module 520 is specifically used for: The network protocol messages are parsed using a protocol stack based on a precise time protocol to generate a pulse signal per second, which serves as the phase synchronization signal.

[0073] This application embodiment achieves high-precision absolute time alignment by utilizing the timestamp exchange and delay compensation mechanism of the Precision Time Protocol (PTP). In addition, the frequency synchronization accuracy achieved through Synchronous Ethernet (SyncE) is higher than that achieved through PTP, thus improving the time synchronization accuracy (reaching the nanosecond level). Therefore, with Synchronous Ethernet providing a stable frequency reference, the burden on the PTP protocol stack is greatly reduced, allowing it to focus more on phase calibration. This together contributes to achieving the final accuracy of the entire synchronization system at the nanosecond level, resulting in excellent time synchronization performance.

[0074] In one possible implementation, the network interface described in this application embodiment is a network interface that supports remote direct memory access.

[0075] This application embodiment adopts a network interface that supports RDMA, and utilizes the high-speed and low-latency characteristics of the RDMA network to provide a high-quality physical channel for the transmission of time synchronization protocol messages such as PTP, thereby further improving the overall performance and reliability of time synchronization.

[0076] In one possible implementation, the frequency synchronization module 510 is also used for: If the frequency synchronization signal is detected to be lost, the frequency synchronization signal is obtained based on the frequency information extracted from the network protocol message.

[0077] This application embodiment prioritizes the use of a high-quality frequency synchronization signal from the physical layer. When this signal fails due to a fault, it can seamlessly degrade to the software frequency synchronization mode provided by the PTP protocol, thereby avoiding the rapid accumulation of time errors caused by frequency lockout. This redundant setting of dual frequency synchronization ensures that even if some hardware links fail, the entire computing node can still maintain high time synchronization availability, greatly enhancing the stability and reliability of the cluster in complex production environments.

[0078] In one possible implementation, the reference clock generation module 530 is also used for: In the event of loss of the phase synchronization signal, the computing node maintains frequency synchronization with the network based on the frequency synchronization signal.

[0079] This application embodiment separates the fault domains of frequency synchronization and phase synchronization. In the event of a phase synchronization failure, the frequency of the local clock oscillator of the computing node is still precisely locked to the same frequency as the network master clock, which will not cause the entire synchronization system to collapse immediately. This degradation capability ensures that even if there are problems at the network protocol level, the system can still maintain the most basic and important frequency synchronization, which greatly improves the resilience of the entire cluster time synchronization system and its survivability in harsh network environments.

[0080] Based on the above, this application also provides a computing node, including a network interface and a clock synthesizer.

[0081] The network interface is configured to recover the frequency synchronization signal from the physical layer of the network interface and use the protocol stack of the Precision Time Protocol to parse the received network protocol messages to obtain the phase synchronization signal. The clock synthesizer is configured to simultaneously take the frequency synchronization signal and the phase synchronization signal as inputs, lock the frequency of the output clock with the frequency synchronization signal, and calibrate the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node, and provide the reference clock as a unified time source to at least one component of the computing node.

[0082] The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0083] This application combines the fast frequency locking capability of the physical layer with the precise phase alignment capability of the protocol layer. By working together, a reference clock is generated. Compared with time synchronization schemes based on network time protocols, this clock synthesis mechanism has higher time synchronization accuracy and smaller error. It can provide computing nodes with a stable and reliable nanosecond-level time reference, effectively solving the performance bottleneck and data consistency problems caused by time asynchrony in large-scale distributed computing.

[0084] Based on the above, this application also provides a cluster system including multiple computing nodes, each of which is used for the above-described time synchronization method for computing nodes; or, The computing node is the computing node described above.

[0085] In one possible implementation, the cluster system could be a GPU cluster, a Neural Processing Unit (NPU) cluster, or the like.

[0086] This application combines the fast frequency locking capability of the physical layer with the precise phase alignment capability of the protocol layer. By working together, a reference clock is generated. Compared with time synchronization schemes based on network time protocols, this clock synthesis mechanism has higher time synchronization accuracy and smaller error. It can provide computing nodes with a stable and reliable nanosecond-level time reference, effectively solving the performance bottleneck and data consistency problems caused by time asynchrony in large-scale distributed computing.

[0087] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this application, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a time synchronization method for the computing node, the method including: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node; Parse the received network protocol messages to obtain the phase synchronization signal; The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, which locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node. The reference clock is then provided as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0088] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] On the other hand, this application also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the time synchronization method for computing nodes provided in the above embodiments, the method including: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node; Parse the received network protocol messages to obtain the phase synchronization signal; The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, which locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node. The reference clock is then provided as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0090] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the time synchronization method for computing nodes provided in the above embodiments, the method comprising: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node; Parse the received network protocol messages to obtain the phase synchronization signal; The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, which locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node. The reference clock is then provided as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A time synchronization method for computing nodes, characterized in that, include: Recover the frequency synchronization signal from the physical layer of the network interface on the computing node; Parse the received network protocol messages to obtain the phase synchronization signal; The frequency synchronization signal and the phase synchronization signal are simultaneously used as inputs to the clock synthesis circuit, which locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node. The reference clock is then provided as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

2. The time synchronization method for computing nodes according to claim 1, characterized in that, The recovery of the frequency synchronization signal from the physical layer of the network interface on the computing node includes: The Ethernet physical layer chip of the network interface recovers the clock frequency signal from the data stream of the physical link as the frequency synchronization signal.

3. The time synchronization method for computing nodes according to claim 1, characterized in that, The process of parsing the received network protocol messages to obtain the phase synchronization signal includes: The network protocol messages are parsed using a protocol stack based on a precise time protocol to generate a pulse signal per second, which serves as the phase synchronization signal.

4. The time synchronization method for computing nodes according to claim 1, characterized in that, The network interface is one that supports remote direct memory access.

5. The time synchronization method for computing nodes according to claim 1, characterized in that, Also includes: If the frequency synchronization signal is detected to be lost, the frequency synchronization signal is obtained based on the frequency information extracted from the network protocol message.

6. The time synchronization method for computing nodes according to claim 1, characterized in that, Also includes: In the event of loss of the phase synchronization signal, the computing node maintains frequency synchronization with the network based on the frequency synchronization signal.

7. A time synchronization device for computing nodes, characterized in that, include: A frequency synchronization module is used to recover a frequency synchronization signal from the physical layer of the network interface on a computing node; wherein the computing node includes one or more of a graphics processor, tensor processor, neural network processor, deep learning processor, accelerator processor, and general-purpose graphics processor. The phase synchronization module is used to parse received network protocol messages and obtain phase synchronization signals. A reference clock generation module is used to simultaneously use the frequency synchronization signal and the phase synchronization signal as inputs to a clock synthesis circuit, so that the clock synthesis circuit locks the frequency of the output clock with the frequency synchronization signal and calibrates the phase of the output clock with the phase synchronization signal as a reference, thereby generating a reference clock for the computing node, and providing the reference clock as a unified time source to at least one component of the computing node.

8. A computing node, characterized in that, Includes network interface and clock synthesizer; The network interface is configured to recover the frequency synchronization signal from the physical layer of the network interface and use the protocol stack of the Precision Time Protocol to parse the received network protocol messages to obtain the phase synchronization signal. The clock synthesizer is configured to simultaneously take the frequency synchronization signal and the phase synchronization signal as inputs, lock the frequency of the output clock with the frequency synchronization signal, and calibrate the phase of the output clock with the phase synchronization signal as a reference to generate a reference clock for the computing node, and provide the reference clock as a unified time source to at least one component of the computing node. The computing nodes include one or more of the following: graphics processors, tensor processors, neural network processors, deep learning processors, accelerator processors, and general-purpose graphics processors.

9. A cluster system, characterized in that, The cluster system includes multiple computing nodes, each of which is used to execute the time synchronization method for computing nodes as described in any one of claims 1-6; or, The computing node is the computing node described in claim 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the time synchronization method for computing nodes as described in any one of claims 1 to 6.

Images