Method and apparatus for data plane control of network time synchronization protocols in multi-host systems

Abstract

To provide accurate methods and apparatus for a network time sync protocol in multi-host systems.SOLUTION: A network interface controller (NIC) 206 includes a primary timer and secondary timers at distributed endpoints, receives packets having network timestamps, and uses a secondary timer to associate a local timestamp with the packets. The network and local timestamps are compared by a network intellectual property block (network IP) by adjusting the primary and secondary timers to match the network time. The network IP increments / decrements the primary and secondary timers to adjust the timers in nanoseconds.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Time distribution to various accelerators in a system-on-chip (SoC) is important to meet various real-time requirements such as shaping, timestamp recording, and latency measurement. A common main timer in a smart NIC (Network Interface Controller) can be distributed to various accelerator blocks in the SoC by different technologies (e.g., bus wiring, serial wiring, etc.). A common main timer or primary timer is useful for maintaining SoC-based time. It can be synchronized with various network sources. Examples include Institute of Electrical and Electronics Engineers (IEEE) 1588-based time references via Ethernet ports, GPS, synchronized Ethernet from ports, and other schemes.

[0002] In existing implementations, the primary timer is controlled by a host (or multiple hosts) external to the SoC or by a CPU local to the SoC. This is alternatively referred to as a control plane time synchronization protocol. The frequency and phase of the primary timer and the various distribution points must remain aligned. Frequency and phase alignment allows the distribution points to be accurate to the centralized primary timer. This enables, for example, the precise timestamps required for SoCs in 5G wireless radio access network (RAN) applications. Distribution of the primary timer to the PHY (physical layer) for local timestamps allows nanosecond (ns) accuracy. [Brief explanation of the drawings]

[0003] The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numerals refer to like parts throughout the various views unless otherwise indicated.

[0004] [Figure 1] FIG. 1 is a diagram of an exemplary primary timer, according to one embodiment.

[0005] [Figure 1a] FIG. 2 is a diagram of a modified version of the primary timer of FIG. 1, including a time step size register.

[0006] [Figure 2] FIG. 1 is a schematic diagram illustrating selected components of a platform including a host module with a software interface operating in a software-based control plane that communicates with a SmartNIC in the data plane.

[0007] [Figure 3] 3 is a schematic diagram illustrating further details of the SmartNIC of FIG. 2 implemented in a multi-host platform, according to one embodiment.

[0008] [Figure 4] FIG. 1 is a schematic diagram combined with a flowchart illustrating operations performed during initialization of a SmartNIC and its primary timer, as well as operations used by a network IP on the SmartNIC during ongoing network operation to adjust the primary and secondary timers to synchronize with network time, according to one embodiment.

[0009] [Figure 5] FIG. 1 is a schematic diagram of a first multi-host computing system using a SmartNIC configured to synchronize its primary and secondary timers with network time.

[0010] [Figure 6]FIG. 1 is a schematic diagram of a second multi-host computing system including a blade server including multiple compute blades and a network adapter mounted in a slot or mating connector in a backplane, midplane, or baseplane, according to one embodiment.

[0011] [Figure 7] FIG. 1 is a diagram of another exemplary computing platform or system that may be implemented using aspects of the embodiments described and illustrated herein. DETAILED DESCRIPTION OF THE INVENTION

[0012] Embodiments of a method and apparatus for data plane control of a network time synchronization protocol in a multi-host system are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of embodiments of the present invention. However, those skilled in the relevant art will recognize that the present invention can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring aspects of the present invention.

[0013] Throughout this specification, a reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or particular characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0014] For clarity, individual components in the figures herein may also be referred to by their labels on the figures rather than by specific reference numbers. Furthermore, reference numbers referring to a particular type of component (as opposed to a particular component) may be indicated by the reference number followed by "(exemplary)," meaning "exemplary." It will be understood that configurations of these components are typical for similar components that may exist but are not shown in the figures for simplicity and clarity, or for otherwise similar components that are not labeled with a separate reference number. Conversely, "(exemplary)" should not be interpreted to mean that a component, element, etc. is typically employed for its disclosed function, implementation, purpose, etc.

[0015] In accordance with aspects of the embodiments disclosed herein, the Network Intellectual Property Block (Network IP) in the data plane provides synchronization (i_sync) signal control for the main timer. The (i_sync) signal control provides increment and decrement features that can be triggered many times per second. The range of adjustment is programmable, allowing for coarse or fine timing changes depending on network constraints. The decision to increment or decrement the timer is made depending on network requirements and can be adjusted continuously in real time. The hardware required to make this decision of increment versus decrement resides in the Network IP in the data plane.

[0016] FIG. 1 illustrates a 96-bit timer 100 used in the main timer block, according to one embodiment. In one embodiment, the clock input (time_clk) 102 to the 96-bit timer 100 is between 500 MHz and 1 GHz, although this range is merely exemplary and non-limiting. When (time_clk) 102 is an 800 MHz signal, setting bit 32 adjusts the time output by approximately 1 ns. In one embodiment, the increment value (INCVAL) is the lower 40 bits. INCVAL is used to both increment and decrement the output of the 96-bit timer 100 based on the 2-bit i_sync signal, described below. In general, in one embodiment, bits 32 through 39 are used to adjust the main timer.

[0017] The 2-bit i_sync signal is labeled i_sync[1:0] and is transmitted on the two-wire interface. The [1:0] bit values ​​are used to achieve the following adjustments to the primary and secondary timers, according to one embodiment: 0:0 → No change 0:1 → increment 1:0 → Decrement 1:1 → No change Generally, the amount of increment or decrement can be predefined (e.g., 1 ns) or can be adjusted using a register associated with the timer or a shared register on the SmartNIC chip.

[0018] FIG. 2 illustrates a host operating in a software-based control plane communicating with a SmartNIC 206 in the data plane. withSelected components of platform 200 are shown, including a host module 202 having a software (SW) interface 204. SmartNIC 206 includes a local microcontroller 208, a primary timer and synchronization (PTS) block 210, and multiple Ethernet PHY ports 212. PTS block 210 includes an i_sync[1:0] register 214, a multiplexer (mux) 216, a primary timer 218, and a delay 220.

[0019] The i_sync[1:0] register 214 is programmed by the local microcontroller 208 based on input received from the SW interface 204, including the host. The output of the i_sync[1:0] register 214 is a two-wire signal 222 containing the two-bit i_sync[1:0] value, which comprises the first input to the mux 216. The second input to the mux 216 is the i_sync[1:0] signal 224 from the data plane output from the SmartNIC networking agent 226, which is part of the network data path.

[0020] A mux control signal 228 provided by firmware (FW) on the SmartNIC 206 is used to select whether the i_sync[1:0] signal 230 output by the mux 216 is the i_sync[1:0] signal 222 controlled by software in the control plane or the i_sync[1:0] signal 224 provided by a SmartNIC networking agent 226 implemented in hardware on the SmartNIC 206. The i_sync[1:0] signal 230 is provided to both the primary timer 218 and each of the secondary 1588 timers in the Ethernet PHY port 212, as shown in this example by secondary 1588 timers 232 and 234. As further shown, the primary timer 218 and each of the secondary 1588 timers 232 and 234 receive a time_clk input 236 from a SmartNIC clock 238. The Ethernet PHY port 212 also receives an Ethernet reference clock (ETH_REFCLK) input 240 .

[0021] The primary timer 218 is used to provide a time signal 242 to a network subsystem (NSS) agent on the SmartNIC 206, as further detailed in Figure 3. The primary timer 218 also outputs a ONE_PPS_OUT signal 244, which is a signal containing one pulse per second.

[0022] 3 illustrates a multi-host platform 300 including four host modules 202-0, 202-1, 202-2, and 202-3, each having a respective SW interface with host modules 204-0, 204-1, 204-2, and 204-3. FIG. 3 illustrates an exemplary set of networking agents along a network data path, including networking agents 302, 304, and 306 (also labeled as networking agents 1, 2, and 3). Networking agents 302, 304, and 306 represent a set of networking agents that may be implemented in a pipeline; in this example, these networking agents are implemented for outbound packets. A similar set of networking agents (not shown) is implemented in the pipeline used to process inbound packets.

[0023] Networking agent 302 is the first networking agent in the pipeline and is configured to receive input from each of host modules 202-0, 202-1, 202-2, and 202-3. Each host module includes software executing on a respective host central processing unit (CPU). In one embodiment, data is transferred between the host central processing unit (CPU) and SmartNIC 206A over a PCIe (Peripheral Component Interconnect Express) link (not shown) coupled between the host CPU and each PCIe interface on the SmartNIC. A mux 308 is shown within networking agent 302 to illustrate the ability to receive data, e.g., PCIe messages, from multiple hosts and also forward data destined for local microcontroller 208. In one embodiment, host module 202-0 executes on a host CPU used to boot a multi-host platform.

[0024] 2 and outputs i_sync[1:0] signal 224 on the two-wire interface between networking agent 304 and mux 216. Networking agent 304 also corresponds to SmartNIC networking agent 226 of FIG. See below in the early boot stage The rest is in the meantime , which corresponds to the network IP used to coordinate the primary and secondary timers.

[0025] Primary timer 218 outputs a 64-bit primary timer signal 242 that is distributed to the NSS agents on SmartNIC 206, as shown by the signal distribution to networking agents 302, 304, and 306. In the illustrated embodiment, 64-bit primary timer signal 242 has a frequency that is the time_clk signal 236 divided by 8. In general, the primary timer signal distributed to the NSS agents may be time_clk / n, where n is an integer greater than or equal to 4 (as determined by the physical design).

[0026] Figure 4 shows a host computer coupled to the SmartNIC206 through an interconnect such as a PCIe link. with Selected components of the platform 400 are shown, including a local host 402 that includes a SW interface 404. Figure 4 also shows an embedded logic block 406 that is implemented in the networking agent 304.

[0027] The local host 402 is used to initialize the platform 400 as well as to start and initialize the SmartNIC 206. This includes booting the SmartNIC 206 in block 408, obtaining the time of day in block 410, initializing the primary timer 218 in block 412, and switching control to firmware on the SmartNIC 206 in block 414. During the SmartNIC boot process, the SmartNIC 206 is initialized, which includes loading the firmware on the SmartNIC into local memory, as illustrated in Figure 5. The local host 402 can obtain the time of day through various means, such as using a clock on the host CPU used to run the local host software or through standard networking mechanisms.

[0028] Timer Control Mechanism

[0029] From a flow perspective, the PTS is brought out of reset by a secure entity, such as the local host, residing in the control plane. During this initial phase, software running within the control plane is used to set an initial time base set for the network. Subsequently, once the network data plane is brought out of reset, one of the network IPs takes control of the primary timer and can dynamically adjust the system time to account for real-time requirements to meet the network's timestamp requirements.

[0030] In one aspect, a configuration register is used to select the time step size for incrementing and decrementing the time, and this configuration register is implemented in the primary timer as well as other distribution points.

[0031] Based on a 100MHz clock frequency Kuto ,This scheme is, 2 Through wire interface hand, 100 million per second times Change Change Used for It can handle deviations of up to + / - 1 second. Combined with a 1ns time step size, we get up to 100 million 1ns times or 100 million of 1ns times (or a combination of these two events). This gives a maximum adjustment range of +0.1 seconds / second or -0.1 seconds / second. For example, a 1 second deviation would take approximately 10 seconds to correct.

[0032] An example of a 96-bit timer 100a including a time step register 104 is shown in Figure 1a. As shown, the inputs to the 96-bit timer 100a are time_clk 102 and a 2-bit i_sync[1:0] signal 106 transmitted over a 2-wire interface 108.

[0033] Returning to FIG. 4 , the primary timer 218 is initialized in the following manner, according to one embodiment. The local host 402 sends data to the microcontroller 208 to set the initial value of the primary timer 218 via a microcontroller-to-primary timer data path (not shown). Optionally, the primary timer 208 may be started using a cold boot or have a predetermined initial value used when exiting a sleep state. The local host 402 may also send data to the microcontroller 208 to program the time step value in the time step size register 104. Subsequently, the local host 402 sends data to the microcontroller 208 to update the i_sync[1:0] register 214 to increase or decrease the primary timer value. During this initialization phase, a mux control from the FW 228 sets the output from the mux 216 to the output from the i_sync[1:0] register 214.

[0034] In one embodiment, the time step size register 104 uses an m-bit value, such as a 2-bit or 3-bit value. For example, for a 2-bit time step size register, the following example mapping may be used: Change in ns per appearance from 00 to 01 / 10 01 → 01 / 10 Change of 8ns per appearance 10 → 01 / 10 Change of 128ns per appearance 11 → 01 / 10 Change of 1024ns per appearance

[0035] At some point following this initial phase, control of the timer is switched to the SmartNIC, as indicated by block 414. During this second ongoing runtime phase, the embedded logic 406 is used by the networking agent 304 to dynamically adjust the primary and secondary clocks to take into account current network requirements. The process proceeds as follows:

[0036] Periodically, network timestamp packets 416 are received at one or more of the PHY ports 212. For example, depending on the network configuration, every ith packet is a network timestamp packet. An exemplary value for i is 100. As shown by signal path 418, two pieces of information are provided from the PHY port 212 that receives the network timestamp packet: the network timestamp packet and a timestamp generated by the PHY port that corresponds to the local time the network timestamp packet was received. B For example, under platform 400, the local timestamp is generated using a secondary 1588 timer. Optionally, a signal from primary timer 218 can be used.

[0037] In one embodiment, a first header field in the network timestamp packet 416 contains a network timestamp value, and a local timestamp value is written into a second header field to create a network timestamp packet 416a, which is then forwarded to the network agent 304.

[0038] Within embedded logic 406, either network timestamp packet 416a is provided to each of blocks 420 and 422, or another logic block (not shown) is used to extract a local timestamp value from network timestamp packet 416a and provide it to block 420. Block 420 obtains or extracts the local timestamp value and provides it as an input to block 424. In parallel, block 422 extracts the network timestamp value from network timestamp packet 416a and provides it as a second input to block 424. Block 424 then compares the local timestamp value with the network timestamp value. Based on the result of this comparison, the primary timer is adjusted in block 426. Under embodiments that support fine and coarse adjustment, a fine or coarse adjustment input is provided to block 426. Block 426 then: corresponds to the i_sync[1:0] signal 224 The 2-bit i_sync[1:0] value m ux 216. Optionally, coarse or fine adjustments can be used by block 426 to program the time step register 104 on the primary timer 218 and any timer distribution points.

[0039] Similar processing of network timestamp packets 416 is performed continually, allowing PTS 210 to adjust primary timer 218 and appropriate secondary timers (e.g., secondary 1588 timers 232 and 234) on SmartNIC 206.

[0040] Exemplary Platform / Computing System

[0041] 5 illustrates a first example of a multi-host computing system 500 using a SmartNIC 506 configured to synchronize its primary and secondary timers with network time, according to the aforementioned embodiments. In this example, computing system 500 includes four hosts, each including a respective host CPU 500-x, where x is the number of CPUs (also designated host CPUs 0, 1, 2, and 3) coupled to host memory 504-x. Each host includes a respective host module 202-x having a SW interface with host 204-x.

[0042] Each of the host CPUs is communicatively coupled to a host I / O interface 508-x on the SmartNIC 506. In one non-limiting example, the host I / O interfaces include PCIe interfaces. In the case of PCIe interfaces, these interfaces would be connected to a PCIe bus (a.k.a., PCIe link) coupled to a PCIe root port in the host CPU. Alternatively, with respect to FIG. 6, below In the blade server embodiment described herein, the PCIe bus is shared by multiple server blades, so SmartNIC 506 only requires a single host I / O interface.

[0043] SmartNIC 506 includes the circuits shown for SmartNIC 206 in the figures above, some of which are shown in Figure 5. In addition, SmartNIC 506 includes an embedded NIC processor 510 coupled to memory 512, an inbound packet processing pipeline 514 including n network receive (RX) IP blocks 514, an outbound packet processing pipeline 516 including m network transmit (TX) IP blocks 518, and one or more network ports 520. SmartNIC 506 also includes direct memory access (DMA) logic 522, which is representative of the circuitry and logic onboard a NIC for facilitating DMA transfers between host memory 504-x and SmartNIC 506.

[0044] The NIC 506 may also include built-in "accelerator" hardware, etc., used to perform packet processing operations, such as flow control, encryption, decryption, etc. For example, the NIC 506 may include one or more cryptographic blocks, as indicated by crypto unit 524, that are coupled to network traffic and configured to perform encryption and decryption. The NIC 506 may also include a hashing unit, as indicated by hashing unit 526, for accelerating hash key matching associated with packet flow lookups.

[0045] SmartNIC 506 may further include firmware storage 528 in which firmware 530 is stored. Firmware 530 includes instructions executed by NIC processor 510 to implement various functions supported by SmartNIC 506. Optionally, firmware 530 may be stored in and accessed from a storage device (not shown) external to NIC 506. As another option, portions of firmware 530 may be loaded from a remote storage device over a network using either an in-band or out-of-band connection.

[0046] In the illustrated example, network port 520 is an Ethernet port that includes an RX port 532 and a TX port 534. RX port 532 includes one or more RX buffers in which received packets (e.g., packets A, B, C, D) are buffered before being processed by inbound packet processing pipeline 512. TX port 534 includes one or more TX buffers in which packets to be transmitted (e.g., packets Q, R, S, T) are buffered after processing performed by outbound packet processing pipeline 516. Network port 520 also includes circuitry and logic for implementing a physical layer (PHY Layer 1) and a media access channel (MAC) (Layer 2) in accordance with one or more applicable Ethernet protocols, including more recent Ethernet protocols supporting bandwidths of 25, 50, 100, 200, and / or 400 Gigabits per second (Gb / s). Under various embodiments, SmartNIC 506 may also implement Network Layer 3 and Transport Layer 4 in inbound and outbound packet processing pipelines 512 and 516. Alternatively, Network Layer 3 and Transport Layer 4 may be implemented in a network stack running on the host.

[0047] Following inbound packet processing, RX packets are buffered or queued in one or more upstream packet queues 536 in memory 512 for subsequent forwarding to the applicable destination host. Packets to be transmitted are buffered in one or more downstream packet queues 538 in memory 512 before being processed by outbound packet processing pipeline 516.

[0048] For simplicity, memory 512 is shown as a single block. In implementations, SmartNIC 506 may use more than one memory device. Memory 512 also represents different types of memory, such as SRAM and DRAM, and is part of a memory subsystem for SmartNIC 506. Aspects of the memory subsystem, such as a memory controller on NIC processor 510, are not shown to avoid complexity, but one skilled in the art would understand that SmartNIC 506 includes these features. Additionally, the memory space of memory 512 may be partitioned to support different access mechanisms, such as for security purposes. For example, a portion of memory 512 may be implemented to support memory-mapped input / output (MMIO), allowing data to be written to and read from the MMIO portion by software running on the host.

[0049] In general, the functionality of the blocks illustrated for SmartNIC 506 may be implemented using some form of embedded logic. Embedded logic generally includes logic implemented in circuitry, for example, using an FPGA (field programmable gate array) or using pre-programmed or fixed hardware logic (or a combination of pre-programmed / hard-coded and programmable logic), as well as firmware running on one or more embedded processors, processing elements, engines, or microcontrollers, etc. For illustrative purposes, an example of firmware running on NIC processor 510 is shown in FIG. 5, but this is not meant to be limiting. NIC processor 510 is a form of embedded processor that may include multiple processing elements, such as cores or microengines.

[0050] Generally, a multi-host system may use two or more hosts. In some implementations, the multi-host system is a multi-socket server having two or more sockets, each associated with a respective CPU / SoC coupled to local memory. Other configurations, such as blade servers and servers using server modules, may also be used.

[0051] 6 illustrates a second exemplary computing system including a blade server 600 in which aspects of the previously described embodiments may be implemented. The blade server includes four compute blades 601-0, 601-1, 601-2, and 601-3, each having a CPU 602 coupled to host memory 604 and coupled to a PCIe interface 608. During operation of the blade server 600, control plane software 610 is loaded into the host memory 604 and executed by the CPU 602. The blade server 600 includes one or more network adapters 612 including a NIC chip 606 coupled to the PCIe interface 614 and coupled to one or more Ethernet ports, as indicated by ports 616 and 618, which are connected to a network 620. The blade server 600 includes a backplane, midplane, or baseplane 622 having multiple expansion slots or connectors, as indicated by slots / connectors 624 and 626. Each of the compute blades 601-0, 601-1, 601-2, and 601-3, as well as the network adapter 612, includes a connector that mounts within a respective expansion slot or couples to a mating connector on the backplane, midplane, or baseplane 622.

[0052] Generally, a blade server implementation may have two or more compute blades and one or more network adapters. Computing systems that use server modules instead of blades may be used in a similar manner. Generally, the NIC chip 606 includes circuitry and embedded logic similar to the SmartNIC 506 described above. In some embodiments, the network adapter 612 may include a firmware storage device 628 separate from the NIC chip 606, on which firmware 630 is stored. During operation, the firmware 630 is read from the firmware storage device 628 and loaded into memory on the NIC chip 606.

[0053] 7 illustrates a third example of a computing system 700 in which aspects of the embodiments disclosed herein may be implemented. The computing system 700 includes one or more processors 710 that provide processing, operational management, and instruction execution for the computing system 700. The processor 710 may include any type of microprocessor, central processing unit (CPU), graphics processing unit (GPU), processing core, multi-core processor, or other processing hardware for providing processing for the computing system 700, or a combination of processors. The processor 710 controls the overall operation of the computing system 700 and may be or include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of such devices.

[0054] In one example, computing system 700 includes an interface 712 coupled to processor 710, which may represent a high-speed or high-throughput interface for system components requiring a higher-bandwidth connection, such as memory subsystem 720, or any graphics interface component 740, or any accelerator 742. Interface 712 represents interface circuitry that may be a standalone component or integrated on the processor die. If present, graphics interface 740 interfaces to a graphics component for providing a visual display to a user of computing system 700. In one example, graphics interface 740 may drive a high-definition (HD) display that provides output to the user. High definition may refer to a display with a pixel density of approximately 100 PPI (pixels per inch) or greater and may include formats such as Full HD (e.g., 1080p), retina display, or 4K (ultra-high definition or UHD). In one example, the display may include a touchscreen display. In one example, graphics interface 740 generates a display based on data stored in memory 730, or based on operations performed by processor 710, or both. In one example, graphics interface 740 generates a display based on data stored in memory 730, or based on operations performed by processor 710, or both.

[0055] Memory subsystem 720 represents the main memory of computing system 700 and provides storage for data values ​​used to execute code or routines executed by processor 710. Memory subsystem 720 may include one or more memory devices 730, such as read-only memory (ROM), flash memory, one or more types of random access memory (RAM), such as DRAM, or other memory devices, or a combination of such devices. Memory 730 stores and hosts, among other things, an operating system (OS) 732, which provides a software platform for the execution of instructions in computing system 700. Additionally, applications 734 may execute on the software platform of OS 732 from memory 730. Applications 734 represent programs having their own operating logic for performing one or more functions. Processes 736 represent agents or routines that provide auxiliary functionality to OS 732, or one or more applications 734, or a combination thereof. OS 732, applications 734, and processes 736 provide the software logic that provides functionality for computing system 700. In one example, memory subsystem 720 includes memory controller 722, which is a memory controller for generating and issuing commands to memory 730. It will be understood that memory controller 722 may be a physical part of processor 710 or a physical part of interface 712. For example, memory controller 722 may be an integrated memory controller integrated into circuitry with processor 710.

[0056] Although not specifically illustrated, it will be understood that computing system 700 may include one or more buses or bus systems between devices, such as a memory bus, a graphics bus, an interface bus, or others. A bus or other signal lines may communicatively or electrically couple components to each other or communicatively and electrically couple components. A bus may include a physical communication line, a point-to-point connection, a bridge, an adapter, a controller, or other circuitry, or a combination thereof. A bus may include, for example, one or more of a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (Firewire).

[0057] In one example, computing system 700 includes an interface 714 that may be coupled to interface 712. In one example, interface 714 represents an interface circuit that may include standalone components and integrated circuits. In one example, multiple user interface and / or peripheral components are coupled to interface 714. Network interface 750 provides computing system 700 with the ability to communicate with remote devices (e.g., servers or other computing devices) over one or more networks. Network interface 750 may include an Ethernet adapter, a wireless interconnection component, a cellular network interconnection component, a Universal Serial Bus (USB), or other wired or wireless, standards-based, or proprietary interface. Network interface 750 may transmit data to a device in the same data center or rack or to a remote device, and may include transmitting data stored in memory. Network interface 750 may receive data from a remote device, and the remote device may include storing the received data in memory. Various embodiments may be used in connection with network interface 750, processor 710, and memory subsystem 720.

[0058] In one example, computing system 700 includes one or more IO interfaces 760. IO interface 760 may include one or more interface components (e.g., audio, alphanumeric, tactile / touch, or other interfaces) through which a user interacts with computing system 700. Peripheral interface 770 may include any hardware interface not specifically mentioned above. Peripherals generally refer to devices that connect dependently to computing system 700. A dependent connection is one in which computing system 700 provides a software or hardware platform, or both, on which operations are performed and with which a user interacts.

[0059] In one example, computing system 700 includes a storage subsystem 780 for storing data in a non-volatile manner. In one example, in some system implementations, at least certain components of storage 780 may overlap with components of memory subsystem 720. Storage subsystem 780 includes storage device 784, which may be or include any conventional medium for storing large amounts of data in a non-volatile manner, such as one or more magnetic, solid-state, or optical-based disks, or a combination thereof.

[0060] In one example, computing system 700 may be implemented using interconnected compute threads of processors, memory, storage, network interfaces, and other components. High-speed interconnects such as Ethernet (IEEE 802.3), Remote Direct Memory Access (RDMA), InfiniBand, Internet Wide Area RDMA Protocol (iWARP), Quick UDP Internet Connection (QUIC), RDMA over Converged Ethernet (RoCE), Peripheral Component Interconnect Express (PCIe), Intel® QuickPath Interconnect (QPI), Intel® UltraPath Interconnect (UPI), Intel® On-Chip System Fabric (IOSF), OmniPath, Compute Express Link (CXL), HyperTransport, High-Speed ​​Fabric, NV-Link, Advanced Microcontroller Bus Architecture (AMBA) Interconnect, OpenCAPI, Gen-Z, Cache Coherent Interconnect for Accelerators (CCIX), 3GPP Long Term Evolution (LTE) (4G), 3GPP 5G, and variations thereof may be used. The data may be copied or stored on a virtualized storage node using a protocol such as NVMe over Fabrics (NVMe-oF) or NVMe.

[0061] In addition to computing systems having a processor or CPU, the teachings and principles disclosed herein may be applied to computing systems and platforms. Other processing units (collectively referred to as XPUs) include one or more of graphics processor units (GPUs) or general-purpose GPUs (GP-GPUs), tensor processing units (TPUs), data processor units (DPUs), infrastructure processing units (IPUs), artificial intelligence (AI) processors or AI inference units and / or other accelerators, FPGAs and / or other programmable logic (used for computational purposes), etc. While some of the figures herein show the use of a processor, this is merely exemplary and non-limiting. In general, any type of XPU may be used in place of a CPU or processor in the illustrated embodiments. Furthermore, as used in the following claims, the term "processor" is used generically to encompass various forms of processors, including CPUs and different forms of XPUs.

[0062] Although some embodiments have been described with reference to particular implementations, other implementations are possible in accordance with some embodiments. Moreover, the arrangement and / or order of elements or other features illustrated in the drawings and / or described herein need not be arranged in the particular manner illustrated and described. Many other arrangements are possible in accordance with some embodiments.

[0063] In each system shown in the figures, elements in some cases may each have the same or different reference numbers to suggest that the represented elements may be different and / or similar. However, elements may be flexible enough to have different implementations and to work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which elements are referred to as first elements and which as second elements is arbitrary.

[0064] In the description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” may be used to indicate that two or more elements are in direct physical or electrical contact with each other. “Coupled” may mean that two or more elements are in direct physical or electrical contact. However, “coupled” may also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other. Furthermore, “communicatively coupled” means that two or more elements, which may or may not be in direct contact with each other, can communicate with each other. For example, if component A is connected to component B, which in turn is connected to component C, then component A may be communicatively coupled to component C using component B as an intermediary component.

[0065] An embodiment is an implementation or example of the present invention. References herein to "an embodiment," "one embodiment," "some embodiments," or "other embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments of the present invention, but not necessarily all embodiments. The various appearances of "an embodiment," "one embodiment," or "some embodiments" do not necessarily all refer to the same embodiments.

[0066] Not all components, features, structures, characteristics, etc. described and illustrated herein need be present in a particular embodiment or embodiments. For example, if the specification states that a component, feature, structure, or characteristic "may," "will," "could," or "can" be included, it does not require that the particular component, feature, structure, or characteristic be included. When the specification or claims refer to "an" or "an" element, this does not mean that there is only one of that element. When the specification or claims refer to "additional" elements, this does not exclude the presence of more than one of the additional element.

[0067] Italics, e.g., "i," "m," "n," etc., in the above detailed description are used to indicate integers, and the use of a particular letter is not limited to a particular embodiment. Furthermore, the same letter may be used in separate claims to represent different integers, or different letters may be used. In addition, the use of a particular letter in the detailed description may or may not match the letter used in a claim relating to the same subject matter in the detailed description.

[0068] As described above, various aspects of the embodiments herein may be facilitated by corresponding software and / or firmware components and applications, such as software and / or firmware executed by an embedded processor or the like. Accordingly, embodiments of the present invention may be used as or to support a software program, software module, firmware, and / or distributed software executed on some form of processor, processing core, or embedded logic, a virtual machine running on a processor or core, or otherwise implemented or realized on or in a non-transitory computer-readable or machine-readable storage medium. A non-transitory computer-readable or machine-readable storage medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a non-transitory computer-readable or machine-readable storage medium includes any mechanism that provides (e.g., stores and / or transmits) information in a form accessible by a computer or computing machine (e.g., computing device, electronic system, etc.), such as recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). The content may be directly executable (in "object" or "executable" form), source code, or difference code ("delta" or "patch" code. A non-transitory computer-readable or machine-readable storage medium may also include a storage or database from which content is downloaded. A non-transitory computer-readable or machine-readable storage medium may also include a device or product that stores content at the time of sale or delivery. Thus, delivering a device with stored content or providing downloaded content over a communications medium may be understood to provide a product that includes a non-transitory computer-readable or machine-readable storage medium with content as described herein.

[0069] The operations and functions performed by the various components described herein may be implemented via embedded hardware, etc., by software executing on a processing element, or by any combination of hardware and software. Such components may be implemented as software modules, hardware modules, dedicated hardware (e.g., application specific hardware, ASICs, DSPs, etc.), embedded controllers, hardwired circuitry, hardware logic, etc. Software content (e.g., data, instructions, configuration information, etc.) may be provided via an article of manufacture including a non-transitory computer-readable or machine-readable storage medium providing content representing executable instructions. The content may cause a computer to perform the various functions / operations described herein.

[0070] As used herein, a list of items joined by the term "at least one of" can mean any combination of the listed terms. For example, the phrase "at least one of A, B, or C" can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0071] The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise form disclosed. While specific embodiments of, and examples for, the invention have been described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

[0072] These modifications may be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and drawings. Rather, the scope of the invention is to be determined solely by the following claims, which are to be construed in accordance with established doctrines of interpretation. [Other possible items] [Item 1] 1. A network interface controller (NIC) configured to be implemented in a network data plane when installed in a computing system including one or more hosts having respective central processing units (CPUs) used to execute software implementing a network control plane associated with the network data plane, the NIC comprising: a primary timer; one or more network ports; one or more secondary timers associated with each network port; The NIC is receiving, at the network port, a network time packet having a network timestamp associated with a network time; associating a local timestamp generated by the secondary timer at the network port with the network time packet; comparing the local timestamp with the network timestamp; adjusting the primary timer based on the comparison between the local timestamp and the network timestamp. The NIC is configured as follows: [Item 2] The data plane further includes a network intellectual property block (network IP) including one of a plurality of networking blocks along a network data path in the data plane, the network intellectual property block comprising: receiving a plurality of network time packets associated with a network timestamp and a local timestamp; comparing the local timestamp with the network timestamp; adjusting the time of the primary timer based on the comparison of the local timestamp and the network timestamp to provide a signal to the primary timer to synchronize the primary timer with the network time; Item 1. The NIC of item 1, configured as follows: [Item 3] Item 3. The NIC of item 2, wherein the network IP is further configured to provide an adjustment signal to at least one secondary timer to adjust the time of the secondary timer based on the comparison of the local timestamp and the network timestamp. [Item 4] Item 4. The NIC of item 3, wherein the network IP is configured to provide a coordination signal in parallel to the primary timer and at least one of the secondary timers. [Item 5] Item 3. The NIC of item 2, wherein the adjustment signal includes an indicator for incrementing or decrementing the primary timer. [Item 6] a host interface for communicating with the one or more host processors, wherein when the NIC is installed in the computing system, initializing the primary timer in response to input received from a host processor via the host interface; transferring control of the primary timer to the network IP; Item 3. The NIC of item 2, further configured as follows: [Item 7] Item 3. The NIC of item 2, further comprising a plurality of networking blocks along a network data path in the data plane, wherein the network IP includes or is implemented in one of the plurality of networking blocks. [Item 8] The primary timer includes a register configured to store a value specifying an amount by which the primary timer is to be incremented or decremented, and the primary timer: receiving an adjustment signal to increment or decrement the timer value; incrementing or decrementing the timer value as a function of the value in the register; Item 1. The NIC of item 1, configured as follows: [Item 9] Item 1. The NIC of item 1, wherein the primary clock is capable of being adjusted to approximately 1 nanosecond or less. [Item 10] 1. A method implemented by a network interface controller (NIC) installed in a computing system having one or more host central processing units (CPUs) executing software to implement a control plane associated with a data plane implemented by the NIC, comprising: receiving, at a network port on the NIC, a network time packet having a network timestamp associated with a network time; upon receiving the network time packet, generating a local timestamp corresponding to a local time via a secondary timer at the network port; comparing the local timestamp with the network timestamp; adjusting a primary timer on the NIC based on the comparison of the local timestamp and the network timestamp; A method comprising: [Item 11] receiving, at a network intellectual property block (network IP) on the NIC, a plurality of network time packets associated with a network timestamp and a local timestamp; comparing the local timestamp with the network timestamp; providing an adjustment signal to the primary timer to adjust the time of the primary timer based on the comparison of the local timestamp and the network timestamp to synchronize the primary timer with the network time; Item 11. The method of item 10, further comprising: [Item 12] Item 12. The method of item 11, further comprising providing the adjustment signal to the primary timer and the secondary timer to adjust the times of the primary timer and the secondary timer in parallel. [Item 13] Item 12. The method of item 11, wherein the adjustment signal includes a two-bit value indicating whether to increment, decrement, or not adjust the primary timer. [Item 14] the NIC includes a host interface for communicating with the one or more host processors; initializing the primary timer in response to input received from a host processor via the host interface; transferring control of the primary timer to the network IP; Item 12. The method of item 11, further comprising: [Item 15] the primary timer includes a register configured to store a multi-bit value that specifies an amount by which the primary timer is incremented or decremented; programming the register with a value; receiving an adjustment signal to increment or decrement a timer value for the primary timer; incrementing or decrementing the timer value as a function of the value in the register; Item 11. The method of item 10, further comprising: [Item 16] one or more hosts, each host including a central processing unit (CPU) coupled to a memory; a network interface controller (NIC) operably coupled to each host, one or more network ports, Primary timer, and a NIC including one or more secondary timers associated with each network port, the NIC comprising: receiving, at the network port, a network time packet having a network timestamp associated with a network time; receiving the network time packet while generating a local timestamp using a secondary timer associated with the network port; comparing the local timestamp with the network timestamp; adjusting the primary timer based on the comparison between the local timestamp and the network timestamp to synchronize the primary timer with the network time; 1. A computing system configured to: [Item 17] The NIC further includes a network intellectual property block (network IP) that includes one of a plurality of networking blocks along a network data path in the data plane, the network intellectual property block comprising: receiving a plurality of network time packets associated with a network timestamp and a local timestamp; comparing the local timestamp with the network timestamp; providing a signal to the primary timer to adjust the time of the primary timer based on the comparison of the local timestamp and the network timestamp; Item 17. The computing system of item 16, configured to: [Item 18] 20. The computing system of claim 17, wherein the network IP is further configured to provide an adjustment signal to at least one secondary timer to adjust the time of the secondary timer based on the comparison of the local timestamp and the network timestamp. [Item 19] Item 18. The computing system of item 17, using a two-bit wire protocol, in which the adjustment signal is used to increment and decrement the primary timer. [Item 20] the NIC further comprises a host input / output (I / O) interface coupled to a first host, the system comprising: initializing the primary timer under control of the first host; transferring control of the primary timer to the NIC; Item 17. The computing system of item 16, configured to:

Claims

1. 1. A network interface controller (NIC) configured to be implemented in a network data plane when provided in a computing system including one or more hosts having respective central processing units (CPUs) used to execute software implementing a network control plane associated with the network data plane, the NIC comprising: a primary timer; one or more network ports; one or more secondary timers associated with each network port; the primary timer and the one or more secondary timers are controlled to be synchronized; The NIC is receiving, at the network port, a network time packet having a network timestamp associated with a network time; associating a local timestamp generated by the one or more secondary timers at the network port with the network time packet; comparing the local timestamp with the network timestamp; adjusting the primary timer based on the comparison between the local timestamp and the network timestamp. The NIC is configured as follows:

2. The primary timer includes a register configured to store a value specifying an amount by which the primary timer is to be incremented or decremented, and the primary timer: receiving an adjustment signal to increment or decrement the timer value; incrementing or decrementing the timer value depending on the value in the register; 10. The NIC of claim 1, configured to:

3. 3. The NIC of claim 1, wherein the primary clock is capable of being adjusted by approximately 1 nanosecond or less.

4. The network data plane further includes a network intellectual property block (Network IP) including one networking block of a plurality of networking blocks along a network data path in the network data plane, the network intellectual property block comprising: receiving a plurality of the network time packets associated with the network timestamp and the local timestamp; comparing the local timestamp with the network timestamp; adjusting the time of the primary timer based on the comparison of the local timestamp and the network timestamp to provide an adjustment signal to the primary timer to synchronize the primary timer with the network time.

3. The NIC of claim 2, configured to:

5. a host interface for communicating with a CPU of the one or more hosts, wherein when the NIC is provided in the computing system, the NIC initializing the primary timer in response to input received from a CPU of the host via the host interface; transferring control of the primary timer to the network IP; The NIC of claim 4 further configured to:

6. A NIC as described in claim 4 or 5, wherein the network IP is implemented in one of the plurality of networking blocks.

7. The NIC of claim 4 , wherein the adjustment signal includes an indicator for incrementing or decrementing the primary timer.

8. 8. The NIC of claim 4, wherein the network IP is further configured to provide the adjustment signal to at least one secondary timer to adjust the time of the one or more secondary timers based on the comparison of the local timestamp and the network timestamp.

9. 9. The NIC of claim 8, wherein the network IP is configured to provide the coordination signal in parallel to the primary timer and at least one secondary timer of the one or more secondary timers.

10. 1. A method implemented by a network interface controller (NIC) in a computing system having one or more host central processing units (CPUs) executing software to implement a control plane associated with a data plane implemented by the NIC, the method comprising: receiving, at a network port on the NIC, a network time packet having a network timestamp associated with a network time; generating a local timestamp corresponding to the local time at which the network time packet is received by a secondary timer on the network port, the primary timer on the NIC and the secondary timer being controlled to be synchronized; comparing the local timestamp with the network timestamp; adjusting a primary timer on the NIC based on the comparison of the local timestamp and the network timestamp; A method comprising:

11. the primary timer includes a register configured to store a multi-bit value that specifies an amount by which the primary timer is to be incremented or decremented, and the method further comprises: programming the register with a value; receiving an adjustment signal to increment or decrement a timer value of the primary timer; incrementing or decrementing the timer value of the primary timer according to the value in the register; The method of claim 10 further comprising:

12. receiving, at a network intellectual property block (network IP) on the NIC, a plurality of the network time packets associated with the network timestamp and the local timestamp; comparing the local timestamp with the network timestamp; adjusting the time of the primary timer based on the comparison of the local timestamp and the network timestamp to provide the adjustment signal to the primary timer to synchronize the primary timer with the network time; The method of claim 11 further comprising:

13. 13. The method of claim 12, further comprising providing the adjustment signal to the primary timer and the secondary timer to adjust the times of the primary timer and the secondary timer in parallel.

14. 14. The method of claim 12 or 13, wherein the adjustment signal comprises a two-bit value indicating whether to increment, decrement, or not adjust the primary timer.

15. the NIC includes a host interface for communicating with the one or more host CPUs, and the method initializing the primary timer in response to input received from a host CPU via the host interface; transferring control of the primary timer to the network IP; 15. The method of any one of claims 12 to 14, further comprising:

16. One or more hosts, each including a central processing unit (CPU) coupled to a memory; a network interface controller (NIC) operably coupled to each host, said NIC comprising: one or more network ports; Primary timer, and one or more secondary timers associated with each network port, the primary timer and the one or more secondary timers being controlled to be synchronized; The NIC is receiving, at the network port, a network time packet having a network timestamp associated with a network time; generating a local timestamp associated with receipt of the network time packet using a secondary timer associated with the network port; comparing the local timestamp with the network timestamp; adjusting the primary timer based on the comparison between the local timestamp and the network timestamp to synchronize the primary timer with the network time; 1. A computing system configured to:

17. the NIC further comprising a host input / output (I / O) interface coupled to a first host, the computing system comprising: initializing the primary timer under control of the first host; transferring control of the primary timer to the NIC; 17. The computing system of claim 16, configured to:

18. The NIC further includes a network intellectual property block (Network IP) that includes one networking block of a plurality of networking blocks along a network data path in a data plane, the network intellectual property block comprising: receiving a plurality of the network time packets associated with the network timestamp and the local timestamp; comparing the local timestamp with the network timestamp; providing an adjustment signal to the primary timer to adjust the time of the primary timer based on the comparison of the local timestamp and the network timestamp; 18. A computing system according to claim 16 or 17, configured to:

19. 20. The computing system of claim 18, wherein the network IP is further configured to provide the adjustment signal to at least one of the secondary timers to adjust the time of the one or more secondary timers based on the comparison of the local timestamp and the network timestamp.

20. 20. The computing system of claim 19, wherein the adjustment signal uses a two-bit wire protocol used to increment and decrement the primary timer.

Images