Predefined fault-tolerant network effect clock synchronization

Abstract

A system and method for a predefined network topology and a predefined and linearly independent time synchronization ring. The predefined network topology includes a predefined symmetric network topology, such as a ring network topology. In the predefined symmetric network topology, a ring is constructed by computing nodes selected based on rules, where the ring construction takes a simplified line-up form to provide a linearly independent ring. These schemes create a time synchronization architecture that can tolerate one or more computing node failures without the need to re-compute the network topology or ring, and are able to add one or more irregular computing nodes and generate additional linearly independent rings. These schemes are also able to accommodate various multi-level physical network configurations and provide a physical network-aware multi-level network effect architecture for clock synchronization.

Description

TECHNICAL FIELD

[0001] The present application relates to a system and method for providing clock synchronization for network computing devices, and more particularly, to a predefined and highly fault-tolerant network effect scheme for clock synchronization. BACKGROUND

[0002] Time synchronization of network computing devices has been used to synchronize clocks between network computing devices. Clock synchronization is a fundamental technique for many distributed computing systems to function. Clock synchronization can be used in a distributed database environment to improve performance and consistency, in a financial system environment to improve transaction execution order, or in a software defined network to improve sequencing of forwarding rule updates.

[0003] Figure 1 Conventional clock synchronization in an example network computing environment 100 is shown. The environment 100 includes a first computer 110 and a second computer 120. To synchronize time, the first computer 110 can send a timing probe 115 to the second computer 120, which can respond with a timing acknowledgment 125. The first computer 110 and the second computer 120 use different clocks, each of which changes over time due to temperature, noise, and other influencing factors, so frequent clock synchronization can be needed. Each clock can have a different resonant frequency, and can behave differently to the same resonant frequency or offset control signal. Clock synchronization can be affected by changing delays in each segment of network connecting the first computer 110 and the second computer 120. These changing delays can include path latency affecting clock synchronization probe delay, network path asymmetry, or other network delay factors.

[0004] Existing clock synchronization schemes focus on network-based precision synchronization methods. The pulse per second (PPS) time synchronization protocol can use a dedicated line of known precise length to send a time pulse at a predefined interval (e.g., 100 ms). The precision time protocol (PTP) or datacenter time protocol (DTP) can attempt to synchronize the clocks in the end-hosts by synchronizing all switches or links between the end-hosts or by having the hosts participate in transparent mode. White Rabbit can extend PTP to use a dedicated synchronous Ethernet to transport time. However, each of these schemes requires hardware upgrades or special protocols and cannot be scaled to long distances or large numbers of nodes. The network time protocol (NTP) uses a time server to synchronize multiple computing devices, but the NTP is large, imprecise, and results in errors of hundreds of microseconds to tens of milliseconds. Thus, improved clock synchronization schemes are needed. SUMMARY

[0005] Various examples are now described, simply by way of introduction, to present some concepts that will be further described in particularity below. The summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended for use in limiting the scope of the claimed subject matter.

[0006] In a network computing environment, improved clock synchronization is needed. In particular, a new approach is needed so that high-resolution, software-based synchronization can be performed between large-scale environments (e.g., between thousands of computers). The embodiments described herein provide a predefined and highly fault-tolerant network effect scheme for clock synchronization. In contrast to other schemes that provide lower-resolution synchronization between a limited number of computing devices, the present scheme uses a predefined network topology and predefined and linearly independent time synchronization loops. The predefined network topology includes a predefined symmetric network topology, such as a ring network topology. In the predefined symmetric network topology, the loops are constructed by selecting computing nodes based on rules, where the loop construction takes a simplified line-up form to provide linearly independent loops. These schemes create a time synchronization architecture that can tolerate one or more computing node failures (e.g., hard failures, soft failures) without requiring recalculation of the network topology or loops and are able to add one or more irregular computing nodes and generate additional linearly independent loops. These schemes are also able to accommodate various multi-level physical network configurations and provide a physical network-aware multi-level network effect architecture for clock synchronization.

[0007] The systems and methods described herein provide a scheme that can be implemented in software, provide synchronization between large-scale environments, and provide high resolution (e.g., less than 100 ns). In some embodiments, these schemes can improve network effect decay by a factor of 3 to 5 in the presence of up to 5% failures in nodes or links, as compared to clock synchronization methods that use randomly constructed network topologies (e.g., the Welch method). Exemplary embodiments are described in connection with network computers, but it should be understood that the techniques described herein can work on other networked electronic devices, such as mobile phones, internet of things (IoT) devices, autonomous and semi-autonomous vehicles, and other networked electronic devices.

[0008] According to a first aspect of the present invention, there is provided a computer-implemented fault-tolerant network effect method for clock synchronization. In a first embodiment of the computer-implemented fault-tolerant network effect method, the method comprises: selecting a plurality of networked computing devices for clock synchronization; determining a symmetric network topology comprising a plurality of nodes, wherein each node of the plurality of nodes is associated with one networked computing device of the plurality of networked computing devices; determining a plurality of linearly independent clock synchronization loops, wherein each clock synchronization loop of the plurality of linearly independent clock synchronization loops has at least one node of the plurality of nodes as a starting point and an ending point; determining a network delay correction vector for each clock synchronization loop of the plurality of linearly independent clock synchronization loops in accordance with a network delay difference vector; and correcting a clock at one networked computing device of the plurality of networked computing devices in accordance with the network delay correction vector.

[0009] In a second embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form.

[0010] In a third embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the symmetric network topology comprises a ring network topology.

[0011] In a fourth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a first loop of the plurality of linearly independent clock synchronization loops comprises: a first node located at a first position, wherein the first node is connected to a second node located at a second position; and an end node located at an end position, wherein the end node is connected to the second node and the first node.

[0012] In a fifth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a first loop of the plurality of linearly independent clock synchronization loops comprises three or more nodes, wherein the three or more nodes comprise the first node, the second node, a third node, and the end node.

[0013] In a sixth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the second position is spaced apart from the first position by a network stride; the end position is spaced apart from the second position and the first position by the network stride.

[0014] In a seventh embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the network stride is determined according to a dimensionality of the ring network topology.

[0015] In an eighth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the ring network topology comprises a K-dimensional topology; the network stride is determined according to half of a size of the K-dimensional topology.

[0016] In a ninth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, the symmetric network topology comprises a plurality of network topology planes.

[0017] In a tenth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a plurality of linearly independent cross-plane clock synchronization loops is determined, wherein each cross-plane clock synchronization loop of the plurality of linearly independent cross-plane clock synchronization loops has at least one node of the plurality of nodes as a starting point and an end point.

[0018] In an eleventh embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a first cross-plane loop of the plurality of linearly independent cross-plane clock synchronization loops comprises: a first plane node at a first plane position in a first plane; a second plane node at a second plane position in a second plane, wherein the second plane node is connected to the first plane node; an end plane node at an end plane position in an end plane, wherein the end plane node is connected to the first plane node.

[0019] In a twelfth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a node fault in the symmetric network topology is identified; the plurality of linearly independent clock synchronization loops associated with the node fault are removed within the network delay correction vector; the network delay correction vector is recomputed for a remaining subset of the plurality of linearly independent clock synchronization loops according to the network delay difference vector.

[0020] In a thirteenth embodiment of the computer-implemented fault-tolerant network effect method according to the first aspect as such, a new node in the symmetric network topology is identified; a plurality of new linearly independent clock synchronization loops are determined, wherein the plurality of new linearly independent clock synchronization loops have the new node as a starting point and an ending point; and the network delay correction vector is recalculated according to the plurality of new linearly independent clock synchronization loops.

[0021] In a first embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, a fault-tolerant network effect system for clock synchronization is provided, wherein the system comprises a memory and a processor coupled to the memory. In the first embodiment of the computer-implemented fault-tolerant network effect system, the processor is configured to: select a plurality of network computing devices for clock synchronization; determine a symmetric network topology comprising a plurality of nodes, wherein each node in the plurality of nodes is associated with each network computing device in the plurality of network computing devices; determine a plurality of linearly independent clock synchronization loops, wherein each clock synchronization loop in the plurality of linearly independent clock synchronization loops has at least one node in the plurality of nodes as a starting point and an ending point; determine a network delay correction vector for each clock synchronization loop in the plurality of linearly independent clock synchronization loops according to a network delay difference vector; and correct a clock at one network computing device in the plurality of network computing devices according to the network delay correction vector.

[0022] In a second embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form.

[0023] In a third embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the symmetric network topology comprises a ring network topology.

[0024] In a fourth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, a first loop in the plurality of linearly independent clock synchronization loops comprises: a first node located at a first position, wherein the first node is connected to a second node located at a second position; and an end node located at an end position, wherein the end node is connected to the second node and the first node.

[0025] In a fifth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, a first loop in the plurality of linearly independent clock synchronization loops comprises three or more nodes, wherein the three or more nodes comprise the first node, the second node, a third node, and the end node.

[0026] In a sixth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the second position is spaced from the first position by a network stride; the end position is spaced from the second position and the first position by the network stride.

[0027] In a seventh embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the processor is further configured to determine the network stride according to a dimension size of the ring network topology.

[0028] In an eighth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the ring network topology comprises a K-dimensional topology; the network stride is determined according to half of a size of the K-dimensional topology.

[0029] In a ninth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the symmetric network topology comprises a plurality of network topology planes.

[0030] In a tenth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the processor is further configured to determine a plurality of linearly independent cross-plane clock synchronization loops, wherein each of the plurality of linearly independent cross-plane clock synchronization loops has at least one of the plurality of nodes as a starting point and an ending point.

[0031] In an eleventh embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, a first cross-plane loop of the plurality of linearly independent cross-plane clock synchronization loops comprises: a first plane node at a first plane position in a first plane; a second plane node at a second plane position in a second plane, wherein the second plane node is connected to the first plane node; an end plane node at an end plane position in an end plane, wherein the end plane node is connected to the first plane node.

[0032] In a twelfth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect as such, the processor is further configured to: identify a node fault in the symmetric network topology; remove, within the network delay correction vector, the plurality of linearly independent clock synchronization loops associated with the node fault; recompute the network delay correction vector for a remaining subset of the plurality of linearly independent clock synchronization loops according to the network delay difference vector.

[0033] In a thirteenth embodiment of the computer-implemented fault-tolerant network effect system according to the first aspect itself, the processor is further configured to: identify an added node in the symmetric network topology; determine a plurality of added linearly independent clock synchronization loops, wherein the plurality of added linearly independent clock synchronization loops are originated and terminated at the added node; and recompute the network delay correction vector based on the plurality of added linearly independent clock synchronization loops.

[0034] The methods can be performed by an apparatus, and the instructions on the computer-readable medium can be processed by the apparatus. Further features of the methods and instructions on the computer-readable medium arise from the functionality of the apparatus. Moreover, the explanations provided for each aspect and its implementation are equally applicable to the other aspects and corresponding implementations. Different embodiments can be implemented in hardware, software, or any combination thereof. Moreover, any of the above examples can be combined with any one or more of the other above examples to create a new embodiment within the scope of the application. BRIEF DESCRIPTION OF DRAWINGS

[0035] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. The drawings are intended to be illustrative, and not restrictive, of various embodiments described herein.

[0036] Figure 1 Conventional clock synchronization in an exemplary network computing environment is shown.

[0037] Figure 2A And Figure 2B A network effect clock synchronization scheme in an exemplary network computing embodiment is shown.

[0038] Figure 3 A predefined symmetric network topology in an exemplary network computing embodiment is shown.

[0039] Figure 4A And Figure 4B An expanded planar symmetric network topology in an exemplary network computing embodiment is shown.

[0040] Figure 5 An expanded planar symmetric network topology with cross-planar loops built in an exemplary network computing embodiment is shown.

[0041] Figure 6A And Figure 6B A predefined fault-tolerant symmetric network topology in an exemplary network computing embodiment is shown.

[0042] Figure 7 An enhanced symmetric network topology in an exemplary network computing embodiment is shown.

[0043] Figure 8A AndFigure 8B A first physical network-aware multi-level network scheme in an example network computing embodiment is shown.

[0044] Figure 9 A second physical network-aware multi-level network scheme in an example network computing embodiment is shown.

[0045] Figure 10 A method flow diagram showing a method of providing a network effect clock synchronization method in an example embodiment is shown.

[0046] Figure 11 A method flow diagram showing a method of providing a fault tolerant network effect method for clock synchronization in an example embodiment is shown.

[0047] Figure 12 A block diagram of circuitry in the form of a processing system, according to an example embodiment, for implementing the system and method of providing time synchronization described above in conjunction with Figures 1 to 11 DETAILED DESCRIPTION

[0048] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods described in connection with the appended drawings are capable of implementation in any number of ways, whether currently known or unknown. The present disclosure is in no way limited to the illustrative implementations, set forth below, and including the examples as set forth herein, but is capable of broader implementation consistent with the principles of the present disclosure as set forth below and including the examples as set forth herein.

[0049] Figure 2A and Figure 2B A network effect clock synchronization scheme 200 in an example network computing embodiment is shown. In Figure 2A ​In the illustrated example embodiment, the first computer 210, the second computer 220, and the third computer 230 can have measured network delay timing latencies of 19 ms, -14 ms, and 4 ms. In one example, the 19 ms network delay timing latency is determined by the first computer 210 sending a time-stamped message to the second computer 220 and the second computer 220 comparing the message receive time to the message timestamp. Once each measured network delay timing latency is determined, the three latencies of 19 ms, -14 ms, and 4 ms can be added together to determine an initial total loop timing margin of 9 ms. This initial total loop timing margin represents the time inaccuracy that can occur if the first computer 210 is synchronized in time according to the network delays from the first computer 210 to the second computer 220, from the second computer 220 to the third computer 230, and from the third computer 230 back to the first computer 210. This loop margin can be represented as a difference vector ΔΡ = [4, 19, 14] and a loop matrix A = [1, 1, -1], with the result of the least-norm solution of this loop margin vector Y = A ΔΡ = [9].

[0050] The scheme 200 is intended to compensate for this network effect by identifying a network correction vector ΔF. The network correction vector ΔF can be identified by recognizing a vector N that also solves Y = AN, where the network correction vector ΔF = ΔΡ - N, such that A ΔF = A ΔΡ - AN = Y - Y = 0. Using the pseudo-inverse, N = A T (AA T ) –1 Y, which is the least-norm solution. A T (AA T ) –1 = [1 / 3 1 / 3 -1 / 3] and N = [3 3 -3], with the result being a network delay correction vector ΔF = [1 16 17], as Figure 2B illustrated. This network delay correction can be applied to the loop consisting of the first computer 210, the second computer 220, and the third computer 230 to compensate the loop timing margin to 0, thereby compensating for the timing inaccuracy due to network delay. When adjusting the internal time of each of the first computer 210, the second computer 220, and the third computer 230, each computer can determine a measured network delay timing latency (e.g., 19 ms, -14 ms, and 4 ms) and apply the network delay correction to provide a corrected internal time adjustment. This network delay correction can be used for larger network architectures, as Figure 3 illustrated.

[0051] Figure 3A predefined symmetric network topology 300 in an exemplary network computing embodiment is shown. In one example, a K-dimensional symmetric network topology is virtually formed by a plurality of nodes, where the nodes are servers selected from a cluster of servers in an existing physical network. The predefined K-dimensional symmetric network topology is predefined and the number of servers selected from the physical network is based on the predefined dimensionality of the K-dimensional symmetric network topology. In Figure 3 In the illustrated embodiment, the symmetric network topology 300 is a two-dimensional (2-dimensional, 2D) representation of a ring network topology comprising 5x3 nodes, where each node has 4 connections. In Figure 3 In the illustrated 2D ring network example, the loop size is selected to be a 2x2 loop connected from left to right and from top to bottom, changing a column when moving horizontally (e.g., horizontal stride of 1) and then changing a row when moving vertically (e.g., vertical stride of 1). For example, node 311 is connected to node 321, node 322, node 312, and back to node 311, forming a first node loop 310. Node loops can wrap around, for example, node 311 is connected to node 351, node 353, node 313, and back to node 311. Linearly independent time synchronization loops can be generated using a simplified row echelon form. In one example, a loop can be generated for a given node at position (i,j) by connecting (i,j) to (i+1,j), (i+1,j+1), (i,j+1), and back to (i,j), where i and j start from connecting node 1 to any two nodes and wrap around to node 1 when i or j exceeds the dimensionality of the topology (e.g., wrap around from 351 to 311, from 313 to 311). For each loop, a network correction vector can be computed to compensate for network effects and provide improved clock synchronization in the predefined symmetric network topology 300. Although Figure 3 While a 2D ring network topology is shown, the identification of nodes and generation of linearly independent time synchronization loops can be extended to other symmetric network topologies, such as dragonfly and hypercube network topologies.

[0052] Figure 4A And Figure 4B An expanded planar symmetric network topology 400 in an exemplary network computing embodiment is shown. In one example, Figure 3 The illustrated 2D network topology can be expanded to add additional network planes, forming a 3D network topology. Additional network planes can be added to expand the network topology to a K-dimensional symmetric network topology. However, adding additional network planes can cause an exponential increase in the number of nodes and network effect computations if each plane node in each plane is connected to adjacent planes as described above. To maintain the number of nodes and network effect computations when expanding the number of network dimensions, multiple stride planes can be used. Figure 4AA horizontal stride 410 or stride vector size of 2 is shown. Similarly, Figure 4B A vertical stride 420 or stride vector size of 2 is shown. Within a given plane with a stride vector size (e.g., (stride size, stride size)), a loop is generated for a given node at position (i, j) by connecting (i, j) to (i + stride size, j), (i + stride size, j + stride size), (i, j + stride size), and back to (i, j), and wrapping around to node 1 when i or j exceeds the topology dimension, similar to Figure 3 A loop is generated for a given node at position (i, j). The given plane is a stride 1 plane (e.g., stride vector size (1, 1)).

[0053] Figure 5 An extended plane-symmetric network topology 500 with cross-plane loops is shown in an example network computing embodiment. The loop stride can be based on the number of network planes. In one example, for K = 2 network topologies with each plane size of 3 x 3 or larger, stride 1 can be used for both vertical and horizontal, as shown. Figure 3 For K = 4 network topologies with each plane size of 6 x 6 or larger, stride 2 can be used for both vertical and horizontal. Similarly, for K = 6, 8, 10, and above, stride K / 2 = 3, 4, 5, and above can be used for both vertical and horizontal.

[0054] In the example of Figure 5 Cross-plane loops are shown between stride 2 planes and stride 1 planes (e.g., horizontal dimension), with generally larger stride size planes between planes. A simplified row echelon form can be used to identify linearly independent loops. Between planes, a loop is generated by connecting (i, j) to (i + 1, j),..., (i + stride size, j), and back to (i, j) with wrap around to i = 1 when i exceeds the topology dimension, and by connecting (i, j) to (i + 1, j),..., (i, j + stride size), and back to (i, j) with wrap around to j = 1 when j exceeds the topology dimension. As shown, Figure 5 Between planes 510, 520, and 530, node 511 is connected to node 521, node 521 is connected to node 531, and then node 531 is back to node 511. For each loop, a network correction vector can be calculated to compensate for network effects and provide improved clock synchronization in the extended multiple plane-symmetric network topology 500.

[0055] Figure 6A and Figure 6BA predefined fault-tolerant symmetric network topology 600 in an exemplary network computing embodiment is shown. The proposed predefined symmetric network topology provides improved network effect clock synchronization even when a given node or link fails. For a given failure, the predetermined loop associated with the failure point is removed. In Figure 6A In the example shown, when a failure in node 610 is detected, loops 612, 614, 616, and 618 are removed. In Figure 6B In the example shown, when a failure in link 620 is detected, loops 622 and 624 are removed. In each case, the remaining linearly independent time synchronization loops can be used to provide time synchronization without the need to recalculate network correction vectors or rebuild alternative loops.

[0056] Figure 7 An enhanced symmetric network topology 700 in an exemplary network computing embodiment is shown. If a new node 715 is added to an existing 2D torus network, additional linearly independent time synchronization loops can be added without changing the existing time synchronization loops. For a new node added to an existing 2D torus network topology, four links can be added to the four neighboring 2x2 nodes. In Figure 7 In the example shown, new node 715 can form a loop with nodes 711 and 712, a loop with nodes 711 and 721, a loop with nodes 721 and 722, and a loop with nodes 712 and 722. For a new node added to an expanded multi-plane symmetric network topology, in any cross-step plane, four additional links can be added to the four neighboring 2x2 nodes. The resulting enhanced symmetric network topology 700 maintains loop linear independence while providing improved time synchronization.

[0057] Figure 8A and Figure 8B A first physical network-aware multi-level network scheme 800 in an exemplary network computing embodiment is shown. As Figure 8AAs shown, the centralized multi-level ring network can include one or more network core nodes 810, one or more network backbone nodes 820, and one or more network leaf nodes 830. Each level of the centralized multi-level ring network can form one or more dimensions of a ring network topology and provide time synchronization within and across levels. Within the one or more network backbone nodes 820, a first level of a predefined symmetric ring network topology can be defined by selecting a plurality of nodes from within the one or more network leaf nodes 830. Between the one or more network backbone nodes 820 within a given core, a second level of a predefined symmetric ring network topology can be defined by selecting a plurality of nodes from the one or more network backbone nodes 820. Between the one or more network core nodes 810, a third level of a predefined symmetric ring network topology can be defined by selecting a plurality of nodes from the one or more network core nodes 810. As Figure 8B As shown, the centralized multi-level ring network forming a loop can be represented as network core nodes 810 along the z-axis, network backbone nodes 820 along the y-axis, and network leaf nodes 830 along the x-axis. In the first physical network-aware multi-level network scheme 800, each level forms Figure 8B As shown, the one or more dimensions of the ring network topology, network time is synchronized between all nodes in the ring network topology.

[0058] Figure 9 A second physical network-aware multi-level network scheme 900 in an exemplary network computing embodiment is shown. As Figure 9 As shown, the centralized multi-level ring network can include one or more network core nodes 910, one or more network backbone nodes 920, and one or more network leaf nodes 930. In contrast to the first scheme 800, which synchronizes time between all network nodes, in the second physical network-aware multi-level network scheme 900, time is independently synchronized within each level. Within the leaf level, one or more nodes within each leaf level are selected to generate each of the one or more network backbone level nodes. Within the network leaf nodes 930, one leaf node 932, 934, 936, and 938 is selected from each network leaf node to form nodes within each backbone level ring network. Within the network backbone nodes 920, one backbone node 922 and 924 is selected from each network backbone node to form nodes within each core level ring network. As a result of the independent synchronization within each level, the second physical network-aware multi-level network scheme 900 reduces complexity and computational requirements while providing improved time synchronization within and across multiple network levels.

[0059] Figure 10A flowchart of a method of providing a network effect clock synchronization method 1000 in an exemplary embodiment is shown. The method 1000 starts at 1010 and includes selecting a plurality of network computing devices for clock synchronization. At 1020, a symmetric network topology is determined. The symmetric network topology includes a plurality of nodes, each of the plurality of nodes being associated with each of the plurality of network computing devices.

[0060] At 1030, a plurality of linearly independent clock synchronization loops is determined. Each of the plurality of linearly independent clock synchronization loops originates and terminates at each of the plurality of nodes. In an exemplary embodiment, the linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form. In an exemplary embodiment, the symmetric network topology includes a ring network topology. In an exemplary embodiment, a first loop of the plurality of linearly independent clock synchronization loops includes a first node located at a first position and an end node located at an end position, wherein the first node is connected to a second node located at a second position and the end node is connected to the second node and the first node. In an exemplary embodiment, the second position is spaced from the first position by a network stride; and the end position is spaced from the second position and the first position by a network stride.

[0061] At 1040, a network delay correction vector is determined for each of the plurality of linearly independent clock synchronization loops according to a network delay difference vector. The network delay correction vector is determined according to a measured network delay timing latency within each of the plurality of linearly independent clock synchronization loops. The measured network delay timing latency can be represented as a difference vector ΔΡ and a loop matrix A, and a result of a least norm solution of the measured network delay timing latency is a loop residual vector AΔΡ = Y. The network delay correction vector ΔF is determined based on identifying a vector N that also solves Y = AN, where the network correction vector ΔF = ΔΡ - N, such that AΔF = AΔΡ - AN = Y - Y = 0. Using a pseudo-inverse, the vector N = A T (AA T ) – 1 Y. The network delay correction vector ΔF is then calculated as ΔΡ - A T (AA T ) –1 Y. The network delay correction vector ΔF causes a network effect compensation loop timing residual to be 0, thereby compensating for timing compensation inaccuracy due to network delay. At 1050, an internal clock at one or more computing nodes is corrected according to the measured network delay timing latency and the network delay correction vector ΔF.

[0062] Figure 11A flowchart of a method of providing a fault-tolerant network effect method for clock synchronization 1100 in an example embodiment is shown. The method 1100 starts from 1110 and includes selecting a plurality of network computing devices for clock synchronization. At 1120, a symmetric network topology is determined. The symmetric network topology includes a plurality of nodes, each of the plurality of nodes being associated with each of the plurality of network computing devices.

[0063] At 1130, a network stride is determined according to a dimension size of a ring network topology. In an example embodiment, the ring network topology includes a K-dimensional topology, and the network stride is determined according to half of a size of the K-dimensional topology. In an example embodiment, the symmetric network topology includes a plurality of network topology planes.

[0064] At 1140, a new node is identified in the symmetric network topology, and a plurality of new linearly independent clock synchronization loops are determined, wherein the plurality of new linearly independent clock synchronization loops have the new node as a start point and an end point.

[0065] At 1150, a plurality of linearly independent clock synchronization loops within a stride plane are determined. Each of the plurality of linearly independent clock synchronization loops has each of the plurality of nodes as a start point and an end point. At 1160, a plurality of linearly independent clock synchronization loops between stride planes are determined. Each of the plurality of linearly independent clock synchronization loops between stride planes has each of the plurality of nodes as a start point and an end point. In an example embodiment, a first cross-plane loop of the plurality of linearly independent clock synchronization loops between stride planes includes a first plane node at a first plane location in a first plane, a second plane node at a second plane location in a second plane, wherein the second plane node is connected to the first plane node, and an end plane node at an end plane location in an end plane, wherein the end plane node is connected to the first plane node.

[0066] At 1170, a plurality of new linearly independent clock synchronization loops are determined. The plurality of new linearly independent clock synchronization loops have the new node as a start point and an end point. In an example embodiment, the linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form. In an example embodiment, the symmetric network topology includes a ring network topology. In an example embodiment, a first loop of the plurality of linearly independent clock synchronization loops includes a first node at a first location and an end node at an end location, wherein the first node is connected to a second node at a second location, and the end node is connected to the second node and the first node. In an example embodiment, the second location is spaced from the first location by the network stride; and the end location is spaced from the second location and the first location by the network stride.

[0067] At 1180, a network delay correction vector is determined for each of the plurality of linearly independent clock synchronization loops according to the network delay difference vector. As described above, the network delay correction vector AF is calculated as AP - A T (AA T ) –1 Y, such that the network effect compensates the loop timing margin to 0. At 1190, the internal clock at one or more computing nodes is corrected according to the measured network delay timing latency and the network delay correction vector AF.

[0068] Figure 12 A block diagram of circuitry in the form of a processing system for implementing the system and method for providing time synchronization described above in conjunction with Figures 1 to 11 not all of the components are utilized in various embodiments. One exemplary computing device in the form of a computer 1200 can include a processing unit 1202, memory 1203, removable storage 1210, and non-removable storage 1212. While the exemplary computing device is illustrated and described as computer 1200, the computing device can be in different forms in different embodiments. For example, the computing device can be a smartphone, a tablet, a smartwatch, or other computing device that includes the same or similar elements as those shown and described in conjunction with Figure 12 the exemplary embodiments. Devices such as smartphones, tablets, and smartwatches are commonly referred to as mobile devices or user devices. Further, while various data storage elements are shown as part of computer 1200, storage can also include or alternatively include cloud-based storage accessed over a network, such as the Internet or server-based storage.

[0069] The memory 1203 can include volatile memory 1214 and nonvolatile memory 1208. The computer 1200 can also include, or be in the communication environment with, various computer-readable media, including volatile memory 1214 and non-volatile memory 1208, removable storage 1210 and non-removable storage 1212, or have access to various computer-readable media, including volatile memory 1214 and non-volatile memory 1208, removable storage 1210 and non-removable storage 1212. Computer storage includes random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0070] The computer 1200 can include, or be in the communication environment with, input interface 1206, output interface 1204, and communication interface 1216. The output interface 1204 can include a display device such as a touchscreen, etc. that can also function as an input device. The input interface 1206 can include one or more of a touchscreen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated within the computer 1200 or coupled to the computer 1200 through a wired or wireless data connection, and other input devices. Alternatively, in some embodiments, the output interface 1204 and the input interface 1206 are combined in a single interface. The computer 1200 can operate in a networked environment using a communication connection to one or more remote computers. The remote computer can include a personal computer (PC), a server, a router, a network PC, a peer device or other common DFD network switch, etc. The communication connection can include a local area network (LAN), a wide area network (WAN), a cellular, Wi-Fi, Bluetooth, or other network. According to one embodiment, the various components of the computer 1200 are connected to a system bus 1220.

[0071] Computer readable instructions stored on a computer-readable medium that when executed by the processing unit 1202 of the computer 1200, e.g., program 1218. In some embodiments, the program 1218 includes software that, when executed by the processing unit 1202, performs the metadata sharing operations according to any of the embodiments included herein. Hard disks, CD-ROMs, and RAM are some examples of articles including non-transitory computer-readable media (e.g., storage devices). The terms computer readable medium and storage devices do not include a carrier wave because a carrier wave is deemed to be a transitory signal. Storage can also include network storage, e.g., a storage area network (SAN). The computer program 1218 can also include an instruction module that, when processed, causes the processing unit 1202 to perform one or more of the methods or algorithms described herein. In some embodiments, the computer 1200 can include other modules or additional modules for performing any one or combination of the steps described in the embodiments. Further, any additional or alternative embodiments or aspects of the methods as shown in any of the drawings or recited in any of the claims are also contemplated as including similar modules.

[0072] While some embodiments have been described in detail, modifications can be made. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential execution of steps, to achieve desirable results. Other steps can be provided, or steps can be eliminated, from the described flows, and other components can be added to, or removed from, the described systems. Other embodiments can be within the scope of the following claims.

[0073] It should also be understood that a software including one or more computer-executable instructions can be installed and sold apart from the one or more computing devices on which the software is executed, and that such software can be a stand-alone product or part of an on-line tool or service. For example, software can be downloaded (e.g., from the Internet) into a memory of a computing device.

[0074] Moreover, those skilled in the art will appreciate that the application is practiced with in the context of a business method, a computer-implemented process, a computer program product, a distributed computing context, or a suitably-programmed computer with a computer-readable memory. In this sense, the application can be embodied as a computer program product of a computer- readable storage medium (e.g., a non-transitory computer-readable medium) with computer readable program code embodied in the medium. The application can also be practiced in the context of a network or distributed computing environment. Additionally, the present application can be implemented as a routine embedded in a computer program product, a program, a set of instructions, or a remote- accessible computer that either runs on a machine or that interfaces with a database or server computer. The program, set of instructions, or routines can run on a computer or server computer, or can be executed by a mobile device, a personal digital assistant, a cell phone, or other device.

[0075] The components of the illustrative devices, systems and methods employed by the illustrated embodiments can be implemented, at least partially, in digital electronic circuitry, analog electronic circuitry, or computer hardware, firmware, software, or combinations thereof. These components can be implemented as a computer program product, e.g., a computer program tangibly embodied in an information carrier, or a machine-readable storage device, for execution by a programmable processor; and / or as a program of instructions carried by a data signal, for example, a computer program downloaded from a website or stored on a storage medium. Such implementation can include any number of processors, operating in concert or in succession, and can include any number of storage media, memory devices, and / or data transmission devices.

[0076] A computer program, such as implemented with the described embodiments, can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed in one computer or in multiple computers at one site or distributed across multiple sites and interconnected by a communication network. Also, functional programs, codes, and code segments for accomplishing the techniques described herein can be easily interpreted "as being within the scope of the claims," by programmers skilled in the functional art(s) to which the technology pertains. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and / or generating output. Method steps can also be implemented by, and computer program(s) executed by, dedicated logic circuitry, e.g., an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or any other hardware device suitable for implementing the techniques described herein.

[0077] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0078] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory and / or a random access memory. Computer elements generally include a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes, or is operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks, e.g., magnetic disks, internal hard disks, or removable disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0079] Those skilled in the art will appreciate that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0080] As used herein, “machine-readable medium” means a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, high-speed cache memory, other types of storage (e.g., erasable programmable read-only memory (EEPROM)) and / or any suitable combination thereof. “Machine-readable medium” should be considered to include a single medium or multiple media capable of storing processor instructions (e.g., a centralized or distributed database, or associated caches and servers). “Machine-readable medium” should also be considered to include any medium or combination of media capable of storing instructions executable by processing unit 1202 to cause processing unit 1202 to perform one or more methods described herein when executed by processing unit 1202. Therefore, “machine-readable medium” refers to a single storage device or apparatus as well as a cloud-based storage system including multiple storage devices.

[0081] Furthermore, without departing from the scope of the invention, the technologies, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments can be combined or integrated with other systems, modules, technologies, or methods. Other items shown or described as coupled, directly coupled, or communicating with each other may be indirectly coupled or communicated electrically, mechanically, or otherwise through some interface, device, or intermediate component. Other examples of variations, substitutions, and modifications can be determined by those skilled in the art and may be exemplified without departing from the scope disclosed herein.

[0082] While the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made to the invention without departing from its scope. Therefore, the specification and drawings are to be regarded only as a description of the invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations, or equivalents falling within the scope of the invention.

Claims

1. A computer-implemented fault-tolerant network effect method for clock synchronization, characterized in that, The method comprises: selecting a plurality of network computing devices for clock synchronization; determining a symmetric network topology comprising a plurality of nodes, wherein each node of the plurality of nodes is associated with a network computing device of the plurality of network computing devices; determining a plurality of linearly independent clock synchronization loops, wherein each clock synchronization loop of the plurality of linearly independent clock synchronization loops has at least one node of the plurality of nodes as a starting point and an ending point; determining a network delay correction vector for each clock synchronization loop of the plurality of linearly independent clock synchronization loops according to a network delay difference vector; correcting a clock at a network computing device of the plurality of network computing devices according to the network delay correction vector.

2. The method of claim 1, wherein, The linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form.

3. The method according to claim 1 or 2, characterized in that, The symmetric network topology comprises a ring network topology.

4. The method according to claim 1 or 2, characterized in that, A first loop of the plurality of linearly independent clock synchronization loops comprises: a first node at a first location, wherein the first node is connected to a second node at a second location; an end node at an end location, wherein the end node is connected to the second node and the first node.

5. The method of claim 4, wherein, The first loop of the plurality of linearly independent clock synchronization loops further comprises a fourth node.

6. The method of claim 4, wherein: the second location is spaced apart from the first location by a network stride; the end location is spaced apart from the second location and the first location by the network stride.

7. The method of claim 6, wherein, Further comprising: determining the network stride according to a dimension size of the symmetric network topology.

8. The method of claim 7, wherein: the symmetric network topology comprises a K-dimensional topology; the network stride is determined according to half of the dimension size of the K-dimensional topology.

9. The method of claim 1 or 2, wherein, The symmetric network topology comprises a plurality of network topology planes.

10. The method of claim 1 or 2, wherein, Further comprising: determining a plurality of linearly independent cross-plane clock synchronization loops, wherein each cross-plane clock synchronization loop of the plurality of linearly independent cross-plane clock synchronization loops has at least one node of the plurality of nodes as a starting point and an ending point.

11. The method of claim 10, wherein, A first cross-plane loop of the plurality of linearly independent cross-plane clock synchronization loops comprises: a first plane node at a first plane location in a first plane; a second plane node at a second plane location in a second plane, wherein the second plane node is connected to the first plane node; an end plane node at an end plane location in an end plane, wherein the end plane node is connected to the first plane node.

12. The method of claim 1 or 2, wherein, Further comprising: identifying a node failure in the symmetric network topology; removing, within the network delay correction vector, the plurality of linearly independent clock synchronization loops associated with the node failure; recomputing the network delay correction vector for a remaining subset of the plurality of linearly independent clock synchronization loops according to the network delay difference vector.

13. The method of claim 1 or 2, wherein, Further comprising: identifying an added node in the symmetric network topology; determining a plurality of added linearly independent clock synchronization loops, wherein the plurality of added linearly independent clock synchronization loops have the added node as a starting point and an ending point; recomputing the network delay correction vectors according to the plurality of newly linearly independent clock synchronization loops.

14. A fault-tolerant network effect system for clock synchronization, characterized in that, The system comprises: a memory; a processor coupled to the memory for: selecting a plurality of network computing devices for clock synchronization; determining a symmetric network topology comprising a plurality of nodes, wherein each node of the plurality of nodes is associated with a network computing device of the plurality of network computing devices; determining a plurality of linearly independent clock synchronization loops, wherein each clock synchronization loop of the plurality of linearly independent clock synchronization loops has at least one node of the plurality of nodes as a starting point and an ending point; determining a network delay correction vector for each clock synchronization loop of the plurality of linearly independent clock synchronization loops according to a network delay difference vector; correcting a clock at a network computing device of the plurality of network computing devices according to the network delay correction vector.

15. The system of claim 14, wherein, The linear independence of the plurality of linearly independent clock synchronization loops is based on a simplified row echelon form.

16. The system of claim 14 or 15, wherein, The symmetric network topology comprises a ring network topology.

17. The system of claim 14 or 15, wherein, A first loop of the plurality of linearly independent clock synchronization loops comprises: a first node at a first location, wherein the first node is connected to a second node at a second location; an end node at an end location, wherein the end node is connected to the second node and the first node.

18. The system of claim 17, wherein, The first loop of the plurality of linearly independent clock synchronization loops further comprises a fourth node.

19. The system of claim 17, wherein: the second location is spaced apart from the first location by a network stride; the end location is spaced apart from the second location and the first location by the network stride.

20. The system of claim 19, wherein, The processor is further configured to determine the network stride according to a dimension size of the symmetric network topology.

21. The system of claim 20, wherein: the symmetric network topology comprises a K-dimensional topology; the network stride is determined according to half of a dimension size of the K-dimensional topology.

22. The system of claim 14 or 15, wherein, The symmetric network topology comprises a plurality of network topology planes.

23. The system of claim 14 or 15, wherein, The processor is further configured to determine a plurality of linearly independent cross-plane clock synchronization loops, wherein each cross-plane clock synchronization loop of the plurality of linearly independent cross-plane clock synchronization loops has at least one node of the plurality of nodes as a starting point and an ending point.

24. The system of claim 23, wherein, A first cross-plane loop of the plurality of linearly independent cross-plane clock synchronization loops comprises: a first plane node at a first plane location in a first plane; a second plane node at a second plane location in a second plane, wherein the second plane node is connected to the first plane node; an end plane node at an end plane location in an end plane, wherein the end plane node is connected to the first plane node.

25. The system of claim 14 or 15, wherein, The processor is further configured to: identify a node failure in the symmetric network topology; remove, within the network delay correction vectors, the plurality of linearly independent clock synchronization loops associated with the node failure; recompute the network delay correction vectors for a remaining subset of the plurality of linearly independent clock synchronization loops according to the network delay difference vector.

26. The system of claim 14 or 15, wherein, The processor is further configured to: identify a new node in the symmetric network topology; determine a plurality of new linearly independent clock synchronization loops, wherein the plurality of new linearly independent clock synchronization loops have the new node as a starting point and a terminal point; recompute the network delay correction vector according to the plurality of new linearly independent clock synchronization loops.

Images