Network clock management via data server

Abstract

The data server's internal time is compared against the respective times of each of a number of devices in the network. The network may require strict time synchronization. The data server utilizes a number of high performance oscillators to maintain its internal time. The data server analyzes the compared times and detects that the time maintained by another device in the network has deviated by more than a threshold value. In response to detecting that the time maintained by another device has deviated by more than a threshold value, remedial action is taken.

Description

[[Technical Field]]

[0001] The present invention relates to network clock management. [[Background Art]]

[0002] Modern computing devices used to support an organization are often expected to operate in various ways to enable the organization to comply with various regulations and meet various standards. These standards and regulations may relate to many different computing variables, such as logging behavior or timing behavior. For example, financial institutions are often required to maintain strict time synchronization of computing equipment compared to an external clock (e.g., compared to Coordinated Universal Time (UTC)). If a financial institution uses equipment that deviates from these time synchronization requirements (such that the equipment "drifts" to be faster or slower than regulations allow), the financial institution may face substantial costs. For example, costs may be accrued in direct proportion to the gross profit of the financial institution.

[0003] Therefore, organizations often seek computing devices and network infrastructure individually or collectively configured to meet such regulations and standards. Specifically, organizations may use various techniques to enable the time of day (TOD) clock on each device to be synchronized to the degree required in modern high-performance computing architectures. For example, a financial institution may use one or more time servers that read time from a reference clock and distribute this time to various devices in a network to increase the likelihood that each of these devices has access to the correct time (e.g., such that each device subsequently uses the received time as the device's respective TOD). Time servers frequently receive this reference clock from Global Positioning System (GPS) signals. [Overview of the project]

[0004] Aspects of this disclosure relate to methods, systems, and computer program products relating to clock management of multiple computing devices on a network. For example, this method includes comparing the internal time of a network data server to the time of each of several devices on the network. The data server maintains its internal time using multiple high-performance oscillators. This method further includes the data server, which analyzes the compared times, detecting if the time maintained by another device on the network is off by more than a threshold. This method also includes taking corrective actions in response to detecting that the time maintained by another device is off by more than a threshold. Systems and computer products configured to perform the above method are also disclosed.

[0005] This disclosure also includes a method for comparing the average internal time of multiple data servers, each maintaining its own internal time using multiple high-performance oscillators, with respect to multiple devices, where both the multiple devices and the multiple data servers are part of a network utilizing a precision time protocol (PTP). The method also includes detecting, by analyzing the compared times, that the time maintained by another device in the network deviates by more than a threshold from the average internal time of all the multiple data servers. The method also includes taking corrective actions in response to identifying that the time maintained by another device in the network deviates by more than a threshold.

[0006] The above summary is not intended to describe any of the embodiments or implementations of this disclosure.

[0007] The drawings included in this application are incorporated herein and form part of this specification. Each drawing illustrates an embodiment of the disclosure and, together with the description, helps to illustrate the principles of the disclosure. Each drawing is merely an example of a particular embodiment and does not limit the disclosure. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates a conceptual system where a controller can manage the timing of data being stored and utilized by multiple computing devices on a network. [Figure 2] This figure shows a conceptual box diagram of an exemplary component of the controller in Figure 1. [Figure 3] Figure 1 shows an illustrative flowchart illustrating how the controller can manage time to be used across multiple computing devices on the network. [Modes for carrying out the invention]

[0009] The present invention accepts various modifications and alternative forms, the details of which are shown as examples in the drawings and described in detail. However, it should be understood that the present invention is not intended to be limited to the specific embodiments described. On the contrary, it is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present invention.

[0010] Aspects of this disclosure relate to managing the clocks of computing devices in a network, while more specific aspects of this disclosure relate to comparing the internal time of one or more data servers using multiple high-performance oscillators with the time of other devices on a local or remote network, and identifying, searching for, and performing corrective actions in response to identifying one or more devices in the network that are experiencing clock drift relative to these data servers. This disclosure is not necessarily limited to such applications, but various aspects of this disclosure will be understood through the description of various embodiments using this context.

[0011] As described in the background technology section, many organizations are required (or tend to be required) to meet various regulations and standards regarding maintaining time synchronization across their computing devices (hereinafter collectively referred to as “devices”). These time synchronization regulations often concern global standards (e.g., standards based on other organizations or regulatory bodies, or both), making it crucial not only whether devices have clocks synchronized with each other, but also with respect to these external entities. To meet such regulations, organizations typically use one or more servers dedicated to the task of collecting time from a (presumably reliable) reference clock and then distributing this time to other devices in the network. Such servers assigned (if not exclusively) to collecting and distributing time across the network are referred to herein as time servers. Time servers are distinct entities from “data servers,” as described herein, where data servers are computing devices configured to store data and perform computational operations for the organization (and not entrusted with being primary time distribution devices). Generally speaking, a data server can be understood as an organization's mainframe or PC data server.

[0012] Such traditional architectures typically rely on various network components (e.g., network switches) to function as expected. For example, if a network switch in an organization's network begins to fail (whether by delaying or altering the time signal sent from the time server, or by some other error as understood by those skilled in the art), some computing devices that rely on that network switch to receive clock information from the time server may drift out of true time as a result of these failed components.

[0013] Some traditional architectures attempt to solve this problem by making various efforts to ensure that all devices are generally synchronized with one another, so as to avoid relative discrepancies. For example, some traditional architectures utilize a high-precision time protocol (PTP) across the network, along with synchronization programs that ensure all devices are synchronized using yet another next-generation (YANG) model. Additionally or alternatively, some traditional architectures may improve the fidelity of messages sent across the network by utilizing a server time protocol (STP), a server-wide function that presents a single perspective of time to the relevant Type 1 hypervisor via STP messages sent over one or more physical data links between servers. Yet another example is a traditional architecture that utilizes a network time protocol (NTP) for clock synchronization. Traditional architectures that use such techniques (whether alone or in conjunction with one or more of the aforementioned procedures) can be quite effective in ensuring that all devices are synchronized with one another.

[0014] However, traditional architectures may fail to identify a specific faulty device that causes individual devices to become misaligned. As those skilled in the art will understand, there are numerous reasons why identifying a specific point of failure is beneficial. For example, the inability to identify a point of failure can make traditional architectures more susceptible to failure over time, such as when the number of failed devices increases to the point where synchronization efforts within the network become ineffective. In another example, the inability to identify a point of failure can make it substantially more difficult for traditional architectures to recover from a final failure (for example, because it is unclear which component needs to be replaced / repaired). In particular, after a failure, traditional methods (where the faulty device is unknown) may involve extensive "trial and error" methods, where individual components are replaced and then the network is tested (and if this does not solve the problem, another component is replaced and the network is retested, etc.).

[0015] Furthermore, even if such conventional network synchronization efforts succeed in synchronizing all devices in the network relative to one another, there is no guarantee that these devices will be synchronized as required relative to an external clock. For example, for conventional synchronization efforts in conventional architectures, it may be difficult or impossible to detect whether the time server itself is failing or receiving a corrupted time source. For instance, if a malicious actor impersonates the reference clock signal used by the time server (e.g., a GPS signal, or in some conventional architectures, even a PTP signal), conventional synchronization efforts may not be able to technically detect that the time received and distributed by the time server is not synchronized relative to an external (e.g., true / actual) time such as UTC.

[0016] In some cases, a network might attempt to solve this by including a single high-quality oscillator configured to maintain internal time within at least one computing device in the network. However, while such a device may be configured to determine that something is wrong when internal time does not match external time, it cannot determine whether the failure is in the device itself or in an external device.

[0017] Aspects of this disclosure can solve or otherwise address these technical problems of conventional computing architectures. For example, the above technical problems can be solved by using a data server that includes multiple high-performance oscillators configured to maintain the internal time of the data server (e.g., the oscillators are high-performance as a result of the oscillators being specified to ± about 2 parts per million at the time of their own construction). Such a data server may further include software that works with the multiple high-performance oscillators to maintain the internal time (e.g., not drifting by more than 2 milliseconds per day). Aspects of this disclosure relate to comparing the internal time (TOD) of one or more such data servers with the internal TOD of various devices to detect devices that are out of sync (wherein used herein, “out of sync” relates to a computing device being ahead or behind a desired time by a non-nominal amount exceeding a threshold, justifying correction), and taking corrective actions in response to such detection. A computing device that includes a processing unit that executes instructions stored in memory may provide this function, and such computing device is referred to herein as a controller. The controller may be configured to detect that any device in the computing environment is out of sync by comparing the different internal times of various devices with one or more data servers utilizing multiple high-performance oscillators as described herein, and further to identify whether this out of sync is caused by a faulty device or by an error / attack related to a received reference clock.

[0018] For example, Figure 1 shows an environment 100 in which a controller 110 monitors and manages the internal time drift of a data server 120A, a time server 120B, and other devices 120C (the data server 120A, the time server 120B, and other devices 120C are collectively referred to as “device 120” in this specification). The controller 110 may include computing devices such as the computing system 200 in Figure 2, which includes a processor communicatively coupled to memory containing instructions, which, when executed by the processor, cause the controller 110 to perform one or more operations described below. For example, the controller 110 may monitor and manage the time drift of any device 120 on the network 140. As described herein, all devices 120 on the network 140 should be synchronized with true time, such as UTC (this may be expressed as otherwise the entire network 140 should be synchronized with external time).

[0019] Each data server 120A includes a plurality of high-performance oscillators 122 configured to maintain the internal time of each data server 120A, as described herein. For example, the high-performance oscillators 122 may be specified to be accurate within a range of ±1.5 to 10 parts per million. A data server 120A includes at least two and as many as eight high-performance oscillators 122, but for the purposes of explanation, a data server 120A will be primarily described as including four high-performance oscillators 122. Each oscillator 122 may be configured to independently track the internal time of the data server 120A, and the final TOD of each data server 120A is the mean time of each of these oscillators 122 (e.g., the mean, median, and mode of each of these times). In this way, each additional high-performance oscillator 122 may provide an additional "vote" in determining what the correct time is, and the more oscillators 122 there are, the more robust the time synchronization efforts become (although each data server 120A also becomes more expensive).

[0020] In some embodiments, the data server 120A further utilizes software to stabilize its internal time. This stabilization may involve reducing the drift by about two or three orders of magnitude, better than the drift achievable by the high-performance oscillator 122 alone, so that the daily drift is in the range of 5 milliseconds to 0.01 milliseconds per day or less.

[0021] As shown in the figure, multiple time servers 120B exist on the network 140, but in other exemplary networks 140, a single time server 120B may provide service. The time server 120B is configured to receive or read time from a reference clock and then distribute this time to some or all of the devices 120 on the network 140. The time server 120B is a computing device that includes components similar to the computing system 200 in Figure 2 (for example, the time server 120B includes an interface 210, a processor 220, and memory 230 as described with respect to Figure 2 in some capabilities). The time server 120 may be understood not to necessarily include a high-performance oscillator 122.

[0022] In comparison, data server 120A is not used for the default task of receiving a reference clock signal and distributing it throughout network 140 (for example, during network 140 initialization, data server 120A is not assigned the initial task of collecting time from the reference clock and distributing the clock signal to devices 120 on network 140). Rather, data server 120A may be understood as a mainframe computing device or PC data server. Therefore, data server 120A may be understood as being used for primary computing operations for organizations utilizing network 140 (for example, so that data server 120A becomes part of the organization's central data repository).

[0023] Other devices 120C include computing devices on network 140 that are synchronized with data server 120A in network 140 but not with time server 120B, and are more susceptible to drift than data server 120A. For example, other devices 120C may include servers that do not include any high-performance oscillator 122, or may include only a single high-performance oscillator 122, or are otherwise not configured to be as resistant to drift as data server 120A. In some embodiments, other devices 120C are mainframe computing devices or PC data servers. Data server 120A and other devices 120C are both computing devices comprising components similar to computing system 200 of FIG. 2 (e.g., both including interface 210, processor 220, and memory 230 in some capacity as described with respect to FIG. 2).

[0024] In some examples, controller 110 is separate from device 120, as shown in FIG. 1, such that controller 110 manages time synchronization of network 140 as part of a computing device that is physically separate from device 120. In other embodiments (not shown), controller 110 may be integrated into one or more of devices 120 (e.g., possibly as a distributed system). For example, controller 110 may be integrated into one data server 120A, or controller 110 may be integrated into each or many of data servers 120A as separate instances, or both.

[0025] As described herein, the time server 120B transmits a reference clock signal to the data server 120A and other devices 120C. The time server 120B transmits the reference clock signal to the data server 120A and other devices 120C via the network 140 using a plurality of switches 130. The switches 130 are network switches that connect the devices 120 by techniques such as packet switching on one or more layers of the open systems interconnection (OSI) model.

[0026] Network 140 may include one or more computer communication networks. An exemplary network 140 may include the Internet, a local area network (LAN), a wide area network (WAN), a wireless network such as a wireless LAN (WLAN), etc. Network 140 may include copper transmission cables, optical transmission fibers, wireless transmitters, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. However, the connection of Network 140 between the time server 120B and the devices 120 may utilize only connections that have the capability for high-speed data transmission required for data synchronization, as described herein. For example, each of the devices 120 and the switch 130 may be connected via the LAN to each time server 120B working for these devices 120. Network adapter cards or network interfaces within each computing device / processing device (e.g., controller 110, data server 120A, time server 120B, other device 120C) may receive messages or instructions, or both, from or through network 140, or both, and transfer these messages or instructions, or both, to the respective memory or processor of each computing device / processing device for storage or execution, etc.

[0027] Although the network 140 is shown as a single entity in FIG. 1 for illustrative purposes, other illustrative networks 140 may comprise a plurality of private networks, public networks, or both, and the controller 110 may manage time via these networks as described herein. For example, in some cases, the network 140 may comprise two clustered sub-networks, in which the devices 120 are connected via respective LANs, and are further connected via a WAN or the like even when the two clustered sub-networks are geographically dispersed. Specifically, the two clustered sub-networks may be located, for example, in different buildings, different cities, or otherwise regions 100,000 kilometers apart. In this example, each of the two geographically dispersed clustered sub-networks comprises at least one time server 120B and at least one data server 120A, and the controller 110 (whether it is one controller 110 or a plurality of different instances of the controller 110) manages time synchronization and time offset of the devices 120 in the two geographically dispersed clustered sub-networks.

[0028] The controller 110 detects when the time maintained by at least one device 120 of the network 140 is more than a threshold away from the time maintained by at least one data server 120A. For example, the threshold may be 50 microseconds, 100 microseconds, or 200 microseconds, and the controller 110 may detect that the time of one of the devices 120's TOD clocks is 51 microseconds, 111 microseconds, or 201 microseconds (each) away from the time maintained by a single data server 120A, and therefore exceeds each threshold. These specific threshold numbers are provided for illustrative purposes only, but those skilled in the art will understand that such numbers depend heavily on the regulations relating to the organization of the network 140 and the specifications / capabilities of the devices 120 of the network 140 (for example, a device 120 with a tighter tolerance capability may have a smaller threshold, or an organization bound by "lower" regulations may have a larger threshold, or both). Therefore, a person skilled in the art will understand that any user-defined threshold for identifying a deviation that is greater than the time synchronization that can be maintained by device 120 and also approaches (or may be defective) an acceptable limit of deviation as defined by various regulations or standards, or both, applicable to the organization, is consistent with the present disclosure.

[0029] The controller 110 takes corrective action in response to detecting this deviation exceeding a threshold. Corrective action may include calling the best master clock algorithm, calling the STP link, changing the propagation of the clock signal from being propagated across the network 140 by the time server 120B to being propagated across the network 140 to the devices 120 by one or more data servers 120A, notifying the administrator of the time deviation (including, for example, identifying which devices 120 are out of sync and by how much), or deactivating one or more faulty oscillators 122.

[0030] In some examples, the controller 110 may compare the time of device 120 to a single data server 120A. For example, a local network 140 may include a single data server 120A, and the controller 110 may take corrective actions as described herein in response to detecting that any of the devices 120 has deviated by a threshold amount of time from the internal time of this single data server 120A. For example, the controller 110 may detect that time server 120B is malfunctioning (or has received an incorrect reference signal) as a result of detecting that the time of time server 120B is different from that of the single data server 120A.

[0031] In other examples, a single network 140 on a single LAN (for example, within a single room or a single building) may include a number of data servers 120A, and the controller 110 may compare the time of an individual device 120 to the average time of multiple data servers 120A. The controller 110 may calculate the average time of multiple data servers 120A by any number of statistical methods, such as calculating the mean, median, mode, or some other statistical method. For example, the controller 110 may compare the time of one server 120B on the LAN of network 140 to some or all of the data servers 120A on the LAN of network 140.

[0032] The controller 110 may compare the internal time of device 120 to the time of data server 120A on a set schedule, in response to the fulfillment of a condition, or both. For example, the controller 110 may compare each internal time of device 120 to the average time of data server 120A, such as once every 30 minutes, once every hour, once every 12 hours, once every day, once every few days, etc. The controller 110 may compare the time of device 120 to the time of data server 120A more frequently to capture possible deviations earlier, while the controller 110 may compare it less frequently to use fewer computing resources. In some embodiments, the controller 110 may be configured to compare the time of device 120 to the time of data server 120A in response to resource utilization falling below a threshold (for example, in response to the utilization of network 140's processing, memory, or bandwidth, or a combination thereof, falling below a certain percentage, indicating that there are surplus computing resources available for use). Additionally or alternatively, the controller 110 may be configured to compare the time of device 120 with the time of data server 120A in response to the detection of something indicating a deviation (e.g., an error, warning, or condition correlated with a deviation in one or more devices 120).

[0033] The controller 110 may autonomously perform corrective actions. More specifically, the controller 110 may perform corrective actions as described herein without human intervention. Furthermore, when the controller 110 detects that one or more of the devices 120 are out of sync, it may perform corrective actions almost immediately, such as within one millisecond or one second, of such detection. By configuring the controller 110 to autonomously and almost immediately perform corrective actions in response to detection that any of the devices 120 are out of sync, this embodiment of the controller 110 can improve the likelihood that the devices 120 will utilize the correct sync (reducing the likelihood that an organization using these devices 120 may have to pay fines or the like as a result of any of the devices 120 not utilizing the correct sync for an extended period).

[0034] As described herein, in some examples, the controller 110 detects that the time maintained by or received by the time server 120B of the network 140, or both, is more than a threshold away from the average time of the multiple data servers 120. In response to such detection, the controller 110 takes corrective action. For example, the controller 110 may notify the administrator and cause device 120 in the environment 100 to receive a clock signal that is the average time of the multiple data servers 120 (rather than a clock signal from the time server 120B that is out of sync). In this way, in response to detecting that the time of time server 120B is out of sync with the average time of the multiple data servers 120A, the controller causes device 120 to utilize the average internal time of the multiple data servers 120A.

[0035] In certain examples, the controller 110 may compare a device 120 at one geographical location to a data server 120A at a different geographical location. For example, the controller 110 may compare a data server 120A at one geographical location to a data server 120A at a second geographical location. Alternatively or additionally, the controller 110 may compare how far a device 120 at a first location is shifted relative to a data server 120A at that first location, and then compare that shift to how far a device 120 at a second location is shifted relative to a data server at that second location. In this way, aspects of the present disclosure may be configured to enable tight time synchronization of a widely distributed network, such as a Graphically Dispersed Parallel Sysplex (GDPS).

[0036] In another example, the controller 110 may compare the time of a first time server 120B at one location with the time of both a second time server 120B and a data server 120A at a second location. More specifically, the controller 110 may detect that the time server 120B at the first location has an incorrect time, and compare this incorrect time with the time of a second time server 120B at a geographically dispersed location from the first location. If the controller 110 detects that both of these time servers 120B have incorrect times that are within a threshold of each other (e.g., within 100 or 1000 microseconds from each other), the controller 110 may conclude that the problem is likely with the time source and not with the time server 120B itself. Thus, aspects of the present disclosure may be configured to determine, for example, that a malicious third party appears to be tampering with a time source, such as GPS signals (e.g., by GPS spoofing). In response to such a decision, the controller 110 may cause the devices 120 at both locations to use the time of the data server 120A instead of the time server 120B, until the administrator can verify the status of at least both locations.

[0037] The controller 110 is configured to detect whether one data server 120A is out of sync with other data servers 120A. If the controller 110 detects that one data server 120A is out of sync with other data servers 120A, the controller 110 may analyze the performance of each oscillator 122 of the out-of-sync data server 120A. Often, the controller 110 identifies at least one oscillator 122 of the out-of-sync data server 120A that is faulty and causing the data server 120A to be out of sync. In response to detecting one or more faulty oscillators 122, the controller 110 may take autonomous action to deactivate these faulty oscillators 122 in the out-of-sync data server 120A. The controller 110 may further notify the administrator, request replacement oscillators 122 for the (already) out-of-sync data server 120A, or both.

[0038] Additionally or alternatively, the controller 110 may deactivate data server 120A in response to detecting that data server 120A is out of sync (for example, in response to a faulty oscillator 122). In another example, the controller 110 may treat this already out-of-sync data server 120A as a new, other device 120C rather than data server 120A in response to detecting that this data server 120A is out of sync (or in response to deactivating one oscillator 122, or both). In other words, the controller 110 does not need to compare other devices 120C to this already out-of-sync data server 120A until this already out-of-sync data server 120A is fully operational and repaired, so as to confirm that this already out-of-sync data server 120A performs in line with other data servers 120A (for example, four properly functioning oscillators 122, each specified at ±2 ppm).

[0039] The controller 110 may be configured to detect when one other device 120C is out of sync. In response to the controller 110 detecting that one other device 120C is out of sync, the controller 110 may check whether any further other devices 120C are out of sync. If further other devices 120C are out of sync, the controller 110 may compare the syncs among these other devices 120C to see if they are similar. If the controller 110 determines that two or more other devices 120C are out of sync by similar amounts, the controller 110 may identify the commonality of the paths through which these other devices 120C received the clock signal from the time server 120B. For example, the controller 110 may determine that in a network 140 containing 40 other devices 120C, eight other devices 120C are experiencing substantially identical syncs, and furthermore, that all eight of these other devices 120C share a common switch 130. In response to this determination that the common switch 130 is involved with all the other out-of-sync devices 120C, the controller 110 may take corrective action, such as changing the path of the clock signal to the other out-of-sync devices 120C (for example, by taking a new path that bypasses the problematic switch 130). The controller 110 may also notify the administrator of the switch 130 that appears to be failing.

[0040] In another example, when the controller 110 analyzes all other devices 120C (in response to detecting that one other device 120C is out of sync), it may determine that, apart from this one other device 120C, no other devices 120C are out of sync (or similarly out of sync). In response to such a determination, the controller 110 may take one or more corrective actions, including reporting the problem to the administrator, calling the best master clock algorithm, or switching the out-of-sync other device 120C to receive time from an adjacent device 120.

[0041] As previously stated, the controller 110 may be part of a computing device that includes a processor configured to execute instructions stored in memory and perform the techniques described herein. For example, Figure 2 is a conceptual box diagram of such a computing system 200 of the controller 110. Although the controller 110 is shown as a single entity (e.g., in a single enclosure) for illustrative purposes, in other examples the controller 110 may include two or more separate physical systems (e.g., in two or more separate enclosures). The controller 110 may include an interface 210, a processor 220, and memory 230. The controller 110 may include any number or quantity of interfaces 210, processors 220, or memory 230, or a combination thereof.

[0042] The controller 110 may include components that enable the controller 110 to communicate with devices outside of the controller 110 (for example, sending data to a device and receiving and utilizing data sent by a device). For example, the controller 110 may include an interface 210 configured to enable the controller 110 and components within the controller 110 (e.g., a processor 220) to communicate with entities outside of the controller 110. In detail, the interface 210 may be configured to enable components of the controller 110 to exchange information with devices 120, switches 130, etc. The interface 210 may include one or more network interface cards, such as Ethernet® cards or any other type of interface device, or both, that can send and receive information. Depending on the specific needs, various numbers of interfaces may be used to perform the functions described.

[0043] As described herein, the controller 110 may be configured to manage time synchronization within a computing network. The controller 110 may utilize a processor 220 to manage time in this manner. The processor 220 may include, for example, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent individual or integrated logic circuits, or a combination thereof. Two or more of the processors 220 may work together to identify whether any of the devices 120 are out of sync and take corrective action accordingly.

[0044] The processor 220 may manage the time of the device 120 in the environment 100 according to instructions 232 stored in the memory 230 of the controller 110. The memory 230 may include a computer-readable storage medium or a computer-readable storage device. In some examples, the memory 230 includes one or more short-term memory or long-term memory. The memory 230 may include, for example, random access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), magnetic hard disks, optical disks, floppy disks, flash memory, electrically programmable memory (EPROM), and electrically erasable and programmable memory (EEPROM).

[0045] In addition to instruction 232, in some examples, collected or predetermined data or techniques, such as those used by the processor 220 to manage time lags and synchronization, as described herein, are stored in memory 230. For example, memory may include time data 234, which may include various thresholds and schedules, and the controller 110 monitors the internal time of device 120 with these thresholds and schedules. Memory 230 may also include data server data 236, time server data 238, and other device data 240. Data server data 236 may include past or present time data of data server 120A or both, while time server data 238 may include past or present time data of time server 120B or both, and other device data 240 may include past or present time data of other device 120C.

[0046] Memory 230 may further include machine learning techniques 242, and the controller 110 may use machine learning techniques 242 to improve the process of managing time synchronization and drift over time as described herein. Machine learning techniques 242 may include algorithms or models generated by performing supervised, unsupervised, or semi-supervised training on a dataset, and then applying the generated algorithms or models to monitor time synchronization or drift as described herein. For example, the controller 110 may use machine learning techniques 242 to determine that a particular drift threshold for a particular type of device 120 is more likely to indicate drift or to result in problematic drift before the next scheduled scan, or both. In another example, the controller 110 may use machine learning techniques 242 to determine that a particular type of improvement action is better or worse in reducing drift over time. The controller 110 may refine rules over time based on whether their ability to reduce time drift improves or decreases based on rule updates. For example, the controller 110 may track whether the amount of deviation requiring corrective action is increasing or decreasing, and may change or stabilize future actions accordingly.

[0047] Machine learning techniques242 may include, but are not limited to, decision tree learning, correlation rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity / metric training, sparse dictionary learning, genetic algorithms, rule-based learning, or other machine learning techniques, or combinations thereof.

[0048] For example, machine learning techniques 242 include K-nearest neighbor (KNN), learning vector quantization (LVQ), self-organizing map (SOM), logistic regression, ordinary least squares regression (OLSR), linear regression, stepwise regression, multivariate adaptive regression spline (MARS), ridge regression, lasso regression (LASSO: least absolute shrinkage and selection operator), elastic networks, least-angle regression (LARS), stochastic classifiers, naive Bayesian classifiers, binary classifiers, linear classifiers, hierarchical classifiers, canonical correlation analysis (CCA), factor analysis, independent component analysis (ICA), linear discriminant analysis (LDA), multidimensional scaling (MDS), and non-negative metric factorization (NMF). Metric factorization, partial least squares regression (PLSR), principal component analysis (PCA), principal component regression (PCR), summon mapping, t-distributed stochastic neighbor embedding (t-SNE), bootstrap aggregation, ensemble averaging, gradient boosted decision tree (GBRT), gradient boosting machine (GBM)Machine learning, inductive bias algorithms, Q-learning, SARSA (state-action-reward-state-action), temporal difference (TD) learning, a priori algorithms, equivalence class transformation (ECLAT) algorithms, Gaussian process regression, gene expression programming, group method of data handling (GMDH), inductive logic programming, example-based learning, logistic model trees, information fuzzy networks (IFN), hidden Markov models, Gaussian naive Bayes, multinomial naive Bayes, averaged one-dependence estimators (AODE), Bayesian networks (BN), classification and regression trees (CART), chi-squared automatic interaction detection (CHAID). One or more of the following exemplary techniques may be used: detection, expectation maximization algorithms, feedforward neural networks, logic learning machines, self-organizing maps, single-linkage clustering, fuzzy clustering, hierarchical clustering, Boltzmann machines, convolutional neural networks, recurrent neural networks, hierarchical temporal memory (HTM), or other machine learning algorithms, or combinations thereof.

[0049] The controller 110 may use these components to manage time synchronization and drift as described herein. In some embodiments, the controller 110 manages the time synchronization of the device 120 according to the flowchart 300 shown in Figure 3. While the flowchart 300 in Figure 3 is illustrated with respect to Figure 1 for illustrative purposes, it should be understood that in other examples, other systems and messages may be used to perform the flowchart 300 in Figure 3. Furthermore, in some embodiments, the controller 110 may perform the flowchart 300 in Figure 3 in a different manner, or the controller 110 may perform the same manner, using more or fewer steps, in a different order, etc.

[0050] Flowchart 300 begins with the controller 110 monitoring the internal time (e.g., TOD) of the devices 120 on the network 140 (302). The controller 110 may determine whether the data servers 120A are matched with each other so that all data servers 120A are within a strict tolerance / threshold from each other (304). If the controller 110 determines that any of the data servers 120A are outside the threshold (branch to "no" from 304), the controller 110 may attempt to identify whether any of the oscillators 122 are faulty (306). The controller 110 may then take corrective action, such as deactivating the faulty oscillator 122, deactivating the faulty data server 120A, requesting a replacement oscillator 122, notifying an administrator, treating the faulty data server 120A as another device 120C, or doing the same (308).

[0051] If controller 110 determines that all data servers 120A are in agreement (branch to "yes" from 304), controller 110 determines whether other devices 120C are in agreement within a threshold (310). If other devices 120C are in agreement (branch to "yes" from 310), controller 110 continues to monitor the time of device 120 (for example, at the next scheduled time). If other devices 120C are not in agreement (branch to "no" from 310), controller 110 may compare the locally occurring deviation with the deviation occurring in geographically distributed devices 120 (for example, other devices 120 on a shared WAN) (312).

[0052] The controller 110 may verify whether similar shifts are occurring in the geographically distributed devices 120 (314). For example, the controller 110 may determine that all other devices 120C in a first geographic location serviced by the first time server 120B are experiencing a first magnitude shift, and all other devices 120C in a second geographic location serviced by the second time server 120B are experiencing a second magnitude shift, and that the first and second shifts are substantially similar (yes branch from 314).

[0053] A person skilled in the art will understand that two time discrepancies are understood to be substantially similar when they are close enough that it is less likely to be accidental, and rather more likely that both discrepancies result from both receiving the same (potentially malicious) incorrect reference clock. The exact value at which such a decision is worthwhile may vary depending on circumstances such as the precision of the components or the precision of a possible attack or both, but an exemplary threshold at which they are identified as substantially similar may be within 0.01 seconds of each other. In response to this decision, controller 110 may take remedial actions for geographically distributed discrepancies (316). Remedial actions may include notifying the administrator that the time source appears to be spoofed, or changing the time source for a particular network 140 to be the (n-mean) time of data server 120A rather than the collected reference time of time server 120B. After taking these remedial actions, controller 110 may return to monitoring device 120 (302).

[0054] If controller 110 determines that no similar drifts are occurring in geographically dispersed locations (branching "no" from 314), controller 110 may track the clock signal and identify errors in local components (318). Components may include switches 130, other devices 120C, or time servers 120B, or a combination thereof. For example, if a number of other devices 120C are all drifting and all share each switch 130, controller 110 may identify that each switch 130 is faulty and take corrective action by routing the clock signal to these other devices 120C without passing through each of these switches 130 (320). If controller 110 determines that a number of other devices 120C that are serviced by a single time server 120B are out of sync, controller 110 may take corrective action such as causing these other devices 120C to receive a clock signal from data server 120A instead (or, where applicable, causing these other devices 120C to be serviced by another time server 120B on network 140) as described herein (320). Alternatively, if controller 110 determines that a single other device 120C is out of sync, controller 110 may take corrective action such as performing STP, causing this single other device 120C to receive time from a nearby device 120, or both (320).

[0055] The descriptions of the various embodiments of this disclosure are presented for illustrative purposes only and are not intended to be exhaustive, nor are they limited to the embodiments disclosed. Many changes and modifications that do not deviate from the scope and spirit of the embodiments described will be apparent to those skilled in the art. The terms used herein have been chosen to describe the principles of the embodiments, their practical applications, or technical improvements beyond the technology available on the market, or to enable those else skilled in the art to understand the embodiments disclosed herein.

[0056] The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of technical detail of integration. The computer program product may include one or more computer-readable storage media containing computer-readable program instructions for causing a processor to perform an aspect of the present invention.

[0057] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of further specific examples of computer-readable storage media includes portable floppy disks, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random-access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved structures on which instructions are recorded, and any suitable combination thereof. When used herein, computer-readable storage media should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmitting media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0058] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing device / processing device, or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). This network may include copper transmission cables, optical transmission fibers, wireless transmitters, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing device / processing device receives computer-readable program instructions from the network and transfers those computer-readable program instructions for storage on a computer-readable storage medium within each computing device / processing device.

[0059] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed as a whole on the user's computer, partially as a standalone software package on the user's computer, partially on the user's computer and a remote computer, respectively, or as a whole on a remote computer or a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be customized by executing computer-readable program instructions using state information of computer-readable program instructions.

[0060] Aspects of the present invention will be described herein by reference to flowcharts or block diagrams, or both, of methods, apparatuses (systems), and computer program products, according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and any combination of blocks contained in a flowchart or block diagram, or both, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions may be provided to a general-purpose computer, a dedicated computer, or a processor of another programmable data processing device to create a machine, so that instructions executed via the processor of a computer or other programmable data processing device can create means to perform functions / operations specified in one or more blocks of a flowchart or block diagram, or both. These computer-readable program instructions may be stored on a computer-readable storage medium containing instructions that include instructions to perform modes of functions / operations specified in one or more blocks of a flowchart or block diagram, or both, and can instruct a computer, a programmable data processing device, or other device, or a combination thereof, to function in a particular manner.

[0062] Computer-readable program instructions may be read into a computer, another programmable data processing device, or other device so that instructions executed on a computer, another programmable device, or other device perform functions / operations specified in one or more blocks of a flowchart or block diagram, or both, thereby causing a series of operable steps to be executed on a computer, another programmable device, or other device that generates a computer implementation process.

[0063] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a defined logical function. In some alternative implementations, the functions shown in the blocks may occur in an order different from the order shown in the figures. For example, two consecutively shown blocks may actually be implemented as a single step, executed simultaneously, executed substantially simultaneously in a way that partially or completely overlaps in time, or possibly executed in reverse order, depending on the functions they contain. It should also be noted that each block in the block diagram or flowchart diagram, or both, and any combination of blocks contained in the block diagram or flowchart diagram, or both, may be implemented by a dedicated hardware-based system that performs a defined function or operation, or a combination of dedicated hardware and computer instructions.

[0064] The following are exemplary clauses relating to aspects of this disclosure.

[0065] The first paragraph relates to a computer implementation method in which a data server of a plurality of devices on a network compares the internal time of the data server with the respective time of each of the plurality of devices, the data server maintaining and comparing its internal time using a plurality of high-performance oscillators, the data server analyzing the compared times detecting that the time maintained by another device on the network is deviating by a threshold, and the data server taking corrective action in response to detecting that the time maintained by another device is deviating by a threshold.

[0066] The second paragraph relates to the computer implementation method of the first paragraph, wherein the multiple high-performance oscillators include at least four oscillators, each specified to be accurate to at least ±2 parts per million, and the data server is configured to use the software together with the multiple high-performance oscillators to maintain the internal time not to deviate by more than 2 milliseconds per day.

[0067] Paragraph 3 relates to either Paragraph 1 or Paragraph 2's computer implementation method, and improvement actions are performed autonomously.

[0068] Paragraph 4 relates to any of the computer implementation methods described in Paragraphs 1 through 3, wherein the data server is one of several data servers, each maintaining its own internal time using multiple high-performance oscillators.

[0069] Paragraph 5 relates to any of the computer implementation methods described in Paragraphs 1 through 4, and the improvement action includes sending a notification to the administrator.

[0070] Paragraph 6 relates to one of the computer implementation methods described in Paragraphs 1 through 5, where the network synchronizes the internal time of multiple internal devices using a high-precision time protocol.

[0071] Section 7 relates to the computer implementation method of Section 6, and the improvement action includes calling the best master clock algorithm.

[0072] Paragraph 8 relates to one of the computer implementation methods described in Paragraphs 1 through 6, where another device is a network time server.

[0073] Paragraph 9 relates to the computer implementation method of Paragraph 8, and includes detecting that the time of a time server is deviating by more than a threshold by comparing the time of the time server to the average internal time of all of the multiple data servers, and the corrective action includes causing the network to use the average internal time of all of the multiple data servers instead of the time of the time server in response to detecting that the time of the time server is deviating by more than a threshold from the average internal time.

[0074] Paragraph 10 relates to any of the computer implementation methods described in Paragraphs 1 through 6, and includes detecting when the time of another device is off by more than a threshold by comparing the time of the other device to the average internal time of all of multiple data servers, the method further includes identifying that a network switch is the cause of the other device being off by more than a threshold by tracking the clock signal to the switch.

[0075] Paragraph 11 relates to any of the computer implementation methods described in paragraphs 1 through 6, wherein another device is another data server on the network, and the corrective action includes autonomously decommissioning the data server.

[0076] Paragraph 12 relates to any of the computer implementation methods described in Paragraphs 1 through 6, wherein the network is geographically distributed and includes two clustered subnetworks, and the data server is one of several data servers in the first subnetwork of the two clustered subnetworks, and each of the several data servers in the first subnetwork maintains its own internal time using several high-performance oscillators, and another device is the time server of the first clustered subnetwork, and the detection of a time deviation of the time server of the first clustered subnetwork exceeding a threshold is performed by the time server of the first clustered subnetwork and the multiple data servers... This method includes comparing the internal time of a time server in a second subnet of two clustered subnets with the average internal time of multiple data servers, further including detecting that the time server in the second clustered subnet has drifted in a substantially similar manner to that of the time server in the first clustered subnet, and detecting, as a result of detecting that the time servers have drifted in a substantially similar manner, that the clock sources of the time servers in both the first and second clustered subnets have been compromised.

[0077] Paragraph 13 relates to any of the computer implementation methods described in Paragraphs 1 through 6, and includes detecting that the time of another device is off by more than a threshold by comparing the time of the other device to the average internal time of all of a plurality of data servers, the other device including another data server among the plurality of data servers, and the corrective action includes deactivating the high-performance oscillator of the other data server in response to identifying that the high-performance oscillator of the other data server is faulty.

Claims

1. The comparison is made between the internal time of a data server on a network and the respective times of each of several devices on the network, wherein the data server maintains its internal time using several high-performance oscillators, each high-performance oscillator individually tracks the internal time of the data server, and the internal time of the data server is the average time obtained from the several high-performance oscillators. By analyzing the compared times, it is possible to detect that the time maintained by another device on the network is deviating by more than a threshold, In response to detecting that the time maintained by the other device has deviated by the threshold, an improvement action is taken. Computer implementation methods, including those mentioned above.

2. The computer implementation method according to claim 1, wherein the other device is the network time server.

3. The aforementioned data server is one of several data servers, each using multiple high-performance oscillators to maintain its own internal time. The detection that the time of the time server is deviating by more than the threshold includes comparing the time of the time server with the average internal time of all of the multiple data servers. The computer implementation method according to claim 2, wherein the improvement action includes, in response to detecting that the time of the time server deviates from the average internal time by a threshold, causing the network to use the average internal time of all of the multiple data servers instead of the time of the time server.

4. The computer implementation method according to claim 3, wherein the improvement action is performed autonomously.

5. The aforementioned data server is one of several data servers, each using multiple high-performance oscillators to maintain its own internal time. The detection that the time of the other device is deviating by more than the threshold includes comparing the time of the other device to the average internal time of all of the multiple data servers, and the method is The computer implementation method according to claim 1, further comprising tracing an incorrect clock signal to a switch, thereby identifying that the switch in the network is causing the other device to deviate beyond the threshold.

6. The aforementioned improvement action is Sending a notification to the administrator, Calling the best master clock algorithm and The computer implementation method according to claim 5, including the method described in claim 5.

7. The computer implementation method according to claim 1, wherein the network synchronizes the internal time of the plurality of devices using a high-precision time protocol.

8. The computer implementation method according to claim 7, wherein the improvement action includes calling the best master clock algorithm.

9. The other device is another data server on the network, The computer implementation method according to claim 1, wherein the improvement action includes autonomously deactivating the data server.

10. The aforementioned plurality of high-performance oscillators include at least four oscillators, each specified to be accurate to at least ±2 parts per million, The computer implementation method according to claim 1, wherein the data server is configured to use software together with the plurality of high-performance oscillators to maintain the internal time so that it does not deviate by more than 2 milliseconds per day.

11. The aforementioned data server is one of several data servers, each using multiple high-performance oscillators to maintain its own internal time. The detection that the time of the other device is deviating by more than the threshold includes comparing the time of the other device with the average internal time of all the multiple data servers, The aforementioned other device includes another data server among the plurality of data servers, The computer implementation method according to claim 1, wherein the improvement action includes deactivating the high-performance oscillator in response to identifying that the high-performance oscillator of the other data server is malfunctioning.

12. A computer implementation method, The comparison of the internal time of a data server on a network with the respective times of each of several devices on the network, wherein the data server maintains the internal time using several high-performance oscillators, and the comparison is performed accordingly. By analyzing the compared times, it is possible to detect that the time maintained by another device on the network is deviating by more than a threshold, In response to detecting that the time maintained by the other device has deviated by the threshold, an improvement action is taken. Includes, The aforementioned network is geographically distributed and includes two clustered subnetworks. The data server is one of a plurality of data servers in the first subnetwork of the two clustered subnetworks, and each of the plurality of data servers in the first subnetwork maintains its internal time using each of the plurality of high-performance oscillators. The aforementioned other device is the time server of the first subnetwork, The detection that the time of the time server in the first subnetwork is off by more than the threshold includes comparing the time of the time server in the first subnetwork with the average internal time of all of the plurality of data servers, The aforementioned computer implementation method The internal time of the time server of the second subnetwork of the two clustered subnetworks is compared with the average internal time of all the data servers. The time server of the second subnetwork detects that it has shifted in a manner substantially similar to the manner in which the time server of the first subnetwork shifted, As a result of detecting that the time server is drifting in substantially similar ways, it is detected that the clock source of the time server in both the first and second subnetworks has been compromised. Computer implementation methods, including further details.

13. A data server on a network of multiple devices, wherein the data server is Multiple high-performance oscillators, Processor and The system includes a memory that communicates with the processor, the memory contains instructions, and when an instruction is executed by the processor, the system communicates to the processor, The comparison involves comparing the internal time of the data server with the time of each of the multiple devices on the network, wherein the data server maintains the internal time using the multiple high-performance oscillators, each high-performance oscillator individually tracks the internal time of the data server, and the internal time of the data server is the average time obtained from the multiple high-performance oscillators. By analyzing the compared times, it is possible to detect that the time maintained by another device on the network is deviating by more than a threshold, In response to detecting that the time maintained by the other device has deviated by the threshold, an improvement action is taken. A data server that executes the data.

14. The data server according to claim 13, wherein the other device is the time server of the network.

15. The aforementioned data server is one of several data servers on the network, each maintaining its own internal time using multiple high-performance oscillators. The detection that the time of the time server is deviating by more than the threshold includes comparing the time of the time server with the average internal time of all of the multiple data servers. The data server according to claim 14, wherein the improvement action includes, in response to detecting that the time of the time server deviates from the average internal time by more than the threshold, causing the network to use the average internal time of all of the plurality of data servers instead of the time of the time server.

16. The aforementioned data server is one of several data servers on the network, each maintaining its own internal time using multiple high-performance oscillators. The detection that the time of the other device is deviating by more than the threshold includes comparing the time of the other device to the average internal time of all the multiple data servers, and the memory includes an additional instruction, and if the additional instruction is executed by the processor, the processor is instructed to The data server according to claim 13, which causes the switch in the network to identify that the other device is causing the clock signal to drift beyond the threshold by tracing the erroneous clock signal to the switch.

17. The data server according to claim 13, wherein the improvement action includes calling a best master clock algorithm.

18. The aforementioned data server is one of several data servers on the network, each maintaining its own internal time using multiple high-performance oscillators. The detection that the time of the other device is deviating by more than the threshold includes comparing the time of the other device with the average internal time of all the multiple data servers, The aforementioned other device includes another data server among the plurality of data servers, The data server according to claim 13, wherein the improvement action includes deactivating the high-performance oscillator in response to identifying that the high-performance oscillator of the other data server is malfunctioning.

19. The aforementioned plurality of high-performance oscillators include at least four oscillators, each specified to be accurate to at least ±2 parts per million, The data server according to claim 13, wherein the memory includes a software application configured to improve the accuracy of the internal time of the data server and maintain the internal time so that it does not deviate by more than 2 milliseconds per day, together with the plurality of high-performance oscillators.

20. A computer program, wherein the computer, The comparison is made between the internal time of a data server on a network and the respective times of each of several devices on the network, wherein the data server maintains its internal time using several high-performance oscillators, each high-performance oscillator individually tracks the internal time of the data server, and the internal time of the data server is the average time obtained from the several high-performance oscillators. By analyzing the compared times, it is possible to detect that the time maintained by another device on the network is deviating by more than a threshold, In response to detecting that the time maintained by the other device has deviated by the threshold, an improvement action is taken. A computer program designed to execute something.

21. This involves using multiple high-performance oscillators to maintain the internal time of multiple data servers as part of a network utilizing a high-precision time protocol (PTP), and comparing the average internal time of these servers with that of another device in the network. By analyzing the compared times, it is possible to detect that the time maintained by another device on the network deviates by more than a threshold from the average internal time of all the data servers. In response to identifying that the time maintained by the other device on the network is deviating by a threshold, perform corrective actions. Computer implementation methods, including those mentioned above.

22. The computer implementation method according to claim 21, wherein the other device is the network time server.

23. The computer implementation method according to claim 22, wherein the improvement action includes, in response to detecting that the other device is the time server, causing the network to utilize the average internal time of all of the plurality of data servers rather than the time of the time server.

24. The computer implementation method according to claim 21, further comprising tracking the clock signal to a switch to identify that the switch in the network is the cause of the other device being out of sync by a threshold.

25. The aforementioned other device includes another data server among the plurality of data servers, The computer implementation method according to claim 21, wherein the improvement action includes deactivating the high-performance oscillator in response to identifying that the high-performance oscillator of the other data server is malfunctioning.

Images