Time-based processing of messages arriving from multiple gateways using gateway-based time perimeters

Abstract

Disclosed herein are systems and methods for performing time ordering manipulation of messages sent from multiple sources to a domain of networked machines. In one embodiment, a system performs a software-based synchronization process for synchronizing each of multiple gateways to a reference clock, each gateway being on the perimeter of a region of networked machines and connected to receive messages from multiple sources. The system receives messages that enter the region through one gateway, which timestamps the message. The system then performs a time ordering operation on the received message based on the absolute time of entry into the region, which is determined based on the reference clock and the timestamp from the gateway.

Description

[Technical field]

[0001] This disclosure generally relates to ensuring that messages received at multiple gateways are processed in a time-based order, even though the sources of the messages may not be synchronized. [Background technology]

[0002] Limitations on clock synchronization accuracy between machines such as gateways impose practical limitations in many applications. For example, in finance and electronic trading, clock synchronization is important to determine trading order, as trading platforms must match bids and offers in the order in which they are received at the gateway. If the clocks of the machines used to send or route bids and offers are not synchronized, the order of bids and offers may not match, leading to a lack of fairness. Financial exchanges attempt to mitigate the lack of fairness by adjusting the placement of gateways that receive messages and are sequenced for processing at equidistant locations relative to the sources of bids and offers, such that the messages travel approximately the same distances and therefore have approximately the same transit time. However, despite this calibration, messages are still delivered out of order due to jitter, and practical constraints are imposed by the need to physically place gateways at such calibrated distances to maintain fairness. Similar problems arise when messages are sent from a central location, such as a trading platform, where sending a message to one party before the other, due to the technical limitations mentioned above, gives an unfair advantage to the recipient of the earlier message.

[0003] Similar problems arise in other networked computer systems, such as distributed databases, distributed ledgers (e.g., blockchains), distributed transaction tracking systems, distributed snapshots of computations or networks, 5G mobile network communications, etc. In these systems, clock synchronization limitations introduce jitter, resulting in biased or non-optimal processing of communications. [Brief description of the drawings]

[0004] Embodiments of the present disclosure have other advantages and features that will become more readily apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawing examples. [Figure 1] Figure 1 is a diagram representing a pipeline for message processing. [Diagram 2] FIG. 2 is a graph of a network of mechanisms according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flow diagram for performing time-sequenced manipulation of messages according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a flow diagram for performing time-triggered distribution of messages according to an embodiment of the present disclosure. [Figure 5A] FIG. 5A is a graph of a computer system using two or more domains of networked machines according to an embodiment of the present disclosure. [Figure 5B] FIG. 5B is a graph of a computer system using two or more domains of networked machines according to an embodiment of the present disclosure. [Figure 6] FIG. 6 is a graph of a network including a coordinator for performing software-based synchronization of gateways according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a flow chart for correcting clock offset and / or frequency according to one embodiment of the present disclosure. [Figure 8]FIG. 8 is a data structure diagram illustrating a probe record and a method for identifying an encoded probe from the probe record according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a graph of a system illustrating loop error identification and correction according to one embodiment of the present disclosure. [Figure 10] FIG. 10 is a block diagram of a model of a control system for correcting a machine's local clock frequency according to one embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram of an adaptive filter bank according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The figures and the following description relate to preferred embodiments by way of example only. It should be noted from the following description that alternative embodiments of the structures and methods disclosed herein are readily discernible as viable alternatives that may be employed without departing from the principles of what is claimed.

[0006] Disclosed herein are systems and methods that ensure that messages arriving from sources passing through different gateways are processed in an order determined based on their timestamps at the different gateways as determined relative to a reference clock, even when the sources are not synchronized to the reference clock. The systems and methods may be software-based, thus avoiding the need to physically calibrate the gateways to achieve this result, which is expensive, impractical, and inaccurate. In this way, fairness is achieved since messages are processed in order without the need to synchronize the source of the message to a reference clock. To achieve such fairness, the gateways are synchronized to within a known time deviation of the reference clock, which may be within the order of nanoseconds of the reference clock.

[0007] FIG. 1 is a representation of a pipeline for processing messages. Shown in FIG. 1 are various machines, each performing a different function. In FIG. 1, the letter "N" is a node. The term "node" as used herein refers to any physical or virtual device where a message is generated, processed, and / or relayed, or where a message is destined. In some embodiments, a node may be a machine. The term "machine" refers to any device that maintains a clock or generates a timestamp, such as a physical or virtual machine, a server, a server blade, etc. Each machine includes a local clock (e.g., implemented in the computer processing unit (CPU) of the machine or implemented in a device operatively coupled to the machine, such as the network interface card (NIC) of the machine). In FIG. 1, the letter "S" represents a source node. A source node is a node from which a message originates. A source node may be under the control of a user, enterprise, or other such entity that is different from the entity that controls the other machines in FIG. 1. The term message refers to any communication (e.g., a single packet, a collection of packets, and / or a single data unit or a collection of data units) sent between nodes. It should be noted that, with reference to FIG. 1, the letter "G" represents a gateway and the letter "0" represents a time-ordered operation. The term "gateway", as well as the term "time-ordered operation", are defined in more detail with reference to FIG. 2 below. The operator node 104 may be controlled, for example, by a financial exchange or another central entity that does not control the source node. The pipeline 100 illustrates an exemplary flow of messages from a source node 102, through various machines 101, to a gateway 103, through various other nodes 101, and finally to an operator node 104 that performs a time-ordered operation on the message. Any number of machines, including no machines at all, may be disposed between the source node 102, the gateway 103, and the operator node 104. Additionally, although not shown, the dashed lines of the source node 102 preceding and the subsequent node 104 in FIG. 1 indicate that additional nodes may be disposed.Although the lines are depicted as a single communication path within the pipeline 100 before the source 102 sends a message and before the time-sequenced operations are performed, this is for illustrative convenience and multiple paths may exist between the blocks of FIG.

[0008] FIG. 2 is a graph of a network of machines according to one embodiment of the disclosure. The network 200 includes a source node 210 and a gateway 220, as well as a coordinator 230 and an operator node 240. As shown, the operator node 240 includes a reference clock 245. The reference clock 245 may be in any machine and need not be part of the operator node 240. The network 200 may include additional nodes, as described above with reference to FIG. 1, that are not depicted in FIG. 2. The source node 210 sends messages to the operator node 240 through the gateway 220 for time-ordered processing. The gateway 220 is synchronized to the reference clock 245 of the operator 240, such that timestamps applied by the gateway 220 to messages from the source node 210 are absolute with respect to the reference clock 245. In one embodiment, the reference clock 245 is in a machine other than the operator node 240. The gateway 220 applies timestamps to messages received from the source node 210 (e.g., to indicate the time each message was received by the gateway 220, to indicate the time the message was processed by the gateway 220, to indicate the time of transmission of the message from the gateway 220, etc.) and then sends the messages to the operator node 240. The coordinator 230 coordinates synchronization between the gateway 220 and the operator node 240, the synchronization process being described in further detail with reference to Figures 6 to 11 below. The operator node 240 performs time-ordered operations on the received messages. For example, it may first sort the messages based on the gateway's timestamps and then perform operations on the time-ordered messages. Time-ordered operations are defined and described in further detail with reference to Figures 3 to 4 below.

[0009] The gateways 220 form a time perimeter 250. In one embodiment, the time perimeter 250 is a logical boundary, and the gateways along the time perimeter 250 are synchronized to the reference clock 245. In one embodiment (e.g., in a financial exchange scenario), machines along or within the time perimeter 250 (i.e., machines within the domain 205) may be synchronized to the reference clock 245, and machines outside the domain 205 may or may not be synchronized to the reference clock 245. In one embodiment, the source node 210 itself may be synchronized to the reference clock 245 if instructed to do so by the coordinator 230.

[0010] FIG. 3 is a flow diagram for performing a time ordering operation on messages, according to one embodiment of the disclosure. FIG. 3 includes four columns: source activity, gateway activity, coordinator activity, and operator activity. In this example, the boxes displayed in the columns are executed by the corresponding entities. The flow 300 begins with a sub-process of establishing software-based synchronization 310. Briefly, the software-based synchronization 310 begins with a coordinator (e.g., coordinator 230) initiating a synchronization operation start 312a on a set of machines including one or more gateways (e.g., gateway 220) and a reference machine (e.g., operator node 240 including reference clock 245). The nodes (including the gateways and reference machine) exchange probes 314. As illustrated, the coordinator estimates 316 the absolute offset of each machine with respect to the reference machine's reference clock based on the exchanged probes, however, in alternative embodiments, the gateways themselves may estimate the absolute offset. Software-based synchronization, including probe replacement and absolute offset estimation, is described in further detail below with respect to Figures 6 through 11. At the end of software-based synchronization 310, a gateway (e.g., gateway 320) is synchronized to the reference clock.

[0011] The data flow 300 proceeds to a subprocess that establishes a subprocess for receiving 320 messages through a time synchronization gateway. A source node (e.g., source node 210) sends 321 a message to a gateway on a time perimeter. The gateway receives 322 the message, e.g., upon receipt, and timestams 324 the message. Thus, the gateway timestamp establishes the time at which each message crosses the time perimeter. Because gateways of a time perimeter are synchronized to a reference clock, the timestamp allows a coordinator and / or operator to reliably determine the time at which a message arrived at a time perimeter according to a given reference clock, even if the message arrived at a different gateway of the time perimeter. The timestamped messages are sent to an operator (e.g., operator node 240) for time ordering processing.

[0012] After the timestamped messages are sent to the operator, the data flow 300 continues with the sub-process of performing a time ordering operation 330 on the messages. The term time ordering operation refers to an operation performed on the messages, which are processed in an order based on the gateway timestamp. In one approach, since messages may arrive at the operator out of order, the operator buffers the timestamped messages as they are received 332. In one embodiment, the messages are buffered until the buffer is full, at which point the operator time orders the timestamped messages 334. Alternatively, the messages are buffered until a predetermined time has elapsed since the first message contained in the buffer was received, or until the end of a predetermined period, at which point the operator time orders the timestamped messages 334. Buffering the messages ensures that all messages received within a given period are time ordered and not missed in the time ordering (e.g., reordering) process, even if subject to jitter. Time ordering the messages 334 includes ordering the messages in the order in which they were received within the domain 205. Messages may be time ordered by physically reordering the messages in a buffer, by appending sequence numbers within the messages or in their headers, by creating a data structure with an ordered list of message identifiers that correspond to a time order, or by using any other means of establishing an order of messages.

[0013] The time ordering of messages may be any order based on the gateway timestamp. For example, messages may be processed in chronological order of the absolute time entry of the messages at the perimeter (e.g., first in first out or last in first out with respect to the entry time at the perimeter). As another example, messages may be ordered by parameters such as the importance of the message, a factor that weights the absolute time entry, such as tier of service, quality of service, etc., in addition to the absolute time entry of the message.

[0014] The time ordering may be based on the gateway through which the message entered the region. As an example, some gateways may be assigned a higher importance than other gateways, and the operator node 240 may weight the absolute time entries in such a way as to prioritize messages at the higher importance gateways earlier than other messages based on the importance of prioritizing messages from these higher importance gateways, even if these latter messages have absolute time entries later than the absolute time entries of the other messages.

[0015] In one embodiment, messages received at a given gateway may be time-stamped by the source from which they originated, as well as jitter or other network conditions. Conditions may cause gateway timestamps to be inconsistent with the order in which the source time-stamped the messages. Such jitter may cause out-of-order processing in which the sources are synchronized with one another. To eliminate problems caused by such jitter or other network issues, the gateway itself may have a buffer that periodically receives messages from the sources, orders them based on their source timestamps, and then includes signaling so that they are processed in the order in which they departed from their source, regardless of the gateway timestamp. For example, a gateway may modify its gateway timestamps to maintain order, or may aggregate messages into a single message containing the same timestamp, so that messages are processed by the operator node 240 in order relative to messages from other gateways according to the gateway timestamps, while maintaining their source order. Additionally, hierarchical perimeters may be established, where messages are time-stamped at each perimeter they cross. The operator node 240 may determine where each timestamp ranks in the hierarchy (e.g., based on a database entry that ranks perimeters in the hierarchy) and may process the message according to the hierarchy, which is discussed in more detail below with respect to Figures 5A and 5B.

[0016] In one embodiment, messages are received by operator node 240 that have not passed through a time perimeter and therefore do not contain a synchronized timestamp. When such messages are encountered, they may be time-ordered by operator node 240 on the fly (e.g., by operator node 240 time-stamping the messages, time-stamping messages received from gateway 220, and ordering the messages based on the timestamp applied by operator node 240 in addition to the perimeter timestamp).

[0017] After the messages are time-ordered, the operator executes 336 the time-ordered operations. Examples of data flows 300 and resulting time-ordering operations are as follows: As an example, if the gateway is part of a financial exchange and the messages are financial orders received from a source with multiple market participants submitting orders through different gateways, the time-ordering operation may be to resolve the financial orders in a chronological order, thereby ensuring fairness. As another example, if the gateway is a conduit for receiving information in a multiplayer real-time network game and the messages are game commands received from multiple players of the game at different gateways, the time-ordering operation may include progressing the game according to the chronological order of the game commands. Fairness may be ensured, for example, in a first-person shooter game, by the user who takes the first shot at the target according to a reference clock being credited for that shot. As yet another example, if the messages are commands sent by multiple controllers over a network with random delays to a robot (e.g., in a manufacturing environment), the time-ordering operation may ensure that the commands are acted on in the order issued by the controllers to the time perimeter, regardless of the order in which the instructions arrive at the robot.

[0018] In another aspect of time ordering, FIG. 4 is a flow diagram for performing time-triggered distribution of messages according to one embodiment of the present disclosure. FIG. 4 includes four columns: source activity, gateway activity, coordinator activity, and operator activity. In this example, the boxes displayed in the columns are executed by the corresponding entities. Flow 400 begins with sub-process 410 of establishing software-based synchronization. Similar to sub-process 310, software-based synchronization 410 begins with a coordinator (e.g., coordinator 230) initiating 412 synchronization operations on a set of machines including one or more gateways (e.g., gateway 220) and a reference machine (e.g., operator node 240 including reference clock 245). The machines (including one or more gateways and reference machine) exchange probes 414 with machines in the domain 205. As illustrated, the coordinator estimates 416 an absolute offset with respect to the reference clock of the reference machine based on the exchanged probes, however, in alternative embodiments, the gateways themselves may estimate the absolute offset. Software-based synchronization, including probe replacement and absolute offset estimation, is described in detail below in Figures 6 through 11. At the end of software-based synchronization 410, a gateway (e.g., gateway 220) is synchronized to the reference clock.

[0019] Sub-process 420 of flow 400 transmits 421 a message from an operator (e.g., operator 240) to a gateway (e.g., gateway 220). For example, the message may be a buy / sell order (so-called "order book information") in a financial exchange, where all buy / sell orders must be distributed to market participants at the same time, even if the market participants are in different geographic locations. In another embodiment, the message from the operator may be confidential information independent of the performance 336 of the time-order information. In either case, the message is generally of a type that should be released to the public (e.g., source node 210) at the same time to achieve fairness, so that no member of the public has an unfair advantage in the use of the information. The transmitted message may be a copy of the same original message sent to multiple gateways for further distribution.

[0020] After receiving 422 the messages, sub-process 430 in which time-ordered distribution is performed occurs as part of flow 400. Each buffer gateway 432 provides the message for distribution. After the message is buffered, the gateway distributes 434 the message to the destination. The distribution 434 of the message occurs simultaneously at each gateway to ensure that the destinations receive the message at the same time (depending on the jitter encountered by the message on its way to the destination). To ensure simultaneous delivery 434 of the message, the gateways are instructed to deliver the message at a time determined from each gateway's local clock adjusted using that gateway's estimated absolute offset, as estimated in sub-process 410. In embodiments where network conditions or physical limitations of network links are known to delay messages to certain destinations, the gateways may calculate this delay by applying an additional offset to the timing at which messages to those particular destinations are distributed. In another embodiment, messages are sent to different financial exchanges for execution by a single market participant where they must simultaneously submit buy / sell orders for execution at the different financial exchanges.

[0021] 5A-5B are graphs of a computer system using two or more regions of network machines, according to an embodiment of the present disclosure. As distinguished from FIG. 2, which includes one region 205, FIG. 5A-5B includes multiple regions, each defined by a different time perimeter. The gateways 520A together form a time perimeter 550A that defines the boundary of the region 505A. The gateways 520B together form a time perimeter 550B that defines the boundary of the region 505B. Messages bundled for processing by the operator node 540 first cross the time perimeter 550A and then cross the time perimeter 550B. As shown, the operator 540 includes a reference clock 545 to which all of the gateways 520A and 520B are synchronized. However, the region 505a, in addition to being synchronized to the reference clock 545, may be synchronized to a different reference clock, such as the reference clock of one of the gateways 520a. In such a scenario, gateway 520A may have two reference clocks, reference clock 545 and an estimated absolute offset with respect to one of the reference clocks of gateway 520A. In such a scenario, software-based synchronization 310 and 410 may be applied to one or both reference clocks. Domains 505A and 505B may be under the control of different entities (e.g., different financial exchanges or different video game servers).

[0022] In one embodiment, perimeters 550A and 550B may be organized hierarchically, where one of the perimeters is known to operator node 540 to have a higher priority than the other in determining how to time-order messages. For example, if perimeter 550A is higher in the hierarchy, a message with timestamps from both perimeters 550A and 550B will be processed according to the timestamp from perimeter 550A. However, if the timestamps from entity 550A are the same or within a threshold distance of each other (e.g., a distance defined by a margin of error such as a few nanoseconds), the timestamp applied at perimeter 550B may be trusted as a tiebreaker for operator node 540 to determine how to time-order messages.

[0023] FIG. 5B illustrates an example embodiment in which one perimeter partially overlaps with another perimeter. As shown, perimeter 552A forming region 515A overlaps with perimeter 552B forming region 515B to form shared region 515C. Gateways 520B together form time perimeter 552A that defines the boundary of region 515A. Gateways 521B together form time perimeter 552B that defines region 515B. The overlap occurs because gateway 522B is part of both perimeters 550B and 551B. When regions 515A and 515B overlap, shared region 515C is formed. Gateway 522B may be synchronized to multiple reference clocks, such as reference clocks 545B and 546B, in such a scenario. In one embodiment, other machines in shared domain 515C may have an estimated absolute offset with respect to the reference clocks of domains 515A and 515B, i.e., reference clocks 545B and 546B. In such a scenario, software-based synchronization 310 and 410 may be applied to one or both of the reference clocks 545B and 546B for machines in shared domain 515C. Domains 515A and 515B may be under the control of different entities (e.g., different financial exchanges or different video game servers). As described above with reference to FIG. 5A, hierarchical processing of time ordering of packets received at the perimeter of shared domain 515C may be performed.

[0024] 6-11 illustrate a software-based synchronization process for one embodiment. FIG. 6 is a graph of a network including a coordinator for implementing an embodiment of the present disclosure. Network 600 illustrates a different implementation of network 200 of FIG. 2, with like reference numbers and terms carrying the same weight description throughout these and other figures. Network 600 includes machines 610 that are nodes of the graph. Machines 610 may be gateways, such as gateway 220 of FIG. 2. As depicted, network 600 is a mesh network, with each machine 610 linked to each other by one or more links (some links are omitted for clarity). However, network 600 may be any other type of network. For example, network 600 may be a network in which machines are connected in series over wires, or may be in any other configuration. The network may be a large network spanning multiple physical regions (e.g., from New York to San Francisco) or a small network, such as a network within a single server blade. In one embodiment, network 600 may be a network of clocks on one or more printed circuit boards.

[0025] The communication links between any pair of machines are represented as edges 620 between nodes in the graph. Each edge 620 typically represents multiple paths between any two machines 610. For example, network 600 may include many additional nodes beyond the machines 610 shown, such that there may be multiple different paths through different nodes between any pair of machines 610.

[0026] The network 600 further includes a coordinator 630 and a reference clock 640. In this example, the coordinator 630 instructs the machines 610 to obtain network observations by probing the other machines 610, as described in more detail below with respect to FIG. 8. The coordinator 630 may store, or cause to be stored, a record of those network observations, as described in more detail below with respect to FIG. 8. The coordinator 630 may also send control signals to the machines 610. The term control signal as used herein may refer to a signal indicating that the frequency of the machine's local clock is to be adjusted by a specified amount (thus compensating for drift of the local clock) and may also refer to a signal indicating that the time indicated by the machine's local clock is to be adjusted by a specified amount (thus compensating for offset of the local clock).

[0027] In one embodiment, the coordinator 630 stores a graph mapping the topology of the network 600, either within the machine housing coordinator 630 or within one or more machines of the network 600. The graph may include a data structure that maps the connections between the machines of the network 600. For example, the graph may map both direct connections between the machines (e.g., machines that are physically or logically next hops from one another) as well as indirect connections between the machines (e.g., each multi-hop path that may be taken for communication of probes, etc.). The graph may further include network observations corresponding to each edge in the graph (e.g., indicating probe transit times for probes that traversed the edge, and / or additional information such as that shown in FIG. 8).

[0028] One of the machines includes a reference clock 640. The reference clock 640 is a clock to which the clocks in the machines of the network 600 are synchronized. In one embodiment, the reference clock 640 is a highly calibrated clock that is not subject to drift, which drift is contained in a different machine 610 than the other machines to be synchronized. In another embodiment, the reference clock 640 may be an off-the-shelf local clock already present in the machine 610 that serves as a master reference for the other machines 610, whether it is a highly regulated clock that is accurate to "absolute time" as may be determined by an atomic clock or some other highly accurate source clock. In such a scenario, the coordinator 630 may select which machine 610 will act as the master reference arbitrarily, or may assign the reference machine based on input from an administrator. In an embodiment in which the reference clock 640 is accurate to "absolute time", the reference clock may be a time source such as a Global Positioning System (GPS) clock, a Precision Time Protocol (PTP) grandmaster clock, an atomic clock, etc. 10-11, the coordinator 630 may use the reference clock 640 when calibrating the control signals. By making signal corrections for frequency and / or offset based on the reference clock 640, the coordinator 630 synchronizes the local clocks of the machines 610 to the reference clock 640 with high precision.

[0029] Although only one reference clock 640 is shown in FIG. 6, in one embodiment, there may be multiple reference clocks 640 (e.g., as described with reference to FIGS. 5A and 5B). For example, additional reference clocks may be used for redundancy in case of failure of the reference clocks or the machines housing them. As another example, the machines 610 may be divided into multiple groups or regions (e.g., as described with reference to FIGS. 5A and 5B) (e.g., based on settings applied by an administrator of the network 100). For example, the network 600 may be configured to divide the machines 610 into groups, each having a predetermined number or percentage of the machines 610 that may improve performance or implementation. Within each group, one machine may be designated as the reference, and that machine's clock is used as the reference clock for the remaining machines. Each group may designate one group as the reference group, such that the reference clock of the reference group will serve as a reference for all groups. The reference clocks and groups may be automatically designated consistent with the clock designations described herein, or may be entered by the network administrator. As an example, if a network contains 5000 machines and an administrator programs the network to divide the machines into groups holding 100 machines each (e.g., based on specifying 100 machines or by specifying a percentage), then there will be 50 groups of machines, each containing 100 machines. One of the 100 machines will be a reference machine and the other 99 of the 100 machines will synchronize their clocks with the reference machine. Furthermore, one of the 50 groups will be the reference group and the other 49 groups will synchronize to the reference group.

[0030] The coordinator 630 may be implemented in a stand-alone server, may be implemented on one or more of the machines 610, or may have its functionality distributed across two or more machines 630 and / or stand-alone servers. The coordinator 630 may be accessible via link 620 in the network 600, or via a link to a machine or server housing the coordinator 630 outside the network 600. The reference clock 640 may be implemented in the coordinator 630 or may be implemented as a separate entity either on the machine 610, on a stand-alone server in the network 600, or on a server or machine outside the network 600.

[0031] FIG. 7 is a data flow diagram for correcting clock frequency and / or offset according to one embodiment of the disclosure. For example, some or all of data flow 700 may be used to achieve software-based synchronization 310 and / or 410. The left column of FIG. 7 describes the activities of a coordinator (e.g., coordinator 630) to achieve high-precision clock synchronization by correcting clock frequency (i.e., drift) and / or offset, and the right column describes the activities of a machine (e.g., machine 610). FIG. 7 can be considered to include three phases: a first phase in which network observations are made by having a machine probe other machines in the network (e.g., network 600), a second phase in which network observations are used to estimate offsets and drifts, and a third phase in which frequency and / or offsets are compensated / corrected to achieve high-precision clock synchronization between the machines.

[0032] As part of the first phase, data flow 600 begins with a coordinator (e.g., coordinator 630) assigning 702 paired machines. The term paired as used herein refers to machines that send probes to each other for the purpose of collecting network observations. As used herein, the term network observations may refer to observable qualities of the network (e.g., effects of network operations defined below, queuing delays, observable drift, offsets, etc.). The term probe as used herein refers to electronic communications sent from one machine to another, the electronic communications being time stamped upon transmission from the transmitting machine and upon reception at the receiving machine. The time stamps may be applied by any component of the machines configured to apply time stamps, such as the respective CPUs of the transmitting and receiving machines and / or respective NICs that are part of or operably coupled to the transmitting and receiving machines. A single machine is typically paired with multiple other machines. When assigning paired machines, the coordinator may assign a machine to be paired with some machines, which may be less than all the machines in the network. In one embodiment, the number and pairing of machines may be predefined or may change dynamically based on network conditions (e.g., congestion, latency, etc.) Machines may be selected randomly or through a deterministic algorithm.

[0033] The data flow 700 proceeds with the coordinator 630 instructing the paired machines to send 704 probes to each other. Network observations collected from the probe transmissions collect 706 probe records. The term probe record as used herein may refer to a data structure that includes network observations obtained from a probe transmission, such as the identity of the sending and receiving machines, the sending timestamp, the receiving timestamp, etc. The transit time of the probe may be determined based on the sending timestamp and the receiving timestamp. Probe records are described in more detail below with respect to FIG. 8. While the embodiment described herein shows the coordinator collecting the probe records, in one embodiment, some or all of the machines may collect probe records for probes sent to or from them, respectively, and may themselves perform processing on the probe records.

[0034] After the probe records are collected, the coordinator (e.g., coordinator 630) enters into a second phase in which it uses the collected probe records to estimate the offset and / or drift of the machine (e.g., machine 610). In this example, to achieve an accurate estimation, the coordinator first filters 708 the probe records to identify coded probes. As used herein, the term coded probe refers to a probe corresponding to a probe record that is not subject to noise, such as delays resulting from probe queuing. One way in which the coordinator identifies coded probes is described in further detail with respect to FIG. 8. A subset of the probe records that correspond to coded probes may be referred to as coded probe records. In embodiments in which the probe records are collected at a given machine, the given machine may perform filtering 708 of the probe records collected by the given machine.

[0035] The data flow 700 continues by applying 710 a classifier to the encoded probe records. The classifier may be a machine learning model trained through supervised learning. An example of a classifier is a support vector machine ("SVM"). The coordinator may input upper and lower bounds derived from the encoded probe data (i.e., transit time samples) from the two paired machines over a period of time. The output of the classifier is linearly fit to the transit time data with a slope and intercept. The data flow 700 then continues with 712 where the coordinator estimates the drift between the paired machines. In one embodiment, the coordinator estimates the drift to be equivalent to or a function of the slope of the linear fit (e.g., an estimate of the rate of change of transit time). The coordinator may also estimate the offset using the intercept of the linear fit. Determining / estimating the offset may be performed in a similar manner as is done for drift elsewhere disclosed. In embodiments where probe records are collected at a given machine, the given machine may apply 710 a classification to the probe records collected by the given machine and perform an estimate 712 of the drift between the pair of machines.

[0036] The drift estimate may not be completely accurate because the coded probe did not suffer from queuing delays, but the coded probe may suffer from network behavior effects. Network behavior effects, as used herein, may refer to noise caused by components of the network. For example, a link or gateway between two pairs of machines may introduce latency or jitter that affects the drift estimate. In one embodiment, the coordinator uses 714 network effects based on frequency drift estimates across three or more machines. Further details for using 714 network effects are described below with respect to FIG. 9.

[0037] The coordinator sends the observations to the control loop of the machine's local clock 716, for example, by applying a filter to the estimated drift based on the effect of network operations, or by feeding the estimated drift and the effect of network operations to a machine learning model that is the absolute drift, where "absolute" drift or offset is relative to a reference clock. Details of the control loop and the coordinator-estimated absolute drift are described in more detail below with respect to Figures 10-11. After estimating the absolute drift, the coordinator may determine whether to correct the clock frequency in real time or near real time (718). The absolute offset may also be corrected 718, or the coordinator may perform an offline correction 720. Methods for determining whether to correct in real time or offline are described further below with reference to Figures 10-11.

[0038] In addition to correcting the clock frequency and / or offset, process 700 periodically recurs for each pair of machines to ensure that any new offsets and drifts that occur after correcting the clock frequency and / or offset are continually corrected. For example, process 700 may occur periodically (e.g., every two seconds) to ensure that synchronization is maintained across a network (e.g., network 100). The updated synchronization is used in each instance of software-based synchronization 310 and 410.

[0039] Returning to the discussion of FIG. 6, a discussion now appears that further details the creation of probe records in activities 202 to 206 of data flow 200. The number of machines 600 paired to a given machine may be a number fixed by an administrator. In one embodiment, the coordinator 630 may dynamically adjust the number of machines paired for probing per given machine based on network conditions. For example, if the network connectivity is above a threshold quality (e.g., based on an amount of low latency), the coordinator 630 may reduce the number of machines paired for probing in proportion to the quality of the network connectivity. On the other hand, if the network connection quality deteriorates, the coordinator 630 may increase the number of machines paired for probing per given machine. The number of machines paired may also vary based on the network size.

[0040] When selecting which machines should be paired with a given machine, the coordinator 630 may randomly determine each machine with which a given machine should be paired. In one embodiment, the coordinator 630 non-randomly determines the pairings based on ease of computation, accuracy (e.g., clock synchronization accuracy as dictated by the network graph), and load distribution across each machine 610. The coordinator 630 may carefully determine the pairings based on design choices, with an administrator selecting the pairings or selecting parameters that cause a particular pairing to be selected. Additionally, the coordinator 630 may dictate that more pairings occur for machines with a high level of diversity compared to some pairings for machines with a low level of diversity. As used herein, the term "diversity" may refer to the number of paths that a probe may take traversing the network 600 to reach a machine from another machine, with a greater number of paths indicating greater diversity.

[0041] It should be noted that the probes may occur at a particular frequency or period that may vary based on one or more parameters. For example, the coordinator 630 may instruct a machine with high diversity to transmit a larger number of probes to one or more paired machines, each following a different path, compared to a machine with low diversity, which may transmit a smaller number of probes. Thus, the coordinator 630 may instruct the machine to transmit to its paired machines at a frequency that varies in proportion to the diversity.

[0042] A coordinator (e.g., coordinator 630) determines 702 that a given machine 610 is paired with various other machines 610 and receives probes from those same machines. The term "exchange" is used here to describe the scenario in which paired machines send and receive probes from each other. As used here, the term exchange does not imply timing aspects such as a requirement that machines transmit at the same time or in response to each other.

[0043] In one embodiment, network 600 may be an untrusted system, such as a system facilitating a blockchain network. In such an embodiment, some of the machines 610 may misbehave and misrepresent the data used to determine offset and / or drift. In such a scenario, in addition to the probes discussed above, with respect to FIG. 8, coordinator 630 may instruct machines 610 to probe more machines (e.g., one-third of the machines in network 600) at a lower frequency (e.g., one in every tenth, or 100, or 1000 probes sent in the normal course). Coordinator 630 may determine whether the data obtained from the greater number of machines matches (e.g., within a threshold tolerance of drift and / or offset) with the fewer number of machines probed in the normal course, and may alert an administrator or take corrective action if inconsistent probe data is detected.

[0044] FIG. 8 is a data structure diagram illustrating a method for identifying probe records and encoded probes from the probe records, according to one embodiment of the disclosure. Data structure 800 includes exemplary probe records for multiple probes. Although data structure 800 is depicted as including a probe record for a single sending machine "A" (e.g., machine 610 of network 600) and a single receiving machine "B", any number of sending or receiving machines may have probe records stored within data structure 800. Column 810 includes identifiers for the probes, with each identifier 1 through 6 corresponding to a probe record for a probe. Column 820 indicates which machine is the probe represented in a given probe record. As illustrated, column 820 indicates that the transmitter labeled "A" transmits each probe, however, this is merely exemplary and various transmitters may be identified in column 820.

[0045] Column 830 indicates which machines received the probe indicated by a given probe record. As shown, column 830 indicates that a receiving machine labeled "B" received each probe, however, this is merely exemplary and various receiving machines may be identified in column 830. Column 840 indicates the transmission time of the probe. The transmission time is a time that is time stamped either by the sending machine itself (e.g., the CPU of sending machine "A" of network 600) or an interface or device operatively coupled to the sending machine (e.g., the NIC of sending machine "A" of network 100). Similarly, column 850 indicates the reception time of the probe, which is a time stamp by the receiving machine or, for example, a NIC of the receiving machine. In one embodiment, a machine with a single CPU may have multiple NICs. In such an embodiment, the coordinator 630 may synchronize multiple NICs of a machine (e.g., a receiving machine) to the clock of the machine's CPU (e.g., by using the NIC as a reference machine and synchronizing its time to the NIC's time, as described herein), and then synchronize other NICs to the CPU, thus synchronizing the machine's multiple NICs.

[0046] The coordinator may command the machines to transmit probes at a specified or predefined time interval between the probes. As used herein, the term "transmission time interval" (δ) refers to a specified or predefined time interval between the transmission times of two probes. The interval may be a fixed value or may be dynamically selected by the coordinator based on the network conditions (e.g., if the network is congested, a longer transmission time interval may be selected). As can be seen in FIG. 8, probe 1 is transmitted at a time Ti, which may be any time or may be a time specified by the coordinator. Probe 2 is transmitted at a time T1+δ, as the coordinator has instructed the transmitting machine A to transmit probes to the receiving machine B at one or more time intervals. Furthermore, probes may be commanded by the coordinator to be transmitted from the transmitter A to the receiver B at a reference time Ti, although for ease of illustration, only two are shown in FIG. 8. Similarly, probes 3 and 4 are sent at times T2 and T2+δ, respectively, and probes 5 and δ are sent at times T3 and T3+δ, respectively.

[0047] Probe IDs 1 and 2, 3 and 4, and 5 and 6 are paired to show how the coordinator determines whether a pair of probes is a coded probe. A coded probe is a probe that is transmitted at a particular interval of δ or within a threshold distance from δ. That is, the probes are coded based on the space between each probe. Delays in time-stamping probes can occur due to queues at the sending machine 820 and / or the receiving machine 830 or through intermediate nodes. Thus, a coded probe is a pair of probes that are sent consecutively by the same transmitter 820 and received by the same receiver 830 and whose reception times are within δ times, or a threshold margin of δ (to correspond to the smallest difference in delay between the two probes). That is, the transit times of the two coded probes are approximately the same. Although pair is primarily used to describe coded probes, this is merely exemplary and the coded probes may also be a triplet, quadruplet, etc. of probes with an interval of δ.

[0048] Probes 1 and 2 illustrate a scenario where the two probes do not form a coded probe since probe 1 has a transit time of TT, but probe 2 has a transit time of TT+D (D represents delay), where D is greater than the threshold margin. That is, probe 2 has a transit time of D longer than probe 2. Probes 3 and 4 illustrate a scenario where the two probes do not form a coded probe since probe 3 has a transit time of D longer than probe 4. However, probes 5 and 6 are coded probes since they have the same transit time (within the acceptable threshold).

[0049] In one embodiment, the data structure 800 is stored in a memory directly accessible to the coordinator 630 (e.g., local memory of the running coordinator 630 machine). In another embodiment, the data structure 800 is distributed across the machines 610, with each machine storing a local data structure 800 for probes exchanged between it and the other machines. Various processes are described below with respect to Figures 9-11 that use the information in the data structure 800, which may be performed by the coordinator 630, but may also be performed by the machine 610. When the machine 610 is performing a process (e.g., identifying encoded probes, applying a support vector machine, etc.), if one machine is overloaded with the other process, the other machine of the pair may retrieve the data in the overloaded machine's data structure 800 and perform the process in place of the overloaded machine.

[0050] As described above with respect to FIG. 7, the encoded probe records may be input to a classifier, such as an SVM classifier, where drift may be estimated. However, the drift estimate may be inaccurate, but can be corrected by using network effects. FIG. 9 is a graph of a system illustrating the identification and correction of loop errors. Machines 910 include machine 1, machine 2, and machine 3 that together form a network loop. Machine 910 has similar functionality as described above with respect to machines 110 and 610. Although only three machines are depicted, this is for convenience only, and any number of machines may form a network loop. Links 920 connect the machines of the network loop, with link 920-1 connecting machine 1 to machine 2, link 920-2 connecting machine 2 to machine 3, and link 9203 connecting machine 3 to machine 1. Each link may represent multiple different paths between each pair of machines.

[0051] The numbers on each link 920 are the drift between the two machines connected by each link in arbitrary units. Thus, link 920-1 indicates a drift of +20 units for the drift of machine 1 relative to the drift of machine 2, link 920-2 has a drift of -15 units between machines 2 and 3, and link 920-3 indicates a drift of +5 units between machines 3 and 1. The sum of the drifts around a given loop (called the loop drift error, which is the result of the network effect applied to the frequency) indicates the error in the estimated clock drift. Thus, if there was no loop drift error, the sum of the drifts of all the links in the loop would be 0 units. However, as shown, the total drift is 10 units (20-15+5=10), which may be caused by an inaccurate clock estimation, but this can be corrected using network effects. When assigning pairs, the coordinator may assign a given machine to be part of multiple loops. The coordinator may combine all the loops for machines of different pairs to more accurately estimate the clock drift using network effects. When allocating pairs, the coordinator is not constrained by the need for path symmetry, and the time (or number of hops) it takes to go from machine 1 to machine 2 does not have to be the same as the time it takes to go from machine 2 to machine 1. In one embodiment, some of the loops include a reference clock 640, thus ensuring that the network effect is determined with respect to a reference clock. In one embodiment (e.g., when the coordinator 630 is not present), the network effect can be used without reference to a reference clock, where each clock determines its frequency drift and a statistical operation (e.g., average) is taken to determine the loop drift error. These loop drift errors around the loop are used to adjust the absolute drift of the machines in the loop. For example, the loop drift error of a loop may be allocated between different machines in the loop.

[0052] 10 is a block diagram of a model of an adaptive stochastic control system for correcting a machine's local clock frequency according to one embodiment of the present disclosure. A control loop 1000 is used to adjust the frequency of a local clock 1015 of a machine 1010 by a control signal 1004. The machine 1010 may be a machine of the network 600 and includes the same functionality as described above with respect to the machines 110 and 610. The coordinator module 1030 is depicted as part of the machine 1010 but may alternatively be fully or partially located in a separate coordinator (e.g., the coordinator 630) as described above with respect to FIG.

[0053] The coordinator module 1030 estimates the absolute offset and absolute drift 1002 of the machine 1010 as described above with respect to Figures 7-9. These are absolute quantities because they are measured relative to a reference clock that is connected to a source of absolute time. The control loop 1000 also includes a filter 1060 and a controller 1070. The filter 1060 may be a predefined filter (e.g., a Kalman filter), a filter selected from an adaptive filter bank based on observations, a machine learning model, etc. The Kalman filter and adaptive filter bank are discussed in more detail with respect to Figure 11, and the use of machine learning models is discussed in more detail with respect to Figure 11.

[0054] The purpose of the filter 1060 is twofold: first, to reduce noise in the drift and offset estimates, and second, to estimate the natural progression of the clock. The process 700 (from FIG. 7) is repeated periodically (e.g., every 2 seconds), and thus the control loop 1000 loops periodically as well. In one embodiment, the clock offset is estimated in the middle of the period (e.g., 1 second into the 2 second period), while the control signal occurs at the end of the period (e.g., the 2 second mark of the 2 second period). Thus, in addition to reducing noise in the estimates, the filter 1060 estimates filtered offset and drift values ​​to output 1003 that are accurate at the time of control. The filtered offset and drift are received by the controller 1070. The controller 1070 outputs a frequency (and offset) adjustment signal 1004 to the local clock 1015 of the machine 1010, the adjustment reflecting the change in the frequency and offset values ​​of the local clock 1015 to remove the offset and drift from the local clock 1015. The frequency and offset adjustments are also fed back to the filter 1060 as parameters for the filter, in addition to the estimated offset and drift for the filter, in subsequent cycles of the control loop. In this control loop, the plant being controlled is determined by the state variables {absolute offset, absolute drift} of the local machine, and an adaptive stochastic controller is used to control the plant. As described below with respect to FIG. 11, adaptive stochastic control refers to adjusting the control signal based on the likelihood that a given adjustment is the correct adjustment compared to other possible adjustments; as the control signal is applied, the actual adjustments are observed and the probability that each possible control signal will lead to the correct adjustment is adjusted.

[0055] 11 is a block diagram of an adaptive filter bank according to one embodiment of the present disclosure. The term adaptive filter bank, as used herein, may refer to a collection of candidate filters, each of which is optimal for removing noise from a signal based on the type and degree of noise. For example, some noises can be observed, such as those discussed with respect to FIGS. 6 to 10 (e.g., queuing delays, network operation effects, loop errors, etc.). However, some noises are inherent to the state of the machine and are unknown to the control loop 1000 (e.g., noise variations in response to control inputs across different manufacturers and models of equipment). The unknown noises are referred to herein as state noises.

[0056] As described above with respect to FIG. 10, the filter 1160, which includes the functionality of the filter 1060, includes a bank of candidate filters 1161 (also referred to herein as an adaptive filter bank), which may be Kalman filters. Each of the candidate filters 1161 corresponds to a different level of state noise. The filter selection module 1162 is a probabilistic selection module that selects a filter from the candidate filters 1161 by calculating the probability that each candidate filter is the best fit, and then selecting the candidate filter with the best fit. First, the filter selection module 1162 receives the observed noise, uses the observed noise to filter the estimated drift and offset 1102, and selects the most probable candidate filter 1161, which is used to output the filtered drift and offset 1103 to the controller 1070. Using adaptive probabilistic control, the filter selection module 1162 initially finds that all filters are equally likely, and is able to select a filter arbitrarily. After selecting a filter and observing how the local clock 1015 responds to control signals, the filter selection module 1162 adjusts the likelihood that each candidate filter 1161 is optimally applied. Thus, as more information about control signals and network observations is fed back to the filters 1160 over time, the selection of the appropriate candidate filter 1161 eventually converges to the best matching candidate filter.

[0057] As discussed with reference to FIG. 7, when deriving the control signal to be sent to the machine's local clock, the corrections may be performed in real time, thus resulting in a real time control signal (or near real time control signal). In one embodiment, the corrections may be performed offline, such as when the observation noise is much smaller than the state noise. For example, the coordinator may determine whether the observation noise is a predetermined amount or factor smaller than the state noise. In response to determining that the observation noise is a predetermined amount or factor smaller than the state noise, the coordinator may perform the adjustments offline (or in a batch manner), otherwise, the coordinator may perform the adjustments in real time or near real time, thus causing the control signal to be sent quickly to the local clock. An administrator of the system may set parameters that determine when offline corrections are made, or may choose not to use offline corrections at all.

[0058] The foregoing description of the embodiments of the present disclosure may be implemented in a software module installed for use with off-the-shelf clocks, including inexpensive, inaccurate clocks such as quartz clocks, to synchronize such clocks very accurately. The foregoing description of the embodiments of the present disclosure has been presented for illustrative purposes and is not intended to be exhaustive or to limit the invention to the precise aspects disclosed, but merely to illustrate different examples. It should be understood that the scope of the present disclosure includes other embodiments not discussed in detail above. Those skilled in the art can appreciate that many modifications and variations are possible in light of the above disclosure without departing from the spirit and scope defined in the appended claims. Thus, the scope of the present disclosure should be determined by the appended claims and their legal equivalents. Similar reference numbers in different drawings have the same weight and effect of their corresponding descriptions in different drawings.

[0059] Finally, the language used in the specification has been selected primarily for ease of reading and educational purposes, and may not have been selected to delineate or limit the subject matter of the present invention. Accordingly, it is intended that the scope of the present invention be limited not by this detailed description, but by any claims issued in an application based on this specification. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the present disclosure, which is set forth in the appended claims.

Claims

1. 1. A non-transitory computer-readable medium including a memory having encoded thereon instructions for performing time-triggered distribution of messages from a region of networked machines to multiple destinations, the instructions, when executed by one or more processors, cause the one or more processors to perform operations, the instructions comprising: executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of said region of networked machines and connected to transmit messages to one or more destinations, said perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along said perimeter; said gateway receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock; the distribution timing ensures that messages received within a given time window at any gateway along the perimeter are transmitted in the order in which they were received along the perimeter; the gateways perform the distribution of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock, the gateway buffering the received messages until a transmission time indicated by the distribution timing is reached; the gateway applies an additional offset to the delivery timing based on known jitter between the gateway and their destinations; A non-transitory computer-readable medium containing instructions.

2. A non-transitory computer-readable medium including a memory having encoded instructions for performing time-triggered distribution of messages from a region of networked machines to multiple destinations, the instructions, when executed by one or more processors, cause the one or more processors to perform an operation, the instructions comprising: executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of said region of networked machines and connected to transmit messages to one or more destinations, said perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along said perimeter; said gateway receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock; the distribution timing ensures that messages received within a given time window at any gateway along the perimeter are transmitted in the order in which they were received along the perimeter; the gateways perform the distribution of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock, the gateway buffering the received messages until a transmission time indicated by the distribution timing is reached; Synchronizing the gateways includes, for at least one of the gateways: correcting a local clock of the gateway to match the reference clock, the distribution timing including referencing a time relative to the reference clock for the distribution; A non-transitory computer-readable medium containing instructions.

3. The non-transitory computer-readable medium of claim 1 or 2, wherein the distribution timing causes simultaneous distribution of at least a portion of the received messages from each of the gateways.

4. The non-transitory computer-readable medium of claim 1 or 2, wherein each of the gateways receives the instructions from a coordinator device including the distribution timing.

5. The non-transitory computer-readable medium of claim 1 or 2, wherein the transmission time indicates an end of the given time window.

6. 3. The non-transitory computer-readable medium of claim 1 or 2, wherein the received messages are buffered such that the delivery of the received messages can be performed in order without missing any messages that arrived before their delivery deadline.

7. 1. A computer-implemented method for performing time-triggered distribution of messages from a domain of networked machines to multiple destinations, the method comprising: executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of said region of networked machines and connected to transmit messages to one or more destinations, said perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along said perimeter; said gateways receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock, said distribution timing ensuring that messages received within a given time window at any gateway along said perimeter are transmitted in the order in which they were received along the perimeter; the gateways performing the distribution of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock, and the gateway buffering the received messages until a send time indicated by the distribution timing is reached; Including, A computer-implemented method, wherein the gateways apply an additional offset to the delivery timing based on known jitter between the gateways and their destinations.

8. A computer-implemented method for performing time-triggered distribution of messages from a domain of networked machines to multiple destinations, the method comprising: executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of said region of networked machines and connected to transmit messages to one or more destinations, said perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along said perimeter; said gateways receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock, said distribution timing ensuring that messages received within a given time window at any gateway along said perimeter are transmitted in the order in which they were received along the perimeter; the gateways performing the distribution of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock, and the gateway buffering the received messages until a send time indicated by the distribution timing is reached; Including, Synchronizing the gateways includes, for at least one of the gateways: A computer-implemented method comprising: correcting a local clock of the gateway to match the reference clock, the distribution timing referencing time relative to the reference clock for the distribution.

9. 9. The computer-implemented method of claim 7 or 8, wherein the distribution timing causes simultaneous distribution of at least a portion of the received message from each of the gateways.

10. 9. The computer-implemented method of claim 7 or 8, wherein each of the gateways receives instructions to perform the time-triggered distribution from a coordinator device that includes the distribution timing.

11. 9. The computer-implemented method of claim 7 or 8, wherein the transmission time indicates an end of the given time window.

12. 9. The computer-implemented method of claim 7 or 8, wherein the received messages are buffered such that the delivery of the received messages can be performed in order without missing any messages that arrive before their delivery deadline.

13. a memory having encoded instructions; one or more processors; A system comprising: The one or more processors, upon executing the instructions, executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of an area of ​​networked machines and connected to transmit messages to one or more destinations, the perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along the perimeter; said gateways receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock, said distribution timing ensuring that messages received within a given time window at any gateway along said perimeter are transmitted in the order in which they were received along the perimeter; the gateways performing the dissemination of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock; Executing an operation including The gateway applies an additional offset to the delivery timing based on known jitter between the gateway and their destinations.

14. A method for programming a program comprising: a memory having encoded instructions; one or more processors; A system comprising: The one or more processors, upon executing the instructions, executing a software-based synchronization process to synchronize each of a plurality of gateways with a reference clock, each gateway being a machine on a perimeter of an area of ​​networked machines and connected to transmit messages to one or more destinations, the perimeter defining a logically connected group of gateways that transmit messages in the order in which they are received along the perimeter; said gateways receiving messages from within said region of networked machines for distribution to destinations outside said region of networked machines according to distribution timing based on absolute time relative to said reference clock, said distribution timing ensuring that messages received within a given time window at any gateway along said perimeter are transmitted in the order in which they were received along the perimeter; the gateways performing the dissemination of the received messages, each gateway determining an absolute time based on the gateway's synchronization with the reference clock; Executing an operation including Synchronizing the gateways includes, for at least one of the gateways: and correcting a local clock of the gateway to match the reference clock, the distribution timing including referencing time relative to the reference clock for the distribution.

15. 15. The system of claim 13 or 14, wherein the distribution timing causes simultaneous distribution of at least some of the received messages from each of the gateways.

Images