System, apparatus and method for synchronizing timestamp devices of a computer platform

Abstract

Apparatus, system and method may control verifying timestamp synchronization at input ports of a timestamp circuit, and verifying timestamp synchronization among timestamp circuits of a distribution system while the distribution system operates to distribute data items received from client computing devices. The distribution system may distribute the data items in chronological order, based on times of receipt of the data items, and accounting for network latencies in transmission of the data items to the distribution system and synchronization offsets respectively of input ports of each timestamp circuit of the distribution system.

Description

SYSTEM, APPARATUS AND METHOD FOR SYNCHRONIZING TIMESTAMP DEVICES OF A COMPUTER PLATFORMTECHNICAL FIELD

[0001] The present disclosure generally relates to synchronizing timestamp devices of a network system, and in particular, verifying timestamp synchronization among timestamp devices of a distribution system during operation of the distribution system.BACKGROUND

[0002] Computing devices of respective entities exchange data with other computing devices of other respective entities over communication networks for multitudes of applications. The speed of transfer of data between respective computing devices depends on network properties and processing operations along or associated with a communication path over which data is conveyed from one computing device to another device. For example, client computing devices may transmit respective data items to a system, such as a distribution system, and timestamp devices of the distribution system may receive data items respectively from one or more of the client computing devices. In the distribution system, each timestamp device may timestamp a data item with a timestamp value based on a time of a same electronic clock when the data item is received at the timestamp device. The distribution system may distribute the data items to one or more destination computing devices in chronological order, based on timestamp values respectively of the data items.

[0003] In one example, in an electronic asset trading system, many trade orders for assets, such as financial assets including U.S. treasuries, or intangible assets, including cryptocurrencies, from multiple computing devices of respective clients, may be received over a short period of time, such as within less than one microsecond, by a computing platform that facilitates submission of the trade orders to one or more trading exchanges. The computing platform strives to provide fairness in distribution of the trade orders to the trading exchanges, based on the chronological times the trade orders are received from the computing devices of clients. The computing platform often includes multiple timestamp devices, each having multiple input ports at which to receive a message containing a trade order. The timestamp device timestamps the trade order with a timestamp value corresponding to a time of an electronic clock of the computing platform when24-2514WO_Appthe message is received at the timestamp device. The timestamp devices, for example, may he part of a programmable integrated circuit, such as a field programmable gate array (“FPGA”). Ensuring that timestamping is synchronized among the input ports of a timestamp device, which determines timestamp values based on a time of a same electronic clock, within a desired range, for example, at about a nanosecond range or less, such that fairness in distribution of trade orders in high frequency trading applications may be achieved, however, presents technical difficulties. In addition, ensuring that synchronized timestamping among timestamp devices of a computing platform that distributes the trade order is maintained during operation of the computing platform without disrupting distribution operations also presents technical difficulties. For example, verifying synchronization by temporarily removing or disabling a timestamp device or other components of the computing platform is undesirable, as distribution system operation or performance capabilities may be adversely affected.

[0004] Accordingly, there exists a need for system, apparatus and method that may verify synchronization of timestamping at input ports of a timestamp device, and verify synchronization of timestamping at input ports respectively of timestamp devices of a computing platform while the computing platform is operable to receive data items and distribute the data items in chronological order based on timestamp values respectively determined for the data items by the timestamp devices.SUMMARY

[0005] In accordance with an aspect of the present disclosure, a system may be for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first field programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, in which the system may include: at least one processor; a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor; in which the at least one processor is configured to cause the second FPGA to, for a given first programmable integrated circuit of the first programmable integrated circuits: generate a first test message and transmit the first test message simultaneously, over a plurality of substantially same length first data paths, to a plurality of input ports of a given first timestamp circuit of the given first programmable integrated circuit, in which the given first timestamp circuit is configured to timestamp the first test message at each24-2514WO_Appof the input ports of the given first timestamp circuit with a first timestamp value corresponding to a current time of an electronic clock when the first test message is received at the input port of the given first timestamp circuit; in which the at least one processor is configured, for the given first programmable integrated circuit: receive, from the given first programmable integrated circuit, the first timestamp values of the first test messages received respectively at the input ports of the given first timestamp circuit; determine whether any of the first timestamp values is outside a first range; determine an offset for each first timestamp value determined to be outside the first range; and for a given input port of the given first timestamp circuit corresponding to a given first timestamp value determined to be outside the first range, store, in a memory, a given offset for the given input port; in which the at least one processor is configured to, while the distribution system is in an operating state in which input ports respectively of the first programmable integrated circuits are operable to receive event messages including event data for distribution by the distribution system based on chronological order of receipt of the event messages at the distribution system, cause the second FPGA to: generate a second test message and transmit the second test message simultaneously, over a plurality of substantially same length second data paths, to a second input port of the first timestamp circuit of each of the first programmable integrated circuits, in which the first timestamp circuit of each first programmable integrated circuits is configured to timestamp the second test message with a second timestamp value corresponding to a current time of a second electronic clock of the distribution system when the second test message is received at the second input port of the first timestamp circuit; in which the at least one processor is configured to, for each of the first programmable integrated circuits: receive, from the first programmable integrated circuit, a given second timestamp value corresponding to a given second test message received at the first programmable integrated circuit; adjust the given second timestamp value with an offset indicated in the memory for a given second input port at which the given second test message is received, to determine a given adjusted second timestamp value; and determine a second range of adjusted second timestamp values.

[0006] In accordance with an aspect of the present disclosure, a system may be for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first field programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, in which the system may include: at least one processor; a second programmable integrated circuit including a second FPGA and24-2514WO_Appcommunicatively coupled to the at least one processor; in which the at least one processor is configured to: cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to input ports respectively of the first programmable integrated circuits, a test message, in which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the input port of the first timestamp circuit with a timestamp value corresponding to a current time of an electronic clock of the distribution system when the test message is received at the input port of the first timestamp circuit; receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the input port of the first timestamp circuit of the first programmable integrated circuit; retrieve from a memory an offset corresponding to a given input port of a given first timestamp circuit of a given first programmable integrated circuit of the first programmable integrated circuits; adjust a given timestamp value corresponding to the given input port of the given first timestamp circuit of the given first programmable integrated circuit with the offset corresponding thereto, to determine a given adjusted timestamp value; and determine a range of the given adjusted timestamp values.

[0007] In accordance with an aspect of the present disclosure, a system may include: a plurality of first programmable integrated circuits, in which each first programmable integrated circuit is communicatively coupled to at least one first transmission control protocol (TCP) client circuit, in which each first TCP client circuit is configured to receive order data from a market participant, in which each first programmable integrated circuit includes a first field programmable gate array (FPGA) and in which each first FPGA includes a first timestamp circuit; at least one processor; a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor; in which each first programmable integrated circuit is configured to: receive, from at least one first TCP client circuit, a stream of data packets representative of at least one order; for each order of the at least one order, identify a last data byte (LDB) data packet in the stream of data packets containing a LDB of a plurality of data bytes forming the order; determine, by the first timestamp circuit of the first FPGA of the first programmable integrated circuit, a timestamp value for the order based on a time indicated by an electronic clock of the system at a time the LDB data packet of the order is received at the first programmable integrated circuit; and generate at least one order message for the order, in which the at least one order message indicates the timestamp value for the order; in which the at least one processor is24-2514WO_Appconfigured to: for each order for which at least one order message is generated: determine an updated timestamp value based on the timestamp value for the order, a network offset associated with a given first TCP client circuit of the first TCP client circuits from which the order is received at the first programmable integrated circuit and an offset corresponding to an input port of the first timestamp circuit at which the order is received, in which the network offset corresponds to a network path extending from the given first TCP circuit client to the first programmable integrated circuit, and in which each first TCP client circuit has a network offset corresponding to a network path extending from each first TCP client circuit to a respective first programmable integrated circuit; and generate an updated order message indicating the updated timestamp value; in which the at least one processor is configured to, while any first programmable integrated circuit is operating to receive, from at least one of the first TCP client circuits, a given stream of data packets representative of at least one order: cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to test input ports respectively of the first programmable integrated circuits, a test message, in which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the test input port of the first timestamp circuit with a timestamp value corresponding to a current time of the electronic clock of the system when the test message is received at the test input port of the first timestamp circuit; receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the test input port of the first timestamp circuit of the first programmable integrated circuit; retrieve from a memory a given offset corresponding to a given test input port of a given first timestamp circuit of each given first programmable integrated circuit of the first programmable integrated circuits; adjust a given timestamp value corresponding to the given test input port of the given first timestamp circuit of the each given first programmable integrated circuit with the given offset corresponding thereto, to determine a given adjusted timestamp value; and determine a range of the given adjusted timestamp values.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a block diagram of an exemplary computing apparatus for verifying timestamping synchronization, according to the present disclosure.24-2514WO_App

[0009] FIG. 2 is a block diagram of an exemplary distribution system including timestamp circuits, according to the present disclosure.

[0010] FIG. 3 is a block diagram of an exemplary computing apparatus for verifying timestamping synchronization at a timestamp circuit, according to the present disclosure.

[0011] FIG. 4 is a block diagram of an exemplary distribution system communicatively coupled to a computing apparatus for verifying timestamping synchronization, according to the present disclosure.

[0012] FIG. 5 illustrates exemplary selected elements of the distribution system of FIG. 4, according to the present disclosure.

[0013] FIG. 6 illustrates an exemplary embodiment of an FPGA of the distribution system of FIG. 4, according to the present disclosure.

[0014] FIG. 7 illustrates a high level flow diagram of an exemplary method for verifying timestamping synchronization at a timestamp circuit, according to the present disclosure.

[0015] FIGs. 8A and 8B illustrate a high level flow diagram of an exemplary method for verifying timestamping synchronization of timestamp circuits of a distribution system, according to the present disclosure.DETAILED DESCRIPTION

[0016] The technology of the present disclosure relates to, by way of example, a computer and networking architecture that may control verifying synchronization of timestamping of data items, which are received from a plurality of client computing devices, by timestamp circuits of a computing platform that timestamp the data items with timestamp values based on a time of an electronic clock of the computing platform when the data items are received, to ensure fairness in distribution of the data items to one or more destination computing devices chronologically based on the timestamp values, and that avoids interfering with operation of the computing platform to distribute the data items.

[0017] In one embodiment, a computing apparatus may include an architecture containing at least one programmable hardware device, for example, a re -programmable logic device such as a field programmable gate array (FPGA), and at least one processor, where the at least one processor causes the FPGA to transmit a first test message simultaneously, over a plurality of substantially same length data paths, to a plurality of input ports of a timestamp circuit of another programmable24-2514WO_Appintegrated circuit, such as a second FPGA. The timestamp circuit may determine timestamp values for the first test messages based on a time of an electronic clock of the computing apparatus or the second FPGA when the first test messages are received respectively at the input ports, and provide, to a switch of the computing apparatus, the timestamp values respectively with indications of the input ports at which the first test messages corresponding thereto are received. The at least one processor may determine whether any of the timestamp values received at the switch is outside a first range, and determine an offset for a given input port of the input ports having a timestamp value that is outside the first range, such that a sum of the timestamp value and the offset is within the first range.

[0018] In one embodiment, a computing apparatus may include at least one programmable hardware device, for example, a re-programmable logic device such as a first FPGA, at least one processor, a tap aggregator and a switch. The computing apparatus may be coupled to a distribution system, such as for distributing trade orders. The distribution system may include an electronic clock; a controller; a plurality of second programmable hardware devices, for example, re-programmable logic devices such as second FPGAs, each including a timestamp circuit; and server circuits, such as Transport Control Protocol (TCP) server circuits. The distribution system may be configured to distribute, from the TCP server circuits, data items received from client devices, such as data items in data packets of a TCP stream from a TCP client circuit, which is of or associated with a client device or alternatively part of the distribution system, to destination computing devices, such as exchange gateways respectively of trading exchanges, in chronological order based on timestamp values determined for the data items which correspond to a time of the electronic clock when the data items are received at input ports respectively of the timestamp circuits. The computing apparatus may be configured to, while the distribution system is operable or operates to distribute data items received from the client devices to destination computing devices, cause the first FPGA to transmit a test message simultaneously, over a plurality of substantially same length data paths, to test input ports respectively of the timestamp circuits. The timestamp circuits may determine timestamp values respectively for the test messages which correspond to a time of the electronic clock when the test messages are received respectively at the test input ports, and provide to the switch the timestamp values with indications of an identifier of the test message and the test input ports of the timestamp circuits to which the timestamp values correspond. The computing apparatus may adjust a timestamp value for a test input port of a24-2514WO_Apptimestamp circuit with an offset that corresponds to the test input port retrieved from a memory and is based on a synchronization verification performed for the timestamp circuit (“sync offset”), to determine an adjusted timestamp value. In addition, the computing apparatus may determine whether a range of adjusted timestamp values respectively for timestamp circuits of the distribution system exceeds a predetermined range, and generate and output alert information indicating a failed operational state for a timestamp circuit of any second FPGA which has an adjusted timestamp value that may cause the range of adjusted timestamp values to exceed the predetermined range.

[0019] In one embodiment, the test message may be in a payload of a data packet of a TCP stream of data packets, and a timestamp circuit may insert a timestamp value in payload of a data packet containing test data of the test message at the application layer, i.e., Layer 7 of the Open Systems Interconnection (OSI) or Layer 4 of the Transport Control Protocol / Internet Protocol (TCP / IP) Model, advantageously providing increased accuracy in timestamping an order of a data packet.

[0020] In one embodiment, the distribution system may include taps, such as an optical or electrical splitter, respectively at test input ports respectively of the timestamp circuits of the second FPGAs, and the taps may be coupled respectively to same length data paths extending from the taps to the aggregator of the computing apparatus. The aggregator may receive, from a tap associated with a test input port of a timestamp circuit of each second FPGA of the distribution system, a portion of a test message received at the tap. The aggregator may determine whether a time that a test message portion from a tap of a second FPGA is received at the aggregator is within a predetermined time range. When a time of a test message portion is determined to be outside the predetermined time range, the computing apparatus may generate and output alert information indicating that a second FPGA associated with the tap from which the test message portion is received, has not received a same test message at a same or substantially the same time that other second FPGAs have received the same test message, such that verification of synchronization of timestamp circuits of the distribution system according with the present disclosure is not available.

[0021] In one embodiment, the distribution system may be configured to operate in a state in which: messages containing data items may be received from client computing devices as data packets at input ports respectively of timestamp circuits of second FPGAs; the timestamp circuits determine timestamp values for the data items in the data packets based on a time of the electronic clock of the distribution system when the messages containing the data items are received; and the24-2514WO_Appserver circuits distribute output messages containing the data packets of data items to destination computing devices, such as one or more trading exchanges, chronologically based on the timestamp values respectively of the data items determined at the timestamp circuits and offsets respectively of the input ports respectively of the timestamp circuits at which the data items are received.

[0022] In one exemplary embodiment, a data item as a trade order may be in payload of a data packet of one or more TCP segments of a TCP stream of data packets from a TCP client circuit, and the distribution system may receive a TCP data packet stream at an input port of a timestamp circuit of an FPGA of the distribution system to which the TCP client circuit is communicatively coupled. In addition, a server circuit of the distribution system may be a TCP server circuit that distributes a trade order to a trading exchange, based on the timestamp value for the trade order, and multiple output messages containing a same trade order may be simultaneously transmitted by fanout from respective TCP server circuits to trading exchanges, such as or similarly as described in METHOD, APPARATUS AND SYSTEM FOR TIME STAMPING AND SEQUENCING DATA ITEMS, US 17 / 818,773 filed August 10, 2022 (773 application), incorporated by reference herein.

[0023] It is to be understood that the features in accordance with the present disclosure may be applied to verifying synchronization of timestamping by timestamp circuits of a distribution system in applications requiring chronological distribution of data items from respective client computing devices to one or more destination computing devices based on timestamp values applied by timestamp circuits, such as, for example, real-time streaming of video or audio data, such as in interactive multi-player games, or event data from sensors, such as sensors in an internet of things (“IOT”) network including health device sensors, traffic device sensors, etc.

[0024] The present disclosure may be implemented using a combination of computer hardware and computer software to form a specialized machine capable of performing operations. Embodiments of the present disclosure may be performed utilizing a combination of central processing units (CPUs), physical memory, physical storage, electronic communication ports, electronic communication lines and other computer hardware. The computer software may include at least a computer operating system and specialized computer processes described herein.

[0025] In the present disclosure, certain specific details are set forth to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will24-2514WO_Apprecognize that embodiments may be practiced without one or more of these specific details, or with other methods, components, etc. In other instances, well-known structures associated with an electronic trading system have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the embodiments.

[0026] The aspects, features and advantages of the present disclosure will be appreciated when considered with reference to the following description of examples and accompanying drawings. In describing the exemplary embodiments of the disclosure illustrated in the drawings, specific terminology will be used for the sake of clarity. However, the disclosure is not intended to be limited to the specific terms used.Example Systems and Apparatuses

[0027] FIG. 1 illustrates a block diagram of an exemplary computing apparatus 10, in accordance with the present disclosure. The apparatus 10 may be communicatively coupled to input ports and output ports respectively of timestamp circuits of a distribution system, such as exemplary distribution system 100 shown in FIG. 2. The distribution system 100 may be configured to receive data items from sources, include timestamp circuits that determine timestamp values for the data items that correspond to times respectively when the data items are received at the distribution system, and distribute the data items to computing devices as recipients of the data items, in chronological order based on the timestamp values respectively. For purposes of illustrating the features of the disclosure, and as described in detail herein in the text accompanying the description of FIGs. 2 and 4-6, a distribution system, such as the distribution system 100 and a distribution system 500, may be an electronic asset trading system, such as described in the 773 application, that timestamps electronic trading orders received from client devices with timestamp values corresponding respectively to times of an electronic clock at times of receipt, and distributes the trading orders in chronological order based on the timestamp values to computing devices, such as exchange gateways of computing platforms as asset trading exchanges that are indicated as destinations for the trading orders.

[0028] In some embodiments, advantageously according to the present disclosure, the apparatus 10 may perform processing functions that control: (i) verifying synchronization of timestamping among input ports of a timestamp circuit (“intra-synchronization”); and (ii) while a distribution system is in an operating state of receiving data items and distributing the data items in chronological order based on timestamp values determined for respective data items by the24-2514WO_Apptimestamp circuits, verifying synchronization of timestamping among timestamp circuits respectively of FPGAs of the distribution system (“intcr-synchronization”), to facilitate receipt, processing and distribution of the data items with low latency, high reliability that the distribution is chronological in accordance with times of receipt based on timestamp values determined therefor and minimized usage of network bandwidth, processing resources and memory resources, as described in detail below.

[0029] Referring to FIG. 1, the computing apparatus 10 may be in the form of a computing device that includes one or more processors 12, one or more memory 14 and other components commonly found in computing devices.

[0030] The memory 14 may store information accessible by the one or more processors 12, including instructions 16 that may be executed by the one or more processors 12.

[0031] The one or more processors 12 may include an architecture configured to include a programmable hardware device, such as a field programmable field array (“FPGA”), an application specific integrated circuit (“ASIC”) or system on chip (“SoCs”). In one embodiment, the architecture may be hardwired on a substrate. In one embodiment, the one or more processors 12 may be any type of processor, such as a CPUs from INTEL, AMD, and APPLE.

[0032] Memory 14 may also include data 18 that can be stored, manipulated or retrieved by the processor. The data 18 may also be used for executing the instructions 16 and / or for performing other functions. The memory 14 may be any type of non-transitory media readable by the one or more processors 12, such as a hard-drive, solid state hard-drive, memory card, ROM, RAM, DVD, CD-ROM, write-capable, read-only memories, etc.

[0033] The instructions 16 may be any set of instructions capable of being read and executed by the one or more processors 12. The instructions may be stored in a location separate from the computing device, such as in a network attached storage drive, or locally at the computing device. The terms “instructions,” “functions,” “application,” “steps,” and “programs” may be used interchangeably herein. The instructions residing in a non-transitory memory may comprise any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by processor 12. In this regard, the terms “instructions,” “scripts,” or “modules” may be used interchangeably herein. The computer executable instructions may be stored in any computer language or format, such as in object code or modules of source code. Furthermore, it is24-2514WO_Appunderstood that the instructions may be implemented in the form of hardware, software, or a combination of hardware and software and that the examples herein arc merely illustrative.

[0034] Data 18 may be stored, retrieved and / or modified by the one or more processors 12 in accordance with the instructions 16. Such data may be stored in one or more formats or structures, such as in a relational or non-relational database, in a SQL database, as a table having many different fields and records, XLS, TXT, or XML documents. The data may also be formatted in any computing device-readable format. In some embodiments the data may be encrypted.

[0035] The apparatus 10 may include a communication device 20 configured to provide wired or wireless communication capabilities.

[0036] The processor 12 may include a test message source 30 configured to generate a test message. In one embodiment, the test message source 30 may be configured as an FPGA that includes a fanout circuit, and to generate a test message in a data packet and simultaneously transmit replicas of the test message in data packets as a fanout from the fanout circuit. In one embodiment, a test message may be in the form of test data in payload of a data packet of a TCP segment of one or more data packets of a TCP stream of data packets, and the test data may indicate an identifier of the test data.

[0037] The apparatus 10 may include a network interface 31 configured to transmit data packets of respective test messages on an optical or electrical signal. The network interface 31 may include an ethernet interface having multiple output ports, and output-sublayers which are communicatively coupled respectively to the output ports. The ethernet interface may be a coaxial, twisted pair or fiber-optic physical media interface. Each output-sublayer may include a physical medium attachment sublayer ("PMA") and a physical coding sublayer ("PCS") that may interface a medium access control sublayer ("MAC") with the PMA. Each output- sublayer may correspond to a physical layer and data link layer within the processor 12 that may be configured to route data packets, such as TCP data packets of a test message from the test message source 30, via an output port of the ethernet interface, to a recipient computing device.

[0038] The apparatus 10 may further include a switch 32 and a tap aggregator 34, each coupled to the processor 12. The switch 32 may be an electronic switching mechanism having multiple switch input ports. Each switch input port may be configured to receive, from an output port of a timestamp circuit, a data packet containing a timestamp value and indicating a test message to which the data packet corresponds and an input port of a timestamp circuit at which the test24-2514WO_Appmessage was received. In addition, the switch 32 may be configured to extract, from each data packet received, and provide to the processor 12, the timestamp value and indications of identifiers of the test message, input port and timestamp circuit corresponding to the timestamp value.

[0039] The tap aggregator 34 may be a tap aggregation device configured to receive electrical or optical signals carrying data packets containing test messages, which were routed from the computing apparatus 10 to a distribution system, and determine differences in, or a range of differences of, times of receipt of the signals at the aggregator 34. In addition, the tap aggregator 34 may be configured to identify a timestamp circuit of the distribution system from which a signal containing a particular test message was not received by the aggregator 34. Further, the tap aggregator 34 may be configured to generate and provide as an output, such as on a display, alert information indicating differences or a range of differences of times of receipt, and an identifier of a timestamp circuit or FPGA of the distribution system from which a signal with a particular test message was not received.

[0040] In one embodiment, the apparatus 10 may include an electronic clock 36 configured to time, for example, at nanosecond or smaller intervals. The clock 36 may be configured to supply to a timestamp circuit of an FPGA a clock signal indicative of a current time of the clock 36, such that a clock of or associated with the timestamp circuit may use the clock signal to determine timestamp values of data packets received respectively at input ports of the timestamp circuit of an FPGA. As discussed below in the text accompanying the description of FIG. 3, the clock signal from the clock of a computing apparatus of the present disclosure may be used by a timestamp circuit of an FPGA to determine timestamp values.

[0041] In one embodiment, the apparatus 10 or an apparatus similar in construction and operation to the apparatus 10, and optionally including the electronic clock 36, may be coupled to or at least partially included in a distribution system, such as the distribution system 100 or the distribution system 500, which are discussed in detail in the text accompanying the description of FIGs. 2 and 4, respectively, such that some or all functionalities of the apparatus 10 as described below may be implemented in components of the distribution system 100 or the distribution system 500.

[0042] Referring to FIG. 1, the apparatus 10 may be communicably interconnected with an input port of a timestamp circuit over optical or electrical cables extending from the network interface 31, and with an output port of a timestamp circuit over an electrical cable or a data bus extending from the switch 32. As described herein, path lengths of communication paths extending from24-2514WO_Appoutput ports of a test message source 30 and coupled to input ports respectively of a timestamp circuit(s), may be the same or substantially the same, to provide that data packets containing a same test message simultaneously transmitted from the test message source 30 are received simultaneously or substantially simultaneously at input ports of a same timestamp circuit or respective timestamp circuits. In addition, path lengths of communication paths extending from output ports of taps associated with respective timestamp circuits of FPGAs to input ports of the tap aggregator 34 may be the same or substantially the same, to permit a determination that data packets containing a same test message simultaneously transmitted from the test message source 30 are received simultaneously or substantially simultaneously at input ports respectively of timestamp circuits, such that inter-synchronization verification may be reliably performed.

[0043] FIG. 1 illustrates the components of the apparatus 10 as being single components, however, the components may comprise multiple programmable hardware devices, such as processors, computers, computing devices, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage media located in housings different from that of the apparatus 10. Accordingly, references to a programmable hardware device, processor, computer, computing device, or memory herein will be understood to include references to a collection of processors, computers, computing devices, or memories that may or may not operate in parallel. Further, although some functions described below are indicated as taking place on a single computing device having a single processor, various aspects of the subject matter described herein may be implemented by a plurality of computing devices in series or in parallel. For example, in one embodiment, functions performed by the apparatus 10 as described below may at least be partially performed at another computing apparatus having the same or similar components as the apparatus 10. In one embodiment, functions described herein as performed by the apparatus 10 may be distributed among one or more computing devices (servers) that operate as a cloud system.

[0044] Although only a single computing apparatus 10 (computer) is depicted herein it should be appreciated that a computing apparatus in accordance with the present disclosure may include additional interconnected computers and programmable hardware devices. It should further be appreciated that apparatus 10 may be an individual node in a network containing a larger number of computers.24-2514WO_App

[0045] In one embodiment, the apparatus 10 may include all the components normally used in connection with a computer. For example, apparatus 10 may have a keyboard and mouse and / or various other types of input devices such as pen-inputs, joysticks, buttons, touch screens, etc., as well as a display, which could include, for instance, a CRT, LCD, plasma screen monitor, TV, projector, etc.

[0046] Referring to FIG. 2, a distribution system 100 in accordance with the present disclosure may be communicatively coupled over a communication network 158 to a plurality of client devices 152 as data sources, such as computing devices at which data items, for example, electronic trading orders, may be entered; and over a communication network 160 to a plurality of computing devices 154 as data recipients of the data items, such as exchange gateways of computing platforms as asset trading exchanges that may be the desired destinations of the trading orders. In some embodiments, the distribution system 100 may perform processing functions that control transmitting data items from a plurality of client devices 152 to one or more computing devices 154 chronologically in accordance with times of receipt at the distribution system 100 based on timestamp values determined by timestamp circuits of the distribution system 100, where the timestamp values of the data items correspond to times of completion of transmission of the data packets with the data items from the client devices 152, or reception of data packets containing the data items respectively at the timestamp circuits.

[0047] Referring to FIG. 2, the distribution system 100 may be in the form of a computing device that includes one or more processors 112, one or more memory 114 and other components commonly found in computing devices. In one embodiment, the one or more processors 112 may include or be configured to operate as one or more servers.

[0048] The memory 114 may store information accessible by the one or more processors 112, including instructions 116 that may be executed by the one or more processors 112.

[0049] The one or more processors 112 may include an architecture configured to include a programmable hardware device, such as an FPGA, an ASIC or SoC. In one embodiment, the architecture may be hardwired on a substrate. In one embodiment, the one or more processors 112 may be any type of processor, such as a CPUs from INTEL, AMD, and APPLE.

[0050] Memory 114 may also include data 118 that can be stored, manipulated or retrieved by the processor. The data 118 may also be used for executing the instructions 6 and / or for performing other functions. The memory 114 may be any type of non-transitory media readable by the one or24-2514WO_Appmore processors, such as a hard-drive, solid state hard-drive, memory card, ROM, RAM, DVD, CD-ROM, writc-capablc, read-only memories, etc.

[0051] The instructions 116 may be any set of instructions capable of being read and executed by the one or more processors 112. The instructions may be stored in a location separate from the computing device, such as in a network attached storage drive, or locally at the computing device. The terms “instructions,” “functions,” “application,” “steps,” and “programs” may be used interchangeably herein. The instructions residing in a non-transitory memory may comprise any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by processor 112. In this regard, the terms “instructions,” “scripts,” or “modules” may be used interchangeably herein. The computer executable instructions may be stored in any computer language or format, such as in object code or modules of source code. Furthermore, it is understood that the instructions may be implemented in the form of hardware, software, or a combination of hardware and software and that the examples herein are merely illustrative.

[0052] Data 118 may be stored, retrieved and / or modified by the one or more processors 112 in accordance with the instructions 116. Such data may be stored in one or more formats or structures, such as in a relational or non-relational database, in a SQL database, as a table having many different fields and records, XLS, TXT, or XML documents. The data may also be formatted in any computing device-readable format. In some embodiments the data may be encrypted.

[0053] The system 100 may include a communication device 120 configured to provide wired or wireless communication capabilities.

[0054] The system 100 may further include an electronic clock 130 and a plurality of reprogrammable hardware circuits 132, such as FPGAs.

[0055] In one embodiment, the clock 130 may be configured as a master clock that supplies a clock signal to multiple destinations, such as timestamp circuits, indicative of a current time of the clock 130, and ensures synchronization of clocks respectively at the multiple destinations, such as by implementation of the Precision Time Protocol (PTP). As discussed below in the text accompanying the description of FIGs. 4-6, the clock signal from the clock 130 may be used for timestamping by timestamp circuits of respective FPGAs.

[0056] Each FPGA 132 may include a timestamp circuit 134 configured to determine a timestamp value for a message contained in a data packet(s) received at an input port thereof, based on a time of the clock 130 when the data packet(s) is received at the timestamp circuit. In one embodiment,24-2514WO_Appthe timestamp circuit 134 may include a local clock that is synchronized with a time of the clock 130 based on implementation of PTP.

[0057] In addition, in one embodiment, the processor 112, or an FPGA 132 or portion thereof, may be configured to include or operate as a: sequencer (not shown) that sorts data items expressed as event data in data packets of received event messages for routing based on timestamp values respectively determined for the data items: a fanout circuit (not shown) that replicates a data packet with a data item to create replica data packets with the same data item: and server circuits (not shown) that transmit data packets respectively containing the data items to destination computing devices based on routing information for the data items, such as indicated in the data packets corresponding to the data items or a memory of the distribution system 100, in chronological order based on timestamp values determined thereof. For example, the sequencing and routing by fanout of data packets containing trade orders, in accordance with timestamp values determined therefor, may be performed at an FPGA of the system 100, similarly as described in the 773 application.

[0058] Referring to FIG. 2, the communication network 158 may be a communication network having predetermined network characteristics, such as bandwidth, communication protocol, communication paths and communication path lengths, and include a local area network (“LAN”), wide area network (“WAN”), virtual private network, local Ethernet network, a private network using a proprietary communication protocol proprietary or like network. The communication network 160 may be a local area network (“LAN”), a wide area network (“WAN”), or the Internet, etc. The communication network 160 and intervening nodes thereof may use various protocols including virtual private networks, local Ethernet networks, private networks using communication protocols proprietary to one or more companies, cellular and wireless networks, HTTP, and various combinations of the foregoing. In addition, the networks 158 and 160 may utilize a variety of networking protocols now available or later developed including, but not limited to, TCP / IP based networking protocols.

[0059] FIG. 2 illustrates the components of the system 100 as being single components, however, the components may comprise multiple programmable hardware devices such as FPGAs, processors, computers, computing devices, or memories that may or may not be stored within the same physical housing. For example, the memory may be a hard drive or other storage media located in housings different from that of the system 100. Accordingly, references to a programmable hardware device, processor, computer, computing device, or memory herein will24-2514WO_Appbe understood to include references to a collection of processors, computers, computing devices, or memories that may or may not operate in parallel. Further, although some functions described below are indicated as taking place on a single computing device having a single processor, various aspects of the subject matter described herein may be implemented by a plurality of computing devices in series or in parallel. For example, in one embodiment, functions performed by the system 100 as described below may at least be partially performed at another computing apparatus having the same or similar components as the system 100. In one embodiment, functions described herein as performed by the system 100 may be distributed among one or more computing devices (servers) that operate as a cloud system.

[0060] It should be appreciated that a distribution system 100 in accordance with the present disclosure may include additional interconnected computers and reprogrammable hardware devices, such as FPGAs. It should further be appreciated that system 100 may be an individual node in a network containing a larger number of computers.

[0061] In one embodiment, the system 100 may include all the components normally used in connection with a computer. For example, system 100 may have a keyboard and mouse and / or various other types of input devices such as pen-inputs, joysticks, buttons, touch screens, etc., as well as a display, which could include, for instance, a CRT, LCD, plasma screen monitor, TV, projector, etc.

[0062] Referring to FIGs. 1 and 2, in some embodiments, the apparatus 10 may be communicatively connected with the system 100, or form at least a portion of the system 100, and configured to implement specific functions and operations in accordance with the present disclosure. In some embodiments, the system 100 may include at least a portion of the apparatus 10 and be programmed with programs to perform some or all of the functions and operations described herein.

[0063] For ease of reference and convenience, the disclosure herein that a component of an apparatus or system may perform a function or operation, is a disclosure that a processor or circuitry of one or more components of the apparatus or system may perform or control the performance of the function or operation.

[0064] In one embodiment, referring to FIG. 3, the apparatus 10 may be configured as an apparatus 210 which is communicatively coupled to an FPGA 260, and which includes the same or similar components as in the apparatus 10 except for the tap aggregator 34.24-2514WO_App

[0065] Referring to FIG. 3, the FPGA 260 may include a timestamp circuit 270 and a processor 267, and include or be coupled to a network interface 271. The network interface 271 may include input ports 268 which are communicatively coupled within the FPGA 260 respectively with input ports 273 of the timestamp circuit 270.

[0066] The apparatus 210 may include a controller 212 including a processor 214, a memory 216 and a communication device 218. The memory 216 may be configured to store instructions to implement specific functions and operations, and data related to verifying synchronization of timestamping by a timestamp circuit, in accordance with the present disclosure. In one embodiment, each component of the apparatus 210 may include a processor and memory including instructions that implement functions of the respective component, as described below.

[0067] The apparatus 210 may include a test message source configured in or as an FPGA 222 (“FPGA source”), a switch 224, data paths 225, an electronic clock 230 and a network interface 227. The FPGA source 222 may include output ports 223 communicatively coupled over the data paths 225 respectively to input ports 219 of the network interface 227.

[0068] The FPGA source 222 may be configured to generate a data packet containing a test message in pay load of the data packet (“test packet”). In addition, the FPGA source 222 may include a fanout circuit (not shown) configured to replicate a test packet and route replica test packets simultaneously over respective data paths 225, which are of a same or substantially the same path length, to the input ports 219 of the network interface 227.

[0069] The network interface 227 may have a same or similar construction as the network interface 31 of the apparatus 10, and generate respective TCP steams of data packets for a same test message. The network interface 227 may include output ports 221 communicatively coupled over respective electrical or optical cables 261 extending to the input ports 268 of the network interface 271. The apparatus 210 may be configured such that time durations for routing data packets from input ports 219 to respective output ports 221 of the network interface 227 are substantially the same.

[0070] The cables 261 may be of a same or substantially the same path length.

[0071] The network interface 271 may include an ethernet interface having the input ports 268, and input-sublayers which are communicatively coupled to the input ports 268 and the input ports 273 of the timestamp circuit 270. The ethernet interface may be a coaxial, twisted pair or fiberoptic physical media interface. Each input-sublayer may include a physical medium attachment24-2514WO_Appsublayer ("PMA") and a physical coding sublayer ("PCS") that may interface a medium access control sublayer ("MAC") with the PMA. Each input- sublayer may correspond to a physical layer and data link layer, such as within the processor 267, that may be configured to receive streams of data packets as TCP segments containing the test data from the FPGA source 222, and provide test data of the respective streams to an input port 273 of the timestamp circuit 270.

[0072] In addition, FPGA 260 may be configured such that time durations for routing data packets from input ports 268 of the network interface 271 to respective input ports 273 of the timestamp circuit 270 are substantially the same. The cables 261 having the same or substantially the same path lengths, the data paths 225 having the same or substantially the same path lengths, and also the time durations for routing being the same or substantially the same in respective data paths in the network interfaces 227 and 271, may provide that test packets simultaneously transmitted from the FPGA source 222 may be received at a same or substantially the same time at the input ports 273 of the timestamp circuit 270.

[0073] The switch 224 may include input ports 280 coupled over communication paths 282, such as data buses, to output ports 272 of the timestamp circuit 270.

[0074] The clock 230 may be configured as an electronic clock similar to the clock 36 as described above.

[0075] The timestamp circuit 270 may be configured similarly as the timestamp circuit 134 described above, and be coupled to the clock 230 to receive the clocking signal therefrom. In addition, the timestamp circuit 270 may be configured to route test packets received respectively at input ports 273 to respective predetermined output ports 272. For example, the input ports 273 may include the input ports 273 A, 273B and 273C and test packets received at the input ports 273A, 273B and 273C are routed within the timestamp circuit 270 to output ports 272A, 272B and 272C of the output ports 272, respectively. The timestamp circuit 270 may be configured to add a timestamp value to a beginning of payload of a TCP test packet containing a test message received at an input port 273. The timestamp value may be based on and correspond to a current time of the clock 230, as indicated by the clocking signal supplied therefrom, when the test packet is received at the input port 273, for example, when a last data byte of the test packet is completely received at the FPGA 260. The FPGA 260 may be configured to generate an output test packet including the test message with the timestamp value determined for the test message in payload of the output test packet. Further, the timestamp circuit 270 may be configured to transmit output24-2514WO_Apptest packets from the output ports 272 respectively over communication paths 282 to the input ports 280 of the switch 224.

[0076] The switch 224 may be configured to receive output test packets from the timestamp circuit 270 respectively at input ports 280, extract the timestamp values from the output test packets and identify the timestamp values as associated with a particular test message indicated in the test packet and respective input ports 268 of the timestamp circuit 270 that correspond to the output ports 223 of the FPGA source 222 from which the test packets were provided.

[0077] As discussed in detail below, the apparatus 210 may be configured to verify that timestamping at input ports of the timestamp circuit 270 based on a current time of an electronic clock 230, is synchronized at least to a level desirable for an expected use of the timestamp circuit 270, and determine an offset to apply to a timestamp value determined for an input port(s) (sync offset), to ensure that timestamping is synchronized among the input ports at least to a predetermined level associated with the expected use. An FPGA with a timestamp circuit 270 verified for timestamping synchronization (intra- synchronization) advantageously may be used in a distribution system to provide for transmission of data packets containing data items received at input ports of timestamp circuits of the distribution system in chronological order, based on time of receipt of data packets containing the data items as indicated by the timestamp values determined therefor by the timestamp circuits.

[0078] Referring to FIG. 4, an apparatus 410, which is similar in construction and operation to the apparatus 210, except that the clock 230 is omitted and a tap aggregator 434 having the same or similar construction and operation as the tap aggregator 34 of the apparatus 10 is included, may be communicatively coupled to a distribution system 500 that is operable to distribute data items of messages as data packets, in chronological order based on timestamp values determined for the messages. The distribution system 500 may be similar in construction and operation as the system 100 and include additional components having a construction and operation that enables the apparatus 410, while the distribution system is in an operating state of distributing data items of data packets received from client devices in chronological order based on the timestamp values determined for the data items at the timestamp circuits, to verify synchronization of timestamping by the timestamp circuits of the distribution system (inter-synchronization). The verification of synchronization of timestamping among the timestamp circuits may be performed by the apparatus 410, or alternatively by the distribution system 500 in an embodiment where the distribution24-2514WO_Appsystem 500 includes components having the same or similar construction and operation of the apparatus 410. For clarity and ease of description, the present disclosure is described herein with the apparatus 410 having components configured to verify timestamping synchronization of the timestamp circuits of the distribution system 500.

[0079] In one embodiment the system 500 may be configured for an electronic asset trading application, and the data item may be, for example, an order for purchase or sale of an asset, and include details of or relating to the order, such as price, quantity, identity of an order session corresponding to the order, identity of a user, as a participant in financial market trading, that enters the order at a client device, and identification information of a client device at which the order is entered, etc.

[0080] Referring to FIG. 4, the apparatus 410 may include a controller 412 including a processor 414, a memory 416 and a communication device 417, a network interface 418, an FPGA source 422, a tap aggregator 434, a switch 424 and data paths 425. The memory 416 may be configured to store instructions to implement specific functions and operations, and data related to verifying synchronization of timestamping among timestamp circuits, in accordance with the present disclosure. In one embodiment, each of the components of the apparatus 410 may include a processor and memory including instructions that implement functions of the respective component, as described below,

[0081] The FPGA source 422, the switch 424, data paths 425 and network interface 418 may have a same or similar construction and operation respectively as the FPGA source 222, switch 224, data paths 225 and network interface 227, respectively, described above. The network interface 418 may include input ports 419 coupled over data paths 425 respectively to output ports 423 of the FPGA source 422, and output ports 421 coupled to respective input ports 419.

[0082] The system 500 may include a controller 512 including at least one processor 514, a memory 516 and a communication device 518. The controller 512 may be communicatively coupled with client devices 604 external to the system 500 via a communication network (not shown), similarly as described above for the distribution system 100. In one embodiment, the controller 512 may be communicatively coupled and configured to control TCP client circuits 602 respectively at the client devices 604.

[0083] In one embodiment, the TCP client circuits 602 may be part of the system 500 and coupled respectively to client devices 604.24-2514WO_App

[0084] In addition, the system 500 may include an electronic clock 564, and multiple FPGAs 560 each including a timestamp circuit 570. The electronic clock 564 and the FPGAs 560 with respective timestamp circuits 570 may have a same or similar construction and operation respectively as the clock 130 and FPGAs 132 including respective timestamp circuits 134, as described above.

[0085] In one embodiment, the system 500 may include a gateway, sequencer and fanout circuit (GSF circuit) 576 and TCP server circuits 580. The GSF circuit 576 may be coupled to output ports 540 of the timestamp circuits 570 of respective FPGAs 560, and input ports 581 respectively of the TCP server circuits 580. In addition, the TCP server circuits 580 may have a same or similar construction and operation as TCP server circuits described in the 773 application. The TCP server circuits 580 may be communicatively coupled over communication cables 591, such as optical cables, via a communication network 592 with a plurality of computing devices, such as exchange gateways of a trading exchange or a matching engine or an exchange.

[0086] Referring to FIG. 6, the GSF circuit 576 may be configured in an FPGA, such as in an FPGA 560, and have a same or similar construction and operation as order entry gateway, sequencer and fanout circuit components as described in the 773 application. In one embodiment, the GSF circuit 576 may include an order entry gateway 577, a sequencer 578 and a fanout circuit 579 which are configured to control transmitting data packets containing data items, such as a trade order in data packets of a TCP segment, which are received from timestamp circuits 570, in chronological order from the server circuits 580, such as TCP server circuits, to selected destination gateways, such as exchange gateways, based on timestamp values determined by the timestamp circuits 570 for the data packets received at input ports of the timestamp circuits 570, such as described in the 773 application.

[0087] Referring to FIG. 5, an exemplary FPGA 560 of the system 500 may include a network interface 571 having input ports 530, and output ports 533 which are communicatively coupled to input ports 535 of the timestamp circuit 570. Output ports 540 of the timestamp circuit 570 may be communicatively coupled respectively to input ports 535. The input ports 530 may include a single test input port 530A and data item input ports 53OB, which are coupled respectively to test output port 533A and output ports 533B. The test output port 533A and output ports 533B are respectively coupled over data path 537A and data paths 537B to test input port 535A and input ports 535B, which in turn are coupled respectively to test output port 540A and output ports 540B.24-2514WO_AppThus, the test input port 530A and input ports 53OB are communicatively coupled to respective test output port 540A and output ports 540B on a one-to-one basis.

[0088] Referring to FIGs. 4 and 5, the system 500 may include a tap 550 at each test input port 53OA disposed in a signal path extending from an output port 423 of the FPGA source 422. The tap 550 may include an input port 552 and output ports 554 and 556. The output port 556 may be coupled directly to the test input port 53OA, and the output port 554 may be coupled over a communication cable 557 to an input port 435 of the tap aggregator 434 of the apparatus 410. In addition, each communication cable 557 between the tap aggregator 434 and a tap 550 may have a same or substantially the same length.

[0089] The tap 550, for example, may be a passive line optical or electrical tap that divides an input signal received at the input port 552 into two identical signal portions which may be at the same or different power levels, for example, each signal portion may be at one half the power level of the input signal. The tap 550 may route one signal portion from the output port 554 and the other signal portion from the output port 556.

[0090] Referring to FIGs. 4 and 5, the output ports 540A of respective FPGAs 560 may be communicatively coupled respectively over data paths 429 to input ports 427 of the switch 424. The TCP client circuits 602 respectively of the client devices 604 may be communicatively coupled over network paths 606 respectively with input ports 53OB. The network paths 606 may constitute a high-speed signal transmission medium, such as optical fiber, electrical cable or the like.

[0091] In addition, the single test input 530A may be communicatively coupled through the output port 556 and input port 552 of the tap 550 over a communication cable 590 to a respective output port 421 of the network interface 418 that outputs a test packet from a respective output port 423 of the FPGA source 422. For each timestamp circuit 570, the time duration of routing a test packet from an output port 423 to the input port 535A of the timestamp circuit 570 may be the same or substantially the same.

[0092] In one embodiment, a timestamp circuit 570 may be configured to determine a timestamp value for a data packet received at an input port 535 based on a time indicated by a clock signal supplied from the clock 564. In addition, a timestamp circuit 570 may be configured to associate the timestamp value with a data item, such as a trade order, in the data packet, for example, insert24-2514WO_Appthe timestamp value in payload of the data packet containing the data item where the data packet is part of a TCP data packet stream.

[0093] In an alternative embodiment, the timestamp circuit 570 may be configured to process User Datagram Protocol (UDP) data packets, and insert a timestamp in payload consistent with the UDP protocol.

[0094] In one embodiment, a timestamp circuit 570 may, at an input port 535, receive a message formed by a TCP stream of data packets containing a data item, such as a test message or trade order, in payload of one or more data packets thereof. The timestamp circuit 570 may insert a timestamp value at a beginning of the pay load of the test message at an application layer of the OSI. In one embodiment, the timestamp circuit 570 may insert a timestamp value in payload of a data packet in a TCP stream containing a first data byte of the data item, where the timestamp value corresponds to a time of receipt, at the input port 535, of a data packet of the stream of data packets containing a last data byte of the data item. For example, the data item may be a trade order, and the timestamp value may correspond to a time when an entirety of the data forming the trade order is received at the timestamp circuit.

[0095] In one embodiment, the FPGA source 422 may cause transmission of a test message, in a payload of a data packet(s) of a TCP segment of a stream of data packets, from the network interface 418, over a communication cable 590, to an input port 552 of the tap 550. The tap 550 may route a portion of the test message from the output port 556 to a test input port 530A of a network interface 571. The network interface 571 may extract a test packet(s) corresponding to the test message from the test message portion, and route the test packet(s) from a test output port 533A to a test input port 535A of a timestamp circuit 570 of an FPGA 560.

[0096] The timestamp circuit 570 may determine a timestamp value for the test message portion, based on a time of receipt of the data packet containing the last data byte of the test message at the test input port 535A. In addition, the timestamp circuit 570 may insert the timestamp value for the test message in payload of a data packet containing the first data byte of the test message. The timestamp circuit 570 may generate an output test message including the data packet(s) corresponding to the test message and containing the timestamp value, and route the output test message from test output port 540A, over a data path 429, such as an electrical cable or data bus, to the switch 424.24-2514WO_App

[0097] Further, the tap 550 may route the other test message portion from output port 554, over a cable 557, to an input port 435 of the tap aggregator 434.

[0098] In one embodiment, the memory 516 may store network offsets respectively for network paths 606 that may be used to compensate for different transmission durations, for example, different path lengths, for the network paths 606 respectively between the TCP client circuits 602 and timestamp circuits 570. The network offsets respectively of the network paths 606 may be stored in the memory to indicate correspondence to predetermined TCP client circuits 602.

[0099] The memory 516 may include a lookup table that indicates correspondence between a network offset, a TCP client circuit 602, a network path 606 and an input port 535B of a timestamp circuit 570 of an FPGA 560. For example, the memory 516 may indicate a network offset of 9 ns as corresponding to a predetermined TCP client circuit 602, and a predetermined network path 606 extending between the predetermined TCP client circuit 602 and a predetermined port 535B of a predetermined timestamp circuit 570.

[0100] The FPGAs 560 may route data packets corresponding to respective data items of trade orders with timestamp values inserted in payload to the GSF circuit 576.

[0101] The GSF circuit 576 may generate an output stream of order messages having a destination corresponding to a TCP server circuit 580, where the order messages are (i) arranged in an order based on timestamp values, any applicable network offset corresponding to the TCP server circuit 580 and the sync offset corresponding to an input port of the timestamp circuit at which the data packets for an order message is received, and (ii) routed to the TCP server circuit 580 in the order that the order messages are arranged in the output stream.

[0102] The TCP server circuits 580 may be configured to transmit simultaneously a given order as a respective TCP stream of output data packets including data for the given order represented in a given order message. In one embodiment, the TCP server circuits 580 may, for example, constitute a trading exchange gateway, and transmit the streams of output data packets representative of a respective order over communication paths 591 of a communication network 592 simultaneously respectively to computing devices, for example, as a trading exchange.

[0103] In one embodiment, referring to FIG. 6, the GSF circuit 576 may be configured as or be part of an FPGA, such as an FGPA 560, and include a gateway, such as an order entry gateway 577, a sequencer 578 and a fanout circuit 579. It is to be understood that the order entry gateway,24-2514WO_Appsequencer, and fanout circuit may be configured as a processor or part of an FPGA or like programmable hardware device, or a combination of the former and latter.

[0104] The order entry gateway 577 may be communicatively coupled over data paths 563 of the FPGA 560 respectively with output ports 540 of timestamp circuits 570. In this embodiment, a timestamp circuit 570 may, based on an input port 535B of the timestamp circuit 570 at which data for a particular order is received, determine from the memory 516 a sync offset to be applied to an output message containing data packets corresponding to the particular order. The FPGA 560 may be configured to generate an order message including all trading data of the particular order, with the timestamp value thereof in payload of a data packet of the order message, and associate with the order message metadata indicating a network offset corresponding to the order message and the sync offset corresponding thereto, and transmit the order message with the data packets of the trading data and associated metadata over a data path 563 to the order entry gateway 577 of the GSF circuit 576. In one embodiment, an FPGA 560 of the distribution system 500 may include data paths 563 extending between output ports 540 of a timestamp circuit 570 and the order entry gateway 577.

[0105] The order entry gateway 577 may control, for each order received from a timestamp circuit 570 in one or more order messages, determining an updated timestamp value. The updated timestamp value may be equal to a sum of: a network offset corresponding to the TCP client circuit from which the order is submitted, a timestamp value determined for the order at the timestamp circuit, and a sync offset corresponding to the input port of the timestamp circuit at which the order is received. In addition, the order entry gateway 577 may control generating an updated order message containing all data for the order and indicating an updated timestamp value, and route the updated order message to the sequencer 578.

[0106] The sequencer 578 may control chronologically ordering the orders respectively of the updated order messages in a sequencer queue, based on the updated timestamp values, and generating an output stream including the updated order messages of the respective orders in chronological order in accordance with a sequence of the orders in the sequencer queue. In one embodiment, the sequencer 578 may control delaying inclusion of an updated order message in the output stream until a hold delay that delays including an updated order message corresponding to an order in the sequencer queue is satisfied. The hold delay may be set to provide for distribution of all orders received from client devices, and fairness in distribution of the orders, to a destination24-2514WO_Appcomputing device, based on a chronological time of transmission of a complete order from a TCP client circuit of a client device, for example, as described in the 773 application.

[0107] The fanout circuit 579 may receive the output stream of updated order messages from the sequencer 578, and route given updated order messages to one or more TCP server circuits 580 in the order that the given updated order messages are arranged in the output stream. In one embodiment, the data items may be arranged in an output stream in chronological order of completion of transmissions of data items respectively from the TCP client circuits to timestamp circuits of the system 500, accounting for network offsets corresponding to transmission duration differences for the network paths extending from the respective TCP client circuits and sync offsets corresponding to input ports respectively of timestamp circuits.

[0108] In one embodiment, several TCP server circuits 580 may simultaneously transmit a given order as a respective TCP stream of output data packets including data for the order represented in a given updated order message. The TCP server circuits 580 may transmit the streams of output data packets representative of a respective order over cable 591 of a communication network 592 simultaneously respectively to computing devices serving, for example, as a trading exchange, a financial venue or exchange, a matching engine, a financial trading clearinghouse, a credit check facility or financial trading compliance office.

[0109] In one embodiment, one or more of the client devices 604 may be a laptop, desktop or mobile computing device, such as a smartphone or tablet. The one or more client devices may execute an “app” to interact with the system 500. The app, for example, may execute on a mobile device operating system, such as Apple Inc.'s iOS®, Google Inc.'s Android®, or Microsoft Inc.'s Windows 10 Mobile®, which provides a platform that allows the app to communicate with particular hardware and software components of the mobile device. For example, the mobile device operating system may provide functionalities for interacting with location services circuitry, wired and wireless network interfaces, user contacts, and other applications, where the functionalities include application programming interfaces (APIs) that enable communication with hardware and software modules executing outside of the app, such as included in the system 500. In one embodiment, the client device 604 may, via the app executing on the client device 604, be configured to communicate with the system 500 via the communication device 518.

[0110] In one embodiment, a communication path between the TCP client circuit and an input port of the timestamp circuit 570 may be configured to facilitate communication using a Point to Point24-2514WO_AppFinancial Information eXchange (FIX) protocol, a binary order entry protocol or any other protocol.

[0111] The distribution system 500 may be configured as or to include an FPGA, and network paths 606 communicatively coupling TCP client circuits 602 to the network interfaces 571 associated with respective timestamp circuits 570.

[0112] In one embodiment, each TCP client circuit 602 may be communicatively coupled over a predetermined network path 606 with a predetermined timestamp circuit 570. For example, a TCP client circuit 602A may be communicatively coupled over a network path 606A with an input port 535B of a timestamp circuit 570A.

[0113] In one embodiment, a data item may be an order of an order session established between a client device and the system 500. The order of the order session may be transmitted as a stream of data packets in one or more TCP segments from a TCP client circuit over a predetermined network path 606 to a predetermined timestamp circuit 570, and a timestamp value determined for the order may be based on a time that a data packet of a TCP segment containing a last data byte of a plurality of data bytes forming the order, transmitted from the TCP client circuit, is received at the timestamp circuit.

[0114] For ease of reference and convenience, the disclosure that a component of the apparatus 410 or system 500 may perform a function or operation, is a disclosure that a processor or circuitry of the component or another component of the apparatus 410 or the system 500 may perform or control the performance of the function or operation.Example Methods

[0115] For purposes of illustrating the features of the present disclosure, a high level block diagram of an exemplary process 700, as shown in FIG. 7, which verifies synchronization of timestamping at input ports of a timestamp circuit and determines sync offsets respectively for input ports of a timestamp circuit to provide that timestamping at the input ports is synchronized at least to a predetermined level, is described below in connection with operations performed at components of the exemplary apparatus 210 and the FPGA 260 as shown in FIG. 3. In an exemplary embodiment of the apparatus 210, the process 700 may control: generating a test message as a data packet containing test data in payload of the data packet (test packet), such as a data packet(s) of a TCP segment; replicating the test message; transmitting the replicated test messages, on electrical or optical signals, simultaneously respectively over same length data paths,24-2514WO_Apprespectively to input ports of a timestamp circuit, where the timestamp circuit is of a programmable integrated circuit, such as an FPGA, and where the timestamp circuit determines a timestamp value for the test packet received at an input port that corresponds to a current time of an electronic clock, where the electronic clock is of the apparatus 210 or external to the apparatus 210, when the test message is received at the programmable integrated circuit; receiving, from output ports of the timestamp circuit, output test messages containing the timestamp values determined for the test messages received respectively at the input ports; determining whether any of the timestamp values is outside a range; determining a sync offset for each timestamp value determined to be outside the range, such that the timestamp values collectively with the sync offsets respectively applied based on correspondence to the input ports, do not exceed the range; and storing, in a memory, for a given input port of a timestamp circuit determined to have a timestamp value outside of the range, a sync offset corresponding to the given input port.

[0116] Referring to FIGs. 3 and 7, in block 702, the FGPA source 222 may generate a data packet as a test message (“test packet”) that includes an identifier of the test message as test data in pay load of the test packet. The test message may be part of a TCP stream including the test packet and other data packets.

[0117] In another embodiment, metadata of the test data may be associated with the test message.

[0118] In block 704, a fanout circuit of the FPGA source 222 may replicate the data packet(s) of the test message including the test packet, and cause simultaneous fanout of the replicated test packets containing the same test data in payload as respective test messages, from the output ports 223 over data paths 225 to input ports 219 of the network interface 227. The network interface 227 may, automatically upon receipt of test messages respectively at input ports 219, without delay, transmit the test messages on an electrical or optical signal from output ports 221, over cables 261, such as an electrical or optical cable, respectively to the input ports 268 of the network interface 271. The network interface 271, without delay, may extract the data packets of the test messages respectively from electrical or optical signals received at the input ports 268, and route the data packets respectively of the test messages within the FPGA 260 respectively to input ports 273 of the timestamp circuit 270.

[0119] In block 706, the timestamp circuit 270 may determine timestamp values respectively for the test messages at the input ports 273, based on a current time indicated in a clock signal supplied from the clock 230 when a given test packet is received at an input port 273. In one embodiment,24-2514WO_Appthe timestamp circuit 270 may insert into payload of a TCP test packet corresponding to a test message, or associate as metadata with the test message, an indication of the timestamp value and an identifier of an input port 273 at which the test message is received. In one embodiment, the timestamp value may be determined to correspond to a time of the clock signal when a last data byte of test data of a test message is received at the FPGA 260.

[0120] In block 708, the timestamp circuit 270 may generate, and route from output ports 272 over the communication paths 282 to the switch 224, output messages including respective test packets of the test messages received at the input ports 273 with the corresponding timestamp values and identifiers of input ports 273 and particular test messages.

[0121] In block 710, the switch 224 may extract, from each output message, a timestamp value, test data identifying a particular test message and an identifier of an input port of the timestamp circuit at which the particular test message was received, and provide the extracted information to the controller 212. The processor 214 may save in memory 216 the information extracted from each output message, such that for each output message, the timestamp value, the test data of a test message corresponding to the output message, and an identifier of the input port of the timestamp circuit at the which test message was received are associated with each other in the memory.

[0122] Further in block 710, the processor 214 may determine a range of timestamp values for the same test message, in other words, same test data, based on the timestamp values stored in the memory 216. The memory 216 may include a threshold range, and the processor 214 may determine from the memory whether a range of timestamp values for a given test message exceeds a threshold range. When the range of timestamp values for a given test message exceeds a threshold range, the processor 214 may determine each input port having a timestamp value that may cause the range of timestamp values to exceed the threshold range. In one embodiment, the processor 214 may determine, for a same test message, a minimum timestamp value and a maximum timestamp value, a range as an absolute value of the difference between the maximum and minimum timestamp values and compare the range with a threshold range.

[0123] For example, for a same test message, the timestamp values may be 10 nanoseconds for input port 273A, 5 nanoseconds for input port 273B, 12 nanoseconds for input port 273C and 16 nanoseconds for input port 273D, and a threshold range may be 10 nanoseconds. In this example, the range for the test message is 11 nanoseconds (the absolute value of the difference between the maximum timestamp value of 16 nanoseconds at the input port 273D and the minimum timestamp24-2514WO_Appvalue of 5 nanoseconds at the input port 273B) and, thus, exceeds the threshold range of 10 nanoseconds. The input port 273D, therefore, may be determined to have a timestamp value that causes the range of timestamp values to exceed the threshold range.

[0124] Also in the example, where, additionally, the same test message is received at other input ports 273 which have respective timestamp values greater than 15 nanoseconds, the processor 214 may determine that all those other input ports 273 have timestamp values that cause the range of timestamp values to exceed the threshold range.

[0125] In block 712, the processor 214 may determine, for an input port of the timestamp circuit 270 having a timestamp value that may cause the range of timestamp values to exceed the threshold range as determined in block 710, a sync offset that, when applied to the input port, synchronizes timestamping at the input port, such that the timestamp values determined respectively for input ports of the timestamp circuit 270 of an FPGA are within a desired range, such as the threshold range. In one embodiment, the sync offset may be a difference between a timestamp value outside the threshold range and the threshold range. Further in block 712, the processor 214 may determine a sync offset of zero (“O’) for an input port of a timestamp circuit whose timestamp value is determined not to cause the range of timestamp values to exceed a threshold range. Referring to the above example, for the input ports 273 A, 273B and 273C, the sync offset of each of these input ports is zero. The processor 214 may store in memory 216 sync offsets determined respectively for input ports 273 of a timestamp circuit 270 of a predetermined FPGA.

[0126] Continuing with the example above, the processor 214 may determine a sync offset of -1 nanosecond for the input port 273D. The sync offset, as applied to the input port 273D, may provide that timestamping by the timestamp circuit for all input ports thereof, in other words, timestamp value determinations respectively for all input ports of the timestamp circuit, is synchronized at least to a predetermined level.

[0127] After sync offsets for input ports of the timestamp circuit 270 are determined, the timestamp circuit 270 is in an intra-synchronized state or intra- synchronized, where a timestamp circuit with timestamp synchronization at a desired level among input ports may be implemented in a distribution system including multiple timestamp circuits each being intra-synchronized. As discussed below, by including intra-synchronized timestamp circuits of FPGAs in a distribution system 500, timestamping of data packets containing data items, such as order data, received at input ports respectively of timestamp circuits of FPGAs may be synchronized, and advantageously24-2514WO_Apppermit accurately and reliably verifying that the data items are distributed in chronological order based on timestamp values determined for the data packets containing the data items at respective timestamp circuits.

[0128] Referring to FIGs. 8 A and 8B for purposes of illustrating further features of the present disclosure, a high level block diagram of an exemplary process 750, which, during operation of a distribution system including timestamp circuits of respective FPGAs, such as in the distribution system 500 as shown in FIG. 4 and where each timestamp circuit desirably is intra-synchronized such as described above, verifies synchronization of timestamping by the timestamp circuits, accounting for sync offsets at respective input ports of a timestamp circuit of an FPGA, and generates information on verification of timestamping synchronization, is described below in connection with operations performed at components of the exemplary system 500 as shown in FIGs. 4-6. For purposes of illustration, an exemplary embodiment of the distribution system 500 may receive streams of TCP data packets containing data items as trading orders for financial assets. The process 750 may control, while the system 500 is in an operating state of distributing trading orders in chronological order of receipt at input ports respectively of timestamp circuits, based on timestamp values determined for the trade orders: generating a test message as a data packet containing test data in payload of the data packet (test packet), such as a data packet(s) of a TCP segment; replicating the test message; transmitting the replicated test messages, on electrical or optical signals, simultaneously respectively over same length data paths, to test input ports respectively of timestamp circuits of respective programmable integrated circuits, such as an FPGA, and where each timestamp circuit determines a timestamp value for the test packet received at an input port that corresponds to a current time of an electronic clock of the system 500 when the test message is received at the timestamp circuit of the programmable integrated circuit; receiving, from a timestamp circuit of each programmable integrated circuit, a timestamp value determined for a test packet for a test message received at a test input port of the timestamp circuit; retrieving, from a memory, a sync offset corresponding to a test input port of a timestamp circuit of a given programmable integrated circuit; adjusting a timestamp value corresponding to a test message based on a sync offset corresponding to a test input port of a timestamp circuit at which the test message is received, to determine an adjusted timestamp value; determining a range of adjusted timestamp values for a same test message received at respective timestamp circuits; and24-2514WO_Appidentifying a timestamp circuit for which an adjusted timestamp value causes a range of adjusted timestamp values to exceed a predetermined range.

[0129] Referring to FIG. 8 A, in block 752, the FPGA source 422 may generate a sync data packet as a sync test message (“sync test packet”) that includes an identifier of the sync test message as sync test data in payload of the sync test packet. The sync test message may be part of a TCP stream including the sync test packet and other data packets.

[0130] In another embodiment, metadata of the sync test data may be associated with the sync test message.

[0131] In block 754, a fanout circuit of the FPGA source 422 may replicate the data packet(s) of the sync test message including a sync test packet, and cause fanout of the replicated sync test packets containing the same sync test data in payload as respective sync test messages, simultaneously from the output ports 423 over data paths 425 to input ports 419 of the network interface 418. The network interface 418 may, automatically upon receipt of sync test messages respectively at input ports 419, without delay, transmit the sync test messages on an electrical or optical signal from output ports 421, over cables 590, such as an electrical or optical cable, to input ports 552 of taps 550 respectively associated with FGPAs 560.

[0132] Further in block 754, each tap 550 associated with a corresponding FPGA 560 may divide a signal, such as an optical signal containing the sync test message having data packets in TCP format, received at an input port 552, into first and second portions identically containing the sync test message. The tap 550 may route the first portion from the output port 556 to a test input port 53OA of a network interface 571 of an FGPA 560 associated with the tap 550, and the second portion from the output port 554 over a cable 557 to an input port 435 of the tap aggregator 434. The network interface 571 may, without delay, extract the data packets including the sync data packet from the first portion, and route the extracted data packets for the sync test message from an output port 533A, over a data path 537A within the FPGA 560, to a test input port 535A of the timestamp circuit 570.

[0133] In block 756, the tap aggregator 434 may determine whether, for a given sync test message transmitted to FPGAs 560 of the distribution system 500, sync test packets containing sync test data corresponding to the given sync test message are received at the same or substantially the same time at test input ports 535A respectively of the timestamp circuits 570. If the tap aggregator 434 determines that one or more of second portions of sync test messages are not received from24-2514WO_Apptaps 550 associated with respective FPGAs 560, or are not received at the same or substantially the same time as other first portions, the tap aggregator 434 may communicate this determination to the controller 412. In such circumstance, the controller 412, based on receipt of this determination, may cease to verify inter-synchronization in accordance with the process 750, and indicate the FPGA 560 from which the second portion was not received at all or at the same or substantially the same time as other second portions as being faulty. In one embodiment, the controller 412 may output alert information, such on a display of the apparatus 410, indicating an identifier of a timestamp circuit 570 for which the second portion was not received at the aggregator 434. In an alternative embodiment, the controller 412 may continue to implement the process 750 from block 758, to verify synchronization of timestamp circuits 570 other than the timestamp circuit 570 of the FPGA 560 determined to be faulty in block 756.

[0134] In block 758, each timestamp circuit 570 may determine a timestamp value for a sync test message received at a test input port 530A thereof, based on a current time indicated by a clock signal supplied by the clock 564 when a sync test packet of the sync test message is received. In one embodiment, the timestamp value may be based on a current time indicated by the clock signal when a last data byte of sync test data in a sync test packet is received at the test input port 53OA. In one embodiment, the timestamp circuit 570 may insert a timestamp value into payload of the sync test packet.

[0135] In block 760, each timestamp circuit 570 may generate an output sync message containing the sync test packet of a sync test message received at a test input port 530A of the timestamp circuit 570 with the timestamp value determined therefor in payload of the sync test packet, and route the output sync message from a test output port 540A to an input port 427 of the switch 424 over a data path 429.

[0136] In block 762, the switch 424 may extract the timestamp values for the same sync test message indicated in received output sync messages. In addition, the processor 514 may retrieve from the memory 516 sync offsets corresponding to input ports 535A respectively of timestamp circuits 570. The processor 514, for each output sync message, may determine an adjusted timestamp value for an input port 535 A of a timestamp circuit 570 as a sum of the timestamp value indicated in the output sync message and a sync offset corresponding to an input port 535A at which the sync test message corresponding to the output sync message was received.24-2514WO_App

[0137] Further in block 764, in one embodiment, the processor 514 may determine, for each timestamp circuit 570, whether an output sync message corresponding to a sync test message expected to be received at the timestamp circuit 570, is received at the switch 424. If the processor 514 determines that an output sync message is not received from a timestamp circuit, the processor 514 may generate alert information and cause output of the alert information, such as on a display, indicating an identifier of the timestamp circuit 570 or FPGA 560 from which an output sync message corresponding to a sync test message is not received.

[0138] In block 766, the switch 424 or the controller 412, based on the adjusted timestamp values for a same sync test message, may determine a range of adjusted timestamp values for the same sync test message. If the range is determined to exceed a predetermined range, the controller 412 may identify a timestamp circuit(s) for which a test input port(s) has a timestamp value that may cause the range of adjusted timestamp values to exceed the predetermined range.

[0139] For example, the adjusted timestamp values may be 10 nanoseconds for test input port 53OA of a timestamp circuit 570A, 5 nanoseconds for test input port 53OA of a timestamp circuit 570B, 12 nanoseconds for test input port 53OA of a timestamp circuit 570C and 16 nanoseconds for test input port 530A of a timestamp circuit 570D, and a predetermined range may be 10 nanoseconds. In this example, a range of adjusted timestamp values for the test input ports is 11 nanoseconds, and exceeds the predetermined range of 10 nanoseconds, such that one or more of the adjusted timestamp values may cause the range to exceed the predetermined range. The switch 424 may provide the controller 512 with the adjusted timestamp values for a sync test message, and the processor 514 may determine the smallest and largest adjusted timestamp values for the sync test message, and a maximum adjusted timestamp value that would not cause the range to exceed the predetermined range. In the example, as the minimum adjusted timestamp value is 5 nanoseconds and the predetermined range is 10 nanoseconds, a maximum adjusted timestamp value that would not cause the range of adjusted timestamp values to exceed the predetermined range is 15 nanoseconds, such that any test input port corresponding to an adjusted timestamp value that is greater than 15 nanoseconds is determined as a test input port of a timestamp circuit having a timestamp value that causes the range of adjusted timestamp values to exceed the predetermined range.24-2514WO_App

[0140] In block 768, the controller 412 may provide as an output, such as on a display, adjusted timestamp values for different sync test messages used to verify inter- synchronization of timestamp circuits of the distribution system 500.

[0141] In one embodiment, the controller 412 may provide as an output multiple second ranges of adjusted timestamp values determined over a time interval.

[0142] Referring again to FIGs. 4-6, in an exemplary embodiment of the distribution system 500 in an operating state, such as described in the 773 application, and furthermore configured for verification of inter-synchronization of the timestamp circuits thereof by the apparatus 410, the controller 512 and / or the GSF circuit 576 may determine, for each order of an order message from a TCP client circuit 602 of a client device 604, an updated timestamp value, based on (i) a timestamp value corresponding to a time, as indicated by the clock 564, when all data of the order is completely received at a timestamp circuit 570, (ii) a network offset corresponding to a path length between a client device or associated client circuit, such as a TCP client circuit 602 that is a source of a trade order contained in payload of a data packet of TCP segment(s), and a timestamp circuit 570 that receives the trade order, and (iii) a sync offset for an input port of the timestamp circuit 570 at which the data packet(s) containing the order is received. In addition, the GSF circuit 576 may (i) generate an updated order message representative of a trade order and indicating an updated timestamp value for the trade order, and an output stream of updated order messages for a destination(s) based on updated timestamp values and applying a hold delay to provide that all orders received at the distribution system 500 are distributed chronologically based on times of complete transmission from the client devices or receipt at timestamp circuits; and (ii) distribute the orders according to an order of updated order messages in the output stream to one or more destinations, desirably simultaneously to multiple destinations when distribution to the multiple destinations is indicated for an order. In one embodiment, the GSF circuit 576 may receive, at a sequencer thereof, updated order messages with updated timestamp values; order orders respectively represented in the updated order messages chronologically in a timestamp (first) queue based on the updated timestamp values; order the orders indicated in the timestamp queue in a sequencer (second) queue based on times of receipt respectively of the updated order messages at the sequencer; hold given orders indicated in the sequencer queue for a hold delay, where, for a given order indicated in the timestamp queue having the earliest updated timestamp value and also indicated in the sequencer queue, the given order may be held until the hold delay elapses for the24-2514WO_Appgiven order; and when the hold delay for the given order elapses, insert into an output stream the updated order message corresponding to the given order and other updated order messages for any other orders respectively indicated in the timestamp queue having an updated timestamp value earlier than the updated timestamp value of the given order.

[0143] Referring to FIG. 4, similarly as described in the 773 application, a client device 604 associated with a TCP client circuit 602 may establish an order session for an order with the system 500, to facilitate transmission of details of the order over a predetermined network path 606 to a predetermined input port 535B of a timestamp circuit 570, and distribution of the order, from an order entry gateway 577 of the GSF circuit 576, to one or more destinations, such as computing devices of respective trading exchanges.

[0144] In one embodiment, the TCP client circuit 602 may generate one or more TCP segments formed from a plurality of TCP data packets representative of an order, and transmit a stream of TCP data packets representative of the order over a predetermined network path 606 that communicatively couples the TCP client circuit 602 via an input port 53OB of a network interface 571 with an input port 535B of a timestamp circuit 570.

[0145] In one embodiment, the timestamp circuit 570 may be configured to read and extract data packets representative of an order from a TCP segment, and other data associated with TCP message transmission, and store the data and the data packets in a memory, such as the memory 516.

[0146] The timestamp circuit 570 may, from information in the headers of respective data packets of an order session, determine a data packet that contains a last data byte transmitted for the order session; determine, from the memory 516, a timestamp value for the order based on a time of receipt of the data packet containing the last data byte for the order of the order session; and associate the timestamp value with the order of the order session, such as by inserting the timestamp value in payload of a data packet containing order data of the order, such as a data packet containing a first data byte of the order received at the distribution system 500. The timestamp circuit may determine the timestamp value based on a current time of the clock 564 when the data packet containing the last data byte for the order of the order session is received.

[0147] In one embodiment, the controller 512 may control determining timestamp values respectively for orders of order sessions received at respective timestamp circuits 570, such that the timestamp values are synchronized with the time of the clock 564. Advantageously, the24-2514WO_Apptechnical problem of chronologically ordering order sessions from multiple client devices based on respective times of completion of transmission of the orders, and verifying timestamping by respective timestamp circuits that determine timestamp values for orders is synchronized at least to a predetermined level, is solved by the technical solution of the present disclosure that (i) determines a timestamp value for an order represented in an order session based on a current time, as indicated by the clock 564, at which all of the data packets of the order have been received at a timestamp circuit, and independently of a format of an order message, such as format of the payload corresponding to the details of an order in a TCP segment(s); and (ii) based on the timestamp value, a network offset and a sync offset determined as described above, chronologically orders the orders for distribution based on times of completion of transmission of the orders at the respective client devices.

[0148] In one embodiment, a timestamp circuit 570 may generate an order message for the order indicating a timestamp value determined for the order in payload of a data packet containing order data of the order. In addition, the timestamp circuit 570 may obtain, from memory 516, a network offset for the order and a sync offset for the order based on the input port of the timestamp circuit 570 at which the data packets containing the order data for the order are received, and indicate the network offset and the sync offset in the order message, such as in a header or payload. The timestamp circuit 570 may generate at least one order message including all data packets of the order, the timestamp value, the network offset and the sync offset.

[0149] Referring to FIG. 6, order entry gateway 577 of the GSF circuit 576 may receive from a timestamp circuit at least one order message corresponding to an order entered at a client device. The order entry gateway 577 may determine an updated timestamp value of the order, based on the timestamp value, a network offset corresponding to the TCP client circuit from which the order originated and a sync offset corresponding to the input port 535B at which the order was received. In one embodiment, the order entry gateway 577 may determine an updated timestamp value for the order based on a sum of the network offset for the order, the timestamp value and a sync offset. For example, for a first order, the timestamp value may be 90.050 microseconds, the network offset may be -0.010 microseconds and the sync offset may be -0.010 microseconds, such that the updated timestamp value for the first order is 90.030 microseconds. In addition, the order entry gateway may generate an updated order message indicating the order details and the updated timestamp value for the order.24-2514WO_App

[0150] The sequencer 578 may receive the updated order messages from the order entry gateway 577. In one embodiment, the sequencer 578 may include and control an electronic sequencer clock indicating time at increments of nanoseconds. The sequencer 578 may, for each updated order message received, assign a sequencer timestamp value to the order of the updated order message that is a time indicated by the electronic sequencer clock when the updated order message is received at the sequencer. In one embodiment, the sequencer may delay including in an output stream an updated order message received at the sequencer for a period equal to a hold delay, to provide that all orders received may be distributed to destination computing devices in chronological order based on times of completion of transmissions respectively from TCP client circuits, as indicated by the updated timestamp values respectively for the orders. The hold delay may be determined to compensate for differences in network properties and processing operations associated with conveying order data from TCP client circuits to and processing the order data in the system 500, to ensure fairness of distribution of all orders chronologically based on times of completion of transmission of the orders from respective TCP client circuits. The sequencer 578 may generate an output stream including the updated order messages in chronological order, where individual updated order messages are inserted into the output stream based on application of the hold delay to one or more the updated order messages.

[0151] The sequencer 578 may generate an output stream of updated order messages in chronological order based on the updated timestamp values, applying a hold delay, similarly as described in the 773 application.

[0152] The sequencer 578 may route the output stream including the updated order messages to the fanout circuit 579 of the GSF circuit 576. The fanout circuit 579 may fanout an updated order message to one or more of the TCP server circuits 580 once the updated order message is completely received. In one embodiment, for each order represented in the output stream, the fanout circuit 579 may instruct one or more of the TCP server circuits 580 to transmit, over a cable 591 and a communication network 592, a TCP stream of data packets representative of the order, to a computing device destination. In one embodiment, the TCP stream of data packets representative of the order in the output stream may be transmitted from the TCP server circuits 580 simultaneously to multiple computing devices.

[0153] Accordingly, the present disclosure advantageously provides for verification of timestamping synchronization of timestamp circuits of a distribution system that determine24-2514WO_Apptimestamp values upon which ordering of orders in an output stream is based, while the distribution system operates to distribute orders to computing device dcstination(s) according to the order of the orders in the output stream, i.e., on a first-in-first-out (FIFO) basis, thereby providing for fairness in distribution based on time of completion of transmission of an order at a client device.

[0154] Additionally, the present technology may also be configured as below.

[0155] (1) A system for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first field programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, the system including:

[0156] at least one processor;

[0157] a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor;

[0158] in which the at least one processor is configured to cause the second FPGA to, for a given first programmable integrated circuit of the first programmable integrated circuits:

[0159] generate a first test message and transmit the first test message simultaneously, over a plurality of substantially same length first data paths, to a plurality of input ports of a given first timestamp circuit of the given first programmable integrated circuit,

[0160] in which the given first timestamp circuit is configured to timestamp the first test message at each of the input ports of the given first timestamp circuit with a first timestamp value corresponding to a current time of an electronic clock when the first test message is received at the input port of the given first timestamp circuit;

[0161] in which the at least one processor is configured, for the given first programmable integrated circuit:

[0162] receive, from the given first programmable integrated circuit, the first timestamp values of the first test messages received respectively at the input ports of the given first timestamp circuit;

[0163] determine whether any of the first timestamp values is outside a first range;

[0164] determine an offset for each first timestamp value determined to be outside the first range; and

[0165] for a given input port of the given first timestamp circuit corresponding to a given first timestamp value determined to be outside the first range, store, in a memory, a given offset for the given input port;24-2514WO_App

[0166] in which the at least one processor is configured to, while the distribution system is in an operating state in which input ports respectively of the first programmable integrated circuits arc operable to receive event messages including event data for distribution by the distribution system based on chronological order of receipt of the event messages at the distribution system, cause the second FPGA to:

[0167] generate a second test message and transmit the second test message simultaneously, over a plurality of substantially same length second data paths, to a second input port of the first timestamp circuit of each of the first programmable integrated circuits,

[0168] in which the first timestamp circuit of each first programmable integrated circuits is configured to timestamp the second test message with a second timestamp value corresponding to a current time of a second electronic clock of the distribution system when the second test message is received at the second input port of the first timestamp circuit;

[0169] in which the at least one processor is configured to, for each of the first programmable integrated circuits:

[0170] receive, from the first programmable integrated circuit, a given second timestamp value corresponding to a given second test message received at the first programmable integrated circuit;

[0171] adjust the given second timestamp value with an offset, indicated in the memory for a given second input port at which the given second test message is received, to determine a given adjusted second timestamp value; and

[0172] determine a second range of adjusted second timestamp values.

[0173] (2) The system according to (1), wherein the second range is a difference between a minimum and maximum of the adjusted second timestamp values.

[0174] (3) The system according to (1) or (2), wherein the at least one processor is configured to output given second ranges determined respectively for given second test messages over time.

[0175] (4) The system according to any one of (1) to (3), further comprising the electronic clock.

[0176] (5) The system according to any one of (1) to (4), wherein the electronic clock is external to the distribution system.

[0177] (6) The system according to any one of (1) to (5), wherein the at least one processor is configured to determine whether the second range exceeds a threshold range.24-2514WO_App

[0178] (7) The system according to any one of (1 ) to (6), wherein the at least one processor is configured to determine a first programmable integrated circuit having a given adjusted second timestamp value that causes the second range to exceed the threshold range.

[0179] (8) The system according to any one of (1) to (7), wherein the at least one processor is configured to generate and cause output of an alert indicating an identifier of the first programmable integrated circuit for which the given adjusted second timestamp value is determined to cause the second range to exceed the threshold range.

[0180] (9) The system according to any one of (1) to (8), further comprising:

[0181] a switch communicatively coupling the at least one processor to second output ports corresponding respectively to the second input ports of the first timestamp circuits of the first programmable integrated circuits,

[0182] in which the second timestamp values are received by the at least one processor via the switch.

[0183] (10) The system according to any one of (1) to (9), in which the at least one processor is configured to: cause the first programmable integrated circuits to provide the second timestamp values via a switch of the system to the at least one processor.

[0184] (11) The system according to any one of (1) to (10), in which the distribution system is configured to chronologically order event data of event messages received at given first input ports of given first timestamp circuits respectively of the first programmable integrated circuits accounting for given offsets respectively of the given first input ports of the given first timestamp circuits.

[0185] (12) A system for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first field programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, the system including:

[0186] at least one processor;

[0187] a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor;

[0188] in which the at least one processor is configured to:24-2514WO_App

[0189] cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to input ports respectively of the first programmable integrated circuits, a test message,

[0190] in which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the input port of the first timestamp circuit with a timestamp value corresponding to a current time of an electronic clock of the distribution system when the test message is received at the input port of the first timestamp circuit;

[0191] receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the input port of the first timestamp circuit of the first programmable integrated circuit;

[0192] retrieve from a memory an offset corresponding to a given input port of a given first timestamp circuit of a given first programmable integrated circuit of the first programmable integrated circuits;

[0193] adjust a given timestamp value corresponding to the given input port of the given first timestamp circuit of the given first programmable integrated circuit with the offset corresponding thereto, to determine a given adjusted timestamp value; and

[0194] determine a range of the given adjusted timestamp values.

[0195] (13) The system according to (12), in which the test message is in a packet of a stream of data packets.

[0196] (14) The system according to (12) or (13), in which the stream of data packets is of a TCP stream of data packets.

[0197] (15) The system according to any one of (12) to (14), , in which the test message is in a payload of a first data packet of the stream.

[0198] (16) The system according to any one of (12) to (15), in which a given adjusted timestamp value is inserted in the payload at an application layer of Open Systems Interconnection (OSI) model.

[0199] (17) The system according to any one of (12) to (16), in which the at least one processor is configured to:

[0200] determine whether a given timestamp value corresponds to a given first programmable integrated circuit; and24-2514WO_App

[0201] identify each given first programmable integrated circuit from which a given timestamp value is not received.

[0202] (18) The system according to any one of (12) to (17), further comprising:

[0203] a tap aggregator configured to:

[0204] for each first programmable integrated circuit, receive a portion of a given test message received at the first programmable integrated circuit;

[0205] determine whether the portions are received within a predetermined time range; and

[0206] identify any first programmable integrated circuit for which the portion corresponding thereto is determined to be received outside the predetermined time range.

[0207] (19) The system according to any one of (12) to (18), in which the test message is in payload of a data packet of a TCP stream of data packets, and

[0208] in which a given timestamp value for the test message is in the payload at an application layer of Open Systems Interconnection (OSI) model.

[0209] (20) A system including:

[0210] a plurality of first programmable integrated circuits,

[0211] in which each first programmable integrated circuit is communicatively coupled to at least one first transmission control protocol (TCP) client circuit, in which each first TCP client circuit is configured to receive order data from a market participant, in which each first programmable integrated circuit includes a first field programmable gate array (FPGA) and in which each first FPGA includes a first timestamp circuit;

[0212] at least one processor;

[0213] a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor;

[0214] in which each first programmable integrated circuit is configured to:

[0215] receive, from at least one first TCP client circuit, a stream of data packets representative of at least one order;

[0216] for each order of the at least one order,

[0217] identify a last data byte (LDB) data packet in the stream of data packets containing a LDB of a plurality of data bytes forming the order;

[0218] determine, by the first timestamp circuit of the first FPGA of the first programmable integrated circuit, a timestamp value for the order based on a time indicated by a an electronic24-2514WO_Appclock of the system at a time the LDB data packet of the order is received at the first programmable integrated circuit; and

[0219] generate at least one order message for the order, in which the at least one order message indicates the timestamp value for the order;

[0220] in which the at least one processor is configured to:

[0221] for each order for which at least one order message is generated:

[0222] determine an updated timestamp value based on the timestamp value for the order, a network offset associated with a given first TCP client circuit of the first TCP client circuits from which the order is received at the first programmable integrated circuit and an offset corresponding to an input port of the first timestamp circuit at which the order is received, in which the network offset corresponds to a network path extending from the given first TCP circuit client to the first programmable integrated circuit, and in which each first TCP client circuit has a network offset corresponding to a network path extending from each first TCP client circuit to a respective first programmable integrated circuit; and

[0223] generate an updated order message indicating the updated timestamp value;

[0224] in which the at least one processor is configured to, while any first programmable integrated circuit is operating to receive, from at least one of the first TCP client circuits, a given stream of data packets representative of at least one order:

[0225] cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to test input ports respectively of the first programmable integrated circuits, a test message,

[0226] in which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the test input port of the first timestamp circuit with a timestamp value corresponding to a current time of the electronic clock of the system when the test message is received at the test input port of the first timestamp circuit;

[0227] receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the test input port of the first timestamp circuit of the first programmable integrated circuit;

[0228] retrieve from a memory a given sync offset corresponding to a given test input port of a given first timestamp circuit of each given first programmable integrated circuit of the first programmable integrated circuits;24-2514WO_App

[0229] adjust a given timestamp value corresponding to the given test input port of the given first timestamp circuit of the each given first programmable integrated circuit with the given offset corresponding thereto, to determine a given adjusted timestamp value; and

[0230] determine a range of the given adjusted timestamp values.

[0231] Although the disclosure herein has been described with reference to particular examples, it is to be understood that these examples are merely illustrative of the principles of the disclosure. It is therefore to be understood that numerous modifications may be made to the examples and that other arrangements may be devised without departing from the spirit and scope of the disclosure as defined by the appended claims. Furthermore, while particular processes are shown in a specific order in the appended drawings, such processes are not limited to any particular order unless such order is expressly set forth herein. Rather, various steps can be handled in a different order or simultaneously, and steps may be omitted or added.24-2514WO_App

Claims

CLAIMS1. A system for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first field programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, in which the system includes: at least one processor; a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor; in which the at least one processor is configured to cause the second FPGA to, for a given first programmable integrated circuit of the first programmable integrated circuits: generate a first test message and transmit the first test message simultaneously, over a plurality of substantially same length first data paths, to a plurality of input ports of a given first timestamp circuit of the given first programmable integrated circuit, in which the given first timestamp circuit is configured to timestamp the first test message at each of the input ports of the given first timestamp circuit with a first timestamp value corresponding to a current time of an electronic clock when the first test message is received at the input port of the given first timestamp circuit; in which the at least one processor is configured, for the given first programmable integrated circuit: receive, from the given first programmable integrated circuit, the first timestamp values of the first test messages received respectively at the input ports of the given first timestamp circuit; determine whether any of the first timestamp values is outside a first range; determine an offset for each first timestamp value determined to be outside the first range; and for a given input port of the given first timestamp circuit corresponding to a given first timestamp value determined to be outside the first range, store, in a memory, a given offset for the given input port; in which the at least one processor is configured to, while the distribution system is in an operating state in which input ports respectively of the first programmable integrated circuits are24-2514WO_Appoperable to receive event messages including event data for distribution by the distribution system based on chronological order of receipt of the event messages at the distribution system, cause the second FPGA to: generate a second test message and transmit the second test message simultaneously, over a plurality of substantially same length second data paths, to a second input port of the first timestamp circuit of each of the first programmable integrated circuits, in which the first timestamp circuit of each first programmable integrated circuits is configured to timestamp the second test message with a second timestamp value corresponding to a current time of a second electronic clock of the distribution system when the second test message is received at the second input port of the first timestamp circuit; in which the at least one processor is configured to, for each of the first programmable integrated circuits: receive, from the first programmable integrated circuit, a given second timestamp value corresponding to a given second test message received at the first programmable integrated circuit; adjust the given second timestamp value with an offset indicated in the memory for a given second input port at which the given second test message is received, to determine a given adjusted second timestamp value; and determine a second range of adjusted second timestamp values.

2. The system of claim 1, wherein the second range is a difference between a minimum and maximum of the adjusted second timestamp values.

3. The system of claim 1, wherein the at least one processor is configured to output given second ranges determined respectively for given second test messages over time.

4. The system of claim 1 further comprising the electronic clock.

5. The system of claim 1, wherein the electronic clock is external to the distribution system.24-2514WO_App6. The system of claim 1, wherein the at least one processor is configured to determine whether the second range exceeds a threshold range.

7. The system of claim 6, wherein the at least one processor is configured to determine a first programmable integrated circuit having a given adjusted second timestamp value that causes the second range to exceed the threshold range.

8. The system of claim 7, wherein the at least one processor is configured to generate and cause output of an alert indicating an identifier of the first programmable integrated circuit for which the given adjusted second timestamp value is determined to cause the second range to exceed the threshold range.

9. The system of claim 1 further comprising: a switch communicatively coupling the at least one processor to second output ports corresponding respectively to the second input ports of the first timestamp circuits of the first programmable integrated circuits, in which the second timestamp values are received by the at least one processor via the switch.

10. The system of claim 1, in which the at least one processor is configured to: cause the first programmable integrated circuits to provide the second timestamp values via a switch of the system to the at least one processor.

11. The system of claim 1, in which the distribution system is configured to chronologically order event data of event messages received at given first input ports of given first timestamp circuits respectively of the first programmable integrated circuits accounting for given offsets respectively of the given first input ports of the given first timestamp circuits.

12. A system for synchronizing timestamping at first programmable integrated circuits of a distribution system, in which each of the first programmable integrated circuits includes a first24-2514WO_Appfield programmable gate array (FPGA) and each first FPGA includes a first timestamp circuit, in which the system includes: at least one processor; a second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor; in which the at least one processor is configured to: cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to input ports respectively of the first programmable integrated circuits, a test message, in which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the input port of the first timestamp circuit with a timestamp value corresponding to a current time of an electronic clock of the distribution system when the test message is received at the input port of the first timestamp circuit; receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the input port of the first timestamp circuit of the first programmable integrated circuit; retrieve from a memory an offset corresponding to a given input port of a given first timestamp circuit of a given first programmable integrated circuit of the first programmable integrated circuits; adjust a given timestamp value corresponding to the given input port of the given first timestamp circuit of the given first programmable integrated circuit with the offset corresponding thereto, to determine a given adjusted timestamp value; and determine a range of the given adjusted timestamp values.

13. The system of claim 12, in which the test message is in a packet of a stream of data packets.

14. The system of claim 13, in which the stream of data packets is of a TCP stream of data packets.

15. The system of claim 13, in which the test message is in a payload of a first data packet of the stream.24-2514WO_App16. The system of claim 15, in which a given adjusted timestamp value is inserted in the payload at an application layer of Open Systems Interconnection (OSI) model.

17. The system of claim 12, in which the at least one processor is configured to: determine whether a given timestamp value corresponds to a given first programmable integrated circuit; and identify each given first programmable integrated circuit from which a given timestamp value is not received.

18. The system of claim 12, further comprising: a tap aggregator configured to: for each first programmable integrated circuit, receive a portion of a given test message received at the first programmable integrated circuit; determine whether the portions are received within a predetermined time range; and identify any first programmable integrated circuit for which the portion corresponding thereto is determined to be received outside the predetermined time range.

19. The system of claim 18, in which the test message is in payload of a data packet of a TCP stream of data packets, and in which a given timestamp value for the test message is in the payload at an application layer of Open Systems Interconnection (OSI) model.

20. A system comprising: a plurality of first programmable integrated circuits, in which each first programmable integrated circuit is communicatively coupled to at least one first transmission control protocol (TCP) client circuit, in which each first TCP client circuit is configured to receive order data from a market participant, in which each first programmable integrated circuit includes a first field programmable gate array (FPGA) and in which each first FPGA includes a first timestamp circuit; at least one processor;24-2514WO_Appa second programmable integrated circuit including a second FPGA and communicatively coupled to the at least one processor; in which each first programmable integrated circuit is configured to: receive, from at least one first TCP client circuit, a stream of data packets representative of at least one order; for each order of the at least one order, identify a last data byte (LDB) data packet in the stream of data packets containing a LDB of a plurality of data bytes forming the order; determine, by the first timestamp circuit of the first FPGA of the first programmable integrated circuit, a timestamp value for the order based on a time indicated by an electronic clock of the system at a time the LDB data packet of the order is received at the first programmable integrated circuit; and generate at least one order message for the order, in which the at least one order message indicates the timestamp value for the order; in which the at least one processor is configured to: for each order for which at least one order message is generated: determine an updated timestamp value based on the timestamp value for the order, a network offset associated with a given first TCP client circuit of the first TCP client circuits from which the order is received at the first programmable integrated circuit and an offset corresponding to an input port of the first timestamp circuit at which the order is received, in which the network offset corresponds to a network path extending from the given first TCP circuit client to the first programmable integrated circuit, and in which each first TCP client circuit has a network offset corresponding to a network path extending from each first TCP client circuit to a respective first programmable integrated circuit; and generate an updated order message indicating the updated timestamp value; in which the at least one processor is configured to, while any first programmable integrated circuit is operating to receive, from at least one of the first TCP client circuits, a given stream of data packets representative of at least one order: cause the second FPGA to generate, and transmit simultaneously over a plurality of substantially same length data paths to test input ports respectively of the first programmable integrated circuits, a test message,24-2514WO_Appin which the first timestamp circuit of each first programmable integrated circuit is configured to timestamp the test message at the test input port of the first timestamp circuit with a timestamp value corresponding to a current time of the electronic clock of the system when the test message is received at the test input port of the first timestamp circuit; receive, from each first programmable integrated circuit, the timestamp value corresponding to the test message received at the test input port of the first timestamp circuit of the first programmable integrated circuit; retrieve from a memory a given offset corresponding to a given test input port of a given first timestamp circuit of each given first programmable integrated circuit of the first programmable integrated circuits; adjust a given timestamp value corresponding to the given test input port of the given first timestamp circuit of the each given first programmable integrated circuit with the given offset corresponding thereto, to determine a given adjusted timestamp value; and determine a range of the given adjusted timestamp values.24-2514WO_App

Images