Highly deterministic latency in distributed systems

Abstract

A distributed computing system that supports the concept of latency fairness, such as can be used to implement an electronic trading system, is provided. The system does not favor any particular client; that is, no device is given an unfair advantage or disadvantage over another device simply because it connects to the system at a particular access point (e.g., via a gateway). This goal is achieved by precisely controlling latency, i.e., the time between when a request message arrives at the system and when the corresponding response message is allowed to depart. This precisely controlled deterministic latency may be fixed over time, may vary according to some predetermined pattern, or may vary randomly within a predetermined range of values.

Description

Related Applications

[0001] This application claims priority to U.S. Patent Application No. 16 / 988,249, filed on August 7, 2020, entitled "Highly Deterministic Latency in a Distributed System," and U.S. Patent Application No. 16 / 988,491, filed on August 7, 2020, entitled "Sequencer Bypass with Transactional Preprocessing in Distributed System," both of which are co-pending, and the entire contents of each application are incorporated herein by reference.

Technical Field

[0002] This application relates to connected devices, and more particularly to providing deterministic latency.

Background Art

[0003] Currently, financial commodity trading systems widely used in major stock exchanges enable traders to electronically place orders, receive confirmation notices, market data, and other information via a communication network. A typical electronic trading system typically includes a matching engine that typically resides within a central server, a plurality of gateways for accessing the matching engine, and a distributed processor. A typical order process can be as follows: A request message indicating an order (e.g., a buy order and / or a sell order, etc.) is sent from a client device (e.g., a trader terminal operated by a human user or a server executing an automated trading algorithm, etc.) and is received. Then, typically, an order receipt confirmation is sent back to the client device via the gateway that forwarded the request. After the exchange performs further processing, an order processing confirmation may be sent back to the client device.

[0004] Furthermore, the exchange system may generate market data output by disseminating information about order messages to other systems, either in its original form or in a different format.

[0005] Generally, latency refers to the time between a system input and a visible response. In the context of communication systems, latency is measured as the difference between the time a message is input to or received by the system and the time a corresponding response message is sent. In high-speed electronic trading systems, where minimizing the time to execute a transaction is desirable, latency is a critical consideration.

[0006] In a solution overview titled "Determinism is the New Latency" (published in 2019) by Arista Networks, Inc. (Non-Patent Literature 1), one approach to controlling latency is described as a "speed bump" approach, which introduces a delay of approximately 350 microseconds by lengthening the optical fiber in the message path. This ensures that every order takes exactly the same amount of time to pass through the fiber. Another approach described in the same document is that frequently used transaction data may be kept in the matching engine's cache memory to minimize latency. The document also touches upon the problems associated with trading systems that transfer orders to multiple matching engines using multiple gateways. Participants may be assigned gateways, which also provides further grounds for the non-determinism discussion. It is pointed out that if the time required for order processing by gateways is not deterministic, two orders sent to the exchange in a certain order may actually be executed in a different order. However, no solutions to these problems are proposed.

[0007] US Pre-grant Publication 2019 / 0097745 (Patent Document 1) describes a communication network that uses timestamps to reduce the effects of non-deterministic delay. The state of the transmission path is estimated by observing the "non-deterministic" delay of packets transmitted in the past. The transmission circuit then holds the packet until deterministic latency occurs in the downlink packet at the packet processing circuit.

[0008] ICON Packet Transport (published in 2016) (Non-Patent Literature 2) by Schweitzer Engineering Laboratories, Inc. is an example of a network device that performs deterministic, low-latency packetization using a jitter buffer.

[0009] US Patent 7,496,086 (Patent Document 2) describes a voice network comprising a series of gateways that equalize delay using a jitter buffer.

[0010] US Patent 7,885,296 (Patent Document 3) assigns a timestamp to a frame and maintains synchronization between multiple timestamp counters distributed across different physical layer (PHY) transceivers.

[0011] US Pre-grant Publication 2018 / 0359195 (Patent Document 4) describes a network switch that uses a special type of tree data structure for determining the timestamp range of received packets, which may be used for streaming media in a Real-Time Transmission Protocol (RTP) network. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] U.S. Patent Application Publication No. 2019 / 097745 [Patent Document 2] U.S. Patent No. 7496086 [Patent Document 3] U.S. Patent No. 7885296 [Patent Document 4] U.S. Patent Application Publication No. 2018 / 359195 [Non-patent literature]

[0013] [Non-Patent Document 1] "Determinism is the New Latency", Solution Brief (c) 2019 by Arista Networks, Inc. [Non-Patent Document 2] ICON Packet Transport, by Schweitzer Engineering Laboratories, Inc. (c) 2016 [Overview of the project] [Problems that the invention aims to solve]

[0014] As described herein, fully deterministic latency is provided by preferred embodiments of distributed computing systems, such as electronic trading systems. [Means for solving the problem]

[0015] In one implementation example, inbound messages, such as requests from market participants or other client nodes, enter the system via one of several gateway nodes, i.e., via a given gateway. At this time, the gateway node receiving the inbound message attaches an acceptance time-based value (which may be a "timestamp" dependent on the acceptance time) to the message. The message (with the timestamp embedded within it) is forwarded to another node in the distributed system for processing. As part of the processing of the message, the same timestamp value (and / or constant) is embedded and held within all corresponding response messages generated by other nodes in the distributed system.

[0016] As the gateway node prepares to send back a response message corresponding to the aforementioned request to the participant / client, the response message first passes through a dispatch "Quality of Service" (QOS) shaper. The QoS shaper ensures that the response message is reliably dispatched out of the system only at a highly precise deterministic time, depending on "acceptance timestamp + some deterministic delay".

[0017] The aforementioned QOS shaper can be implemented as a "packet scheduler" that organizes outbound messages into a series of indexed temporary storage locations (i.e., "buckets" associated with discrete, high-precision timing intervals), thereby ensuring that entries entering a specific location within the scheduler are released at their corresponding precise time intervals.

[0018] As another implementation, instead of assigning a timestamp to the arriving message, the gateway may directly associate the inbound message with an indexed location in the packet scheduler corresponding to the desired transmission time. As with other implementations, this transmission time travels through the system along with the same message. As a result, the corresponding response message processed and generated at the core of the distributed system will be transmitted at the assigned time.

[0019] One advantage of the system described herein is that, unlike conventional trading systems, all users of the system receive responses with the same latency. Whether the user is a market participant or simply a subscriber to a market data feed, all users of the system experience the same deterministic response time. This deterministic response time can be a fixed and unchanging time value. However, as an alternative fairness concept, the deterministic time may follow a predetermined pattern or be a randomly selected time from within a given range of possible deterministic response times.

[0020] Other novel configurations and advantages of the approach described herein will become apparent from the following description and the accompanying drawings.

Brief Description of the Drawings

[0021] [Figure 1A] It is a high-level block diagram of a distributed electronic trading system. [Figure 1B] It is a diagram showing messages moving in a direct path from the gateway to the compute node and messages moving via the sequencer node. [Figure 1C] It is a diagram showing an example of the message format. [Figure 2] It is a detailed diagram of system components such as the gateway and compute node. [Figure 3A]This figure shows how time-based values ​​are attached to inbound and outbound messages. [Figure 3B] This is a diagram of the packet scheduler. [Figure 4] This figure shows an example of the system implementing a deterministic latency of 1000 hours. [Figure 5] This diagram shows an asynchronous outbound message. [Figure 6A] This figure shows an example of the system setting a deterministic latency of 2000 hours selected from a deterministic range. [Figure 6B] This example illustrates how the system can implement a deterministic latency of 3500 hours, selected from a deterministic range. [Figure 7] This diagram shows how a series of deterministic latency values ​​are determined. [Figure 8] This diagram shows how a fixed latency time is selected. [Figure 9] This shows how a set of fixed latency times within a given range is selected. [Figure 10] This figure illustrates an example of how the deterministic latency experienced by different participants changes based on additional parameters such as the financial transaction protocol used. [Modes for carrying out the invention]

[0022] (System Overview) The exemplary embodiments disclosed herein relate to a high-speed electronic trading system that provides a market in which buy and sell orders for financial instruments (such as stocks, bonds, commodities, futures, and options) are traded among market participants (such as traders and brokers). The electronic trading system offers low latency, fairness, fault tolerance, deterministic latency, and other features described in detail later.

[0023] The aforementioned electronic trading system primarily plays the role of "matching" trading orders with each other. For example, an offer to "buy" a certain product is matched with a corresponding counter-offer to "sell" it. The matched offer and counter-offer must at least partially meet the desired price, and the remaining quantity that does not meet the price is passed on to another appropriate counter-order. Then, the matched order pair is completed and the trade is executed.

[0024] Orders that are not fully fulfilled or are only partially fulfilled are maintained in a data structure called an "order book." Information held about unmatched trading orders can be used by the matching engine to fulfill subsequent trading orders. Typically, an order book is maintained for each commodity and generally defines or symbolizes the market conditions of that particular commodity. An order book may include, for example, the latest prices and quantities at which market participants have expressed their intention to buy or sell.

[0025] Furthermore, the matching results can be made visible to market participants through a streaming data service known as a market data feed. Typically, a market data feed contains individual messages carrying relevant information such as the pricing of each trading commodity, as well as volume and other statistics.

[0026] Figure 1A shows an example of an electronic trading system 100 comprising a number of gateways 120-1, 120-2, ..., 120-g (collectively referred to as gateway 120), a series of core compute nodes 140-1, 140-2, ..., 140-c (collectively referred to as core compute node 140, or compute node 140), and one or more sequencers 150-1, 150-2, ..., 150-s (collectively referred to as sequencer 150). In other words, in some embodiments, gateways 120, core compute nodes 140, and sequencers 150 are considered to be nodes within the electronic trading system 100. As will be described in more detail later, in one embodiment, gateways 120, compute nodes 140, and sequencers 150 are connected to each other directly, preferably by a low-latency dedicated connection 180.

[0027] In relation to the description of System 100, the term "peer" refers to other devices within the electronic trading system 100 that commonly perform the same function (for example, "gateway" vs. "core compute node" vs. "sequencer"). For example, gateways 120-2, ..., 120-g are peers of gateway 120-1, core compute nodes 140-2, ..., 140-c are peers of core compute node 140-1, and sequencers 150-2, ..., 150-s are peers of sequencer 150-1.

[0028] In the description of System 100, the terms "active" and "standby" may refer to the high availability (HA) roles / states / modes of a system / component. Generally speaking, a standby system / component is a redundant (backup) system / component that is powered on and capable of taking over the functions performed by the active system / component. Without limiting the scope of this invention, such a switchover / failover, i.e., a transition from the standby role / state / mode to the active role / state / mode, may occur automatically in response to a failure of the currently active system / component.

[0029] The electronic trading system 100 processes trading orders from one or more participant computing devices 130-1, 130-2, ..., 130-p (collectively, participant device 130) and provides related information to said devices. Participant devices 130 are devices that interact with the system 100 and may be one or more personal computers, tablets, smartphones, servers, or other data processing devices configured to display and receive trading order information. Participant devices 130 may be operated by a human via a graphical user interface (GUI) or by a high-speed automated trading method operating on some physical or virtual data processing platform.

[0030] Each participant device 130 can exchange messages with the electronic trading system 100 (i.e., send and receive messages) via a connection established with the gateway 120. In Figure 1A, each participant device 130 is depicted as being connected to the electronic trading system 100 via a single connection to the gateway 120, but it should be understood that a given participant device 130 may be connected to the electronic trading system 100 via multiple connections to one or more gateway devices 120.

[0031] While each gateway 120-1 may engage only one participant device 130, it typically engages multiple participant devices 130.

[0032] Compute nodes 140-1, 140-2, ..., 140-c (also referred to herein as matching engine 140 or compute engine 140) provide the matching functions described above and may generate downlink, outlink, or outgoing messages to be delivered to one or more participant devices 130. Each compute node 140 is a high-performance data processor and typically maintains one or more data structures for retrieving and maintaining one or more order books 145-1, 145-2, ..., 145-b. An order book 145-1 may be maintained, for example, for each product handled (responsible) by the core compute node 140-1. In addition, one or more compute nodes 140 and / or one or more gateways 120 may provide a market data feed 147. The market data feed 147 may also be delivered (e.g., by multicast) to subscribers, which may be participant devices 130 or any other suitable computing devices.

[0033] Some of the downlink messages generated by the core compute node 140 may be synchronous, that is, directly generated by the core compute node 140 in response to one or more uplink messages or incoming messages received from one or more participant devices 130 (for example, a corresponding downlink "acknowledgment message" or "execution message" in response to an uplink "new order" message). However, in some embodiments, at least some of the downlink messages may be asynchronous, initiated by the trading system 100, such as certain "unsolicited" cancellation messages, "trade break" messages, or "trade bust" messages.

[0034] A distributed computing environment such as the electronic trading system 100 may consist of multiple matching engines operating in parallel on multiple compute nodes 140.

[0035] The sequencer 150 ensures that the correct sequence of any order-dependent processing is maintained. To ensure that processing of upstream messages is not performed in an incorrect order, typically, upstream messages received by one or more gateways 120 (e.g., new trade order messages from a given participant device 130) pass through at least one sequencer 150 (e.g., a single currently active sequencer, and possibly one or more standby sequencers), and may be marked by a sequence identifier (by the single currently active sequencer, assuming multiple sequencers exist). This identifier may be a unique monotonically increasing value used to determine the relative ordering of messages in subsequent processing across the distributed system 100 (e.g., an electronic trading system 100) and to uniquely identify messages across the entire electronic trading system 100. In some embodiments, the sequence identifier may represent the order in which the messages arrived at the sequencer (i.e., the sequence). For example, the sequence identifier may be a value that is monotonically increased or decreased at regular intervals by the sequencer for each incoming message; for example, the sequence identifier may be increased by 1 for each incoming message. However, it should be understood that the sequence identifier is not limited to a monotonically increasing or decreasing value, as long as it is unique. In some embodiments, the original unmarked message and the sequence-marked message may be essentially identical, except that the marked version of the message contains the value of the sequence identifier. Once sequenced, typically, the upstream marked messages, i.e., sequence-marked messages, are forwarded by one or more sequencers 150 to other downstream compute nodes 140 for processing, and possibly order-dependent processing.In this way, the sequence identifier assigned by the sequencer 150 uniquely identifies a message throughout the electronic trading system 100, and can also define the relative ordering of each marked message relative to other marked messages within the electronic trading system 100.

[0036] Thus, unlike other uses of sequence identifiers, the unique sequence identifiers disclosed herein can be used to ensure a definitive order (i.e., sequence) of electronic transaction message processing. The unique sequence identifier represents a unique definitive ordering (i.e., sequence) instruction for the processing of a given electronic transaction message within an electronic transaction system relative to other transaction messages. In one exemplary embodiment, though not limiting to the invention, the sequence identifier may be entered into the sequence ID fields 110-14 of a message, as will be further disclosed later with respect to Figure 1C.

[0037] In some embodiments, messages may also flow in another direction, namely from the core compute node 140 through one or more gateways 120 to one or more participant devices 130. Such downlink messages generated by the core compute node 140 may also be order-dependent (i.e., sequence-order-dependent) and, accordingly, may be marked with a sequence identifier by first passing through a sequencer 150. The sequencer 150 may then forward the marked response messages to the gateways 120 so that they are delivered to the participant devices 130 in the appropriate deterministic order.

[0038] By using the sequencer 150 to generate a unique sequence number and marking the message or its representation with that sequence number, i.e., generating a sequence-marked message, the correct processing order is ensured throughout the distributed system, i.e., the entire electronic trading system 100, regardless of which compute node or set of compute nodes 140 processes the message. This approach provides "state determinism," meaning the state of the entire system is deterministic and reproducible (and potentially reproducible in another location, such as a disaster recovery site), resulting in fault tolerance, high availability, and disaster resilience.

[0039] Another important point is that the generating node (i.e., a node that introduces a new message into the electronic trading system 100, for example, by generating a new message and / or forwarding a message received from the participant device 130) and its peer node receive the sequence number assigned to the message. This configuration, in which the generating node receives the sequence number of the message it generates, can be useful not only for the generating node and its peer node in terms of processing messages sequentially according to the sequence number, but also in terms of associating the message generated by its node with the sequence identifier of the same message used throughout the rest of the electronic trading system 100. As will be further discussed in relation to Figure 1C, such association between the unmarked version of the message introduced into the electronic trading system by the generating node or node generation and the sequence-marked version of the same message output by the sequencer can be done using the identification information within both versions of the message. Subsequent messages generated within the electronic trading system 100 are assigned their own sequence numbers, while also referencing one or more sequence numbers of related preceding messages. In other words, a node may need to quickly access (by sequence number) messages it generated before, for example, because the sequence number of a message it generated will be referenced in subsequent messages.

[0040] In some embodiments, the generating node may first send a message to the sequencer 150, wait until it receives a sequence number for the message from the sequencer, and then forward the message to another node in the electronic trading system 100.

[0041] In another exemplary embodiment, to avoid one or more hops that could result in an undesirable increase in latency within the electronic trading system 100, the sequencer 150 may, after receiving the pre-sequenced message from the generating node, not only send a sequenced version of the message (e.g., a sequence-marked message) to the destination node, but also reply to the sending node and its peers with the sequenced version of the message almost simultaneously. For example, after assigning a sequence number to an uplink message sent from gateway 120-1 to core compute node 140, the sequencer 150 may not only forward the sequenced version of the message to core compute node 140, but also reply to gateway 120-1 and other gateways 120. This allows any gateway 120 to easily identify the corresponding original message generated at gateway 120-1 by its sequence number if any subsequent message generated at core compute node 140 references that sequence number.

[0042] Similarly, in some further embodiments, a sequenced version of the downlink message, generated by the core compute node 140 and sent to the gateway 120 and sequenced by the sequencer 150, may be forwarded / returned by the sequencer 150 to both the gateway 120 and the core compute node 140.

[0043] Some embodiments may include multiple sequencers 150 for high availability purposes, for example, to ensure that another sequencer is available if the first sequencer fails. In embodiments with multiple sequencers 150 (e.g., the currently active sequencer 150-1 and one or more standby sequencers 150-2, ..., 150-s, etc.), the currently active sequencer 150-1 may maintain a system state log (not shown) of all messages that have passed through sequencer 150-1 and the corresponding sequence numbers of those messages. This system state log may be continuously or periodically transmitted to the standby sequencers to provide the standby sequencers with the system state necessary to take over the active sequencer as needed. Alternatively, the system state log may be stored in a data store accessible to the multiple sequencers 150.

[0044] Furthermore, the system status log may be continuously or periodically replicated to one or more sequencers in a standby replica (details not shown) of the electronic trading system located at disaster recovery site 155, thereby enabling electronic trading to continue in exactly the same state at disaster recovery site 155 even if the primary site of system 100 suffers a catastrophic failure.

[0045] In an exemplary embodiment, the currently active sequencer among a plurality of sequencers may store the system state log in a data store (not shown). This data store may be accessible to the plurality of sequencers via a shared sequencer network, such as the sequencer-wide shared network 182-s, which will be further disclosed later with respect to Figure 1A. When the role (state) of a given sequencer among the plurality of sequencers changes from standby to active, the sequencer may synchronize its state with the state of the original active sequencer by retrieving the system state log from the data store.

[0046] In some embodiments, the system status log may also be provided to a drop copy service 152 which may be implemented by one or more sequencers and / or one or more other nodes in the electronic trading system 100. The drop copy service 152 may provide a record of daily trading activity via the electronic trading system 100, which may be distributed to regulatory bodies and / or clients, for example, via participant devices 130. In an alternative embodiment, the drop copy service 152 may be implemented by one or more gateways 120. In addition to or instead of referring to the system status log, the drop copy service 152 may record trading activity based on the content of uplink and downlink messages transmitted in the electronic trading system 100. For example, in some embodiments, the gateway 120 implementing the drop copy service 152 may receive all messages exchanged in the electronic trading system 100 from the sequencer 150 (and / or from the core compute node 140 or other gateways 120). A participant device 130 configured to receive records of daily trading activity from the drop copy service 152 is not necessarily a participant device that sends trading orders to the electronic trading system 100 or utilizes the matching function of the electronic trading system 100.

[0047] Messages exchanged between participant devices 130 and gateway 120 may conform to any appropriate protocol that can be used for financial transactions (for convenience, referred to as "financial transaction protocols"). For example, these messages may be exchanged in accordance with custom protocols or established standard protocols, including both binary protocols (such as NasdaqOUCH and NYSE UTP) and text-based protocols (such as NYSE FIX CCG). In some embodiments, the electronic trading system 100 may support the exchange of messages conforming to multiple financial transaction protocols simultaneously, including multiple protocols simultaneously on the same gateway 120. For example, participant devices 130-1, 130-2, and 130-3 may establish multiple simultaneous trading connections and exchange messages with gateway 120-1 in accordance with NasdaqOUCH, NYSE UTP, and NYSE FIX CCG, respectively.

[0048] In some embodiments, the gateway 120 may convert a financial transaction protocol-compliant message received from the participant device 130 into a normalized (e.g., standardized) message format used for exchanging messages between nodes within the electronic trading system 100. This normalized transaction format may be an existing protocol, but may generally be of a different size and data format from any financial transaction protocol used for exchanging messages with the participant device 130. For example, the normalized transaction format may include one or more additional fields or parameters, or omit one or more fields or parameters, compared to the original uplink message financial transaction protocol received by the gateway 120 from the participant device 130, and / or each field or parameter of the normalized message may be of a different data type or size than the corresponding message received by the gateway 120 from the participant device 130. Similarly, in the reverse direction, the gateway 120 may convert a downlink message generated in the normalized format by the electronic trading system 100 into a message in the format of one or more financial transaction protocols used by the participant device 130 to communicate with the gateway 120. In Figure 1B, which will be disclosed next, such uplink / downlink messages (e.g., uplink message 103, downlink message 105, etc.) are exchanged between gateway 120-1 and participant device 130.

[0049] Figure 1B is a block diagram of an exemplary embodiment of the electronic trading system 100 of Figure 1A disclosed earlier. In this specific embodiment, the electronic trading system 100 includes a gateway 120-1 connected to a core compute node 140-1 via an activation link 180-1-1 and an ordering (i.e., sequenced) path 117. The electronic trading system 100 further includes a sequencer 150-1 electronically provided within the ordering path 117. The gateway 120-1 is configured to send a message (not shown) to the core compute node 140-1 via the activation link 180-1-1 and the ordering path 117 upon receiving an uplink message 103. The core compute node 140-1 is configured to receive the same message (also called a non-sequenced message) from the gateway 120-1, as well as a sequenced version of the message (not shown) from the sequencer 150-1.

[0050] Without limiting the present invention, a sequence-marked version includes a sequence identifier (ID), which may be contained in the sequence ID field 110-14 of the sequence-marked message, as will be further disclosed later with respect to Figure 1C. The sequence ID indicates the definitive location of the sequence-marked version of the message among several other messages communicated via the activation link 180-1-1 and several other messages of the sequence-marked version received by the sequencer 150-1 via the ordering path 117. The messages to which the sequence ID indicates the definitive location also include other messages of the sequence-marked version received by the core compute node 140-1 via the ordering path 117. The messages (e.g., unsequenced messages) and their sequence-marked versions include common metadata (not shown). The sequence ID of the message is identified by associating the message with its sequence-marked version using this common metadata. The sequence ID also represents the definitive position of the message within the electronic trading system 100, among all messages that have passed through the sequencer 150-1 (i.e., after being sequence-marked by the sequencer 150-1).

[0051] Messages communicated to components of the electronic trading system 100 may be timestamped by those components. However, it should be understood that the sequence ID determined by the sequencer 150-1 determines the position (order / priority) of messages exchanged within the electronic trading system 100. There is a possibility that multiple systems may assign the same timestamp to a message, in which case the order / priority of such messages would need to be resolved at the receiving end of the message. In the electronic trading system 100, this does not occur because the sequencer 150-1 can be the sole determiner of the order / priority of messages exchanged within the electronic trading system 100.

[0052] The core compute node 140-1 may be configured to: (i) initiate matching function activity for electronic trading in response to the receipt of a message via activation link 180-1-1; and (ii) use the sequence identifier to complete the matching function activity and prioritize providing electronic trading services in response to the receipt of a sequence-marked version via ordering path 117.

[0053] Core compute node 140-1 may initiate electronic trading functions (i.e., matching function activities) and begin processing an unsequenced message upon receiving a message (i.e., an unsequenced message), but may not complete the processing and / or commit the results of processing the message until it receives a sequenced message. If the deterministic ordering for processing the sequenced message, determined by the sequence identifier of the sequenced message, is unknown, message processing by compute node 140-1 can become unpredictable, for example. As an example of a possible unpredictable outcome, though not limiting to the present invention, in financial securities trading, there may be multiple unsequenced outstanding messages that are contra-side match candidates. Since only a portion of the contra-side trading orders can likely be executed, it is beneficial to have a deterministic method for resolving these multiple match candidates.

[0054] In some embodiments, after the compute node 140-1 receives both an unsequenced message and a sequenced message, it may associate the unsequenced message with the sequenced message using the identification information in both versions of the message, as described later in relation to Figure 1C. When the compute node 140-1 receives a sequenced message via the ordering path 117, it may determine the appropriate processing sequence for that message (i.e., the sequenced version of the message) in relation to other messages in the electronic trading system 100. The compute node 140-1 may then complete message processing, including sending an appropriate response message, by optionally referring to the sequence identifier assigned by the sequencer 150-1 and included in the sequenced message. Returning to the previous example, which is not limited to the present invention, where multiple messages that are potential match candidates on the contra side occur in a financial securities transaction, the compute node 140-1, upon receiving the sequenced version of the message that is a potential match candidate, may accurately determine the sequence of match occurrences and complete the electronic trading matching function.

[0055] In one exemplary embodiment, in addition to sending a sequence-marked message to compute node 140-1 via a third direct connection 180-c1-s1, the sequence-marked message may also be sent to gateway 120-1 via a second direct connection 180-gw1-s1 (of the ordering path 117). By supplying a sequence-marked message to the message sender (i.e., gateway 120-1), the sender can associate the sequence number (as assigned to the message) with other identifying information within the message (as described later in relation to Figure 1C), thereby facilitating the processing of subsequent messages that reference the sequence number.

[0056] Earlier, we disclosed how compute node 140-1 activates processing based on a message received via activation link 180-1-1. Similarly, gateway 120-1 may activate processing of a response message upon receiving an unsequenced response message from compute node 140-1 via activation link 180-1-1, even before receiving a sequenced version of the response message. Without limiting the invention, the activation of processing may include updating the state of gateway 120-1's open trading order database and / or accumulating a downlink message 105 ready for transmission to participant device 130. However, in some embodiments, gateway 120-1 may not complete processing of a response message, including sending the downlink message 105 to participant device 130, until it receives a sequenced response message containing a sequence identifier indicating the definitive location of the response message within a message sequence including other messages in the electronic trading system 100.

[0057] In some embodiments, after receiving both an unsequenced response message and a sequenced response message, the gateway 120-1 may associate the unsequenced response message with the sequenced response message using the identification information in both versions of the response message, as described later in relation to Figure 1C. This determines the definitive location of the response message when the sequenced response message is received. In some embodiments, the processing of the response message, including the commit of the downlink message 105 to be sent to a participant device such as participant device 130 in Figure 1A, may then be completed.

[0058] Continuing to refer to Figure 1B, the message sent via activation path 180-1-1 and the sequence-marked version of the message sent via ordering path 117 may contain common metadata. Core compute node 140-1 may be further configured to associate the sequence-marked version with the message based on the common metadata upon receiving the sequence-marked version via ordering path 117.

[0059] In the exemplary embodiment shown in Figure 1B, a message is sent to the core compute node 140-1 via activation link 180-1-1 in the forward direction of the activation link (i.e., act-link-fwd-dir113a) and to the core compute node 140-1 via ordering path 117 in the forward direction of the ordering path (i.e., order-path-fwd-dir115a). In addition to completing the matching function activity, the core compute node 140-1 may send a response (not shown) to gateway 120-1 via activation link 180-1-1 in the reverse direction of the activation link (i.e., act-link-rev-dir113b) and to the ordering path reverse direction via ordering path 117 (i.e., order-path-rev-dir115b).

[0060] Activation link 180-1-1 is a single direct connection, whereas the ordering path 117 includes multiple direct connections. For example, the ordering path 117 in this exemplary embodiment includes both direct connection 180-gw1-s1 and direct connection 180-c1-s1.

[0061] The gateway 120-1, the sequencer 150-1, and the core compute node 140-1 are located within a point-to-point mesh topology referred to as the point-to-point mesh system 102. The core compute node 140-1 may be configured to perform a matching function (i.e., an electronic transaction matching function) toward service provision for transaction requests received from the participant device 130 and introduced into the point-to-point mesh topology via the gateway 120-1. In the exemplary embodiment shown in Figure 1B, the point-to-point mesh system 102 has a first direct connection (i.e., 180-1-1), a second direct connection (i.e., 180-gw1-s1), and a third direct connection (i.e., 180-c1-s1). The sequencer 150-1 may be configured to (i) determine a deterministic order (i.e., sequence) of messages exchanged between the gateway 120-1 and the core compute node 140-1 via a first direct connection, and messages received by the sequencer 150-1 from the gateway 120-1 and the core compute node 140-1 via a second direct connection and a third direct connection, respectively. The sequencer 150-1 may be further configured to (ii) communicate the position of the messages in the deterministic order by sending sequence-marked versions of the messages to the gateway 120-1 and the core compute node 140-1 via the second and third direct connections, respectively. The messages constitute a transaction request or a response thereto, as disclosed herein. The message format of such messages will be further disclosed later with reference to Figure 1C.

[0062] The amount of preprocessing that may be performed on an unsequenced message, and whether the results of that preprocessing may need to be discarded or rolled back, may depend on the in-message fields, such as the message type field 110-1, symbol field 110-2, side field 110-3, and price field 110-4 in the embodiment of Figure 1C, which will be further disclosed later. The amount may also depend on whether there are currently other unsequenced (i.e., unprocessed) messages that refer to the same values ​​for common parameters, such as having the same ticker symbol in the message.

[0063] For example, if core compute node 140-1 receives an unsequenced message of the message type “New Order”, it may read symbolic information for the relevant portion of the order book into high-speed memory. If the new order matches an open order in the order book, compute node 140-1 may begin generating a “Confirmed” message accordingly, while it may withhold committing the order book update and sending the “Confirmed” message until it receives a sequenced version of the same message.

[0064] However, if compute node 140-1 has received other unsequentially processed "new order" messages referencing the same ticker symbol, side, price, etc., and these also constitute potential matches for the same open order in the order book, the preprocessing performed by core compute node 140-1 may differ. In some embodiments, core compute node 140-1 may generate competing "completion" message candidates for each of two unsequentially processed "new order" messages that could match the open order. Based on the sequenced version of the message, one of the "completion" message candidates may be discarded, while the other is committed to the order book and sent to gateway 120. In other embodiments, compute node 140-1 may choose not to perform any preprocessing that might need to be discarded or rolled back (e.g., by not generating "completion" message candidates) if multiple unsequentially processed messages could be potential matches for the same open order, or it may choose not to perform any such preprocessing for unsequentially processed messages.

[0065] In another example, if an unsequenced, unprocessed "New Order" message is a candidate match for an open order in the order book, this could conflict with an unsequenced, unprocessed "Order Replacement" or "Order Cancellation" message that is attempting to replace or cancel the same open order in the order book that is a candidate match for the "New Order" message. In such a case, depending on the sequence (order) of the "New Order" message and the "Order Replacement / Cancellation" message assigned by the sequencer, the final result could be either a match between the open order in the order book and the "New Order" message, or the open order being cancelled or replaced with a new order of a different price or quantity. Compute node 140-1 cannot decide which of these two results to pursue until it receives a sequenced version of the conflicting, unsequenced, unprocessed messages from sequencer 150-1.

[0066] In such cases, the preprocessing performed by compute node 140-1 can vary. In some embodiments, compute node 140-1 may simply perform preprocessing that does not need to be rolled back or discarded, such as loading the relevant portion of the order book relating to symbols referenced in both conflicting messages into high-speed memory, if there are multiple conflicting unprocessed messages. In other embodiments, compute node 140-1 may perform additional preprocessing, such as forming one or more provisional response candidates corresponding to one of several conflict scenarios. For example, compute node 140-1 may generate "accepted" message candidates and / or "replacement accepted" message candidates or "cancellation accepted" message candidates, and may also perform provisional updates to the order book corresponding to one or more of the multiple result candidates. In some embodiments, compute node 140-1 may perform this additional preprocessing for all such conflicting scenarios, while in other embodiments, compute node 140-1 may perform the additional preprocessing for only one or some of the conflict scenarios. For example, compute node 140-1 may perform the additional preprocessing on an unsequenced, unprocessed message only if there are no other unprocessed, unsequenced, conflicting messages. Alternatively, or in addition to this, compute node 140-1 may prioritize the execution of additional preprocessing on unsequenced, unprocessed, conflicting messages depending on the amount of time and / or complexity involved in rolling back or discarding the results of the preprocessing. When compute node 140-1 receives a sequenced version of an unsequenced, unprocessed message, it may determine the processing sequence (order) of the unsequenced, unprocessed message (assigned by sequencer 150-1) and complete processing of the message in that sequence. In some embodiments, this processing may include rolling back or discarding one or more results of the preprocessing.

[0067] In addition to the types of preprocessing already described above, in some embodiments, compute node 140-1 may perform message validation preprocessing to determine whether to accept or reject a message, in addition to or instead of the above configuration. For example, such preprocessing may include performing real-time risk checks on messages, such as confirming that the price or quantity specified in the message does not exceed upper limits (i.e., "price limit check" or "quantity limit check"), confirming that symbols in the message are known symbols (i.e., "unknown symbol check"), confirming that trading of the symbol is currently permitted (i.e., "symbol suspension check"), and confirming that the price is properly specified with the correct number of decimal places (i.e., "subpenny check"). In some embodiments, the types of preprocessing may further include "self-dealing prevention" validation checks to prevent a trading client from making a specific match candidate into a self-dealing trade that is a match candidate for itself, if "self-dealing prevention" is enabled for a particular client or trading order. If a trading order fails one or more of these validation checks, the electronic trading system 100 may respond with an appropriate rejection message. While the above embodiments describe such verification checks being performed by compute node 140-1, it should be understood that in some embodiments, at least some of these types of verification checks may be performed by gateway 120 or other nodes within the electronic trading system 100, either instead or in addition to compute node 140-1.

[0068] In a further embodiment, it may be beneficial or necessary for gateway 120-1 to be informed of a unique system-wide sequence identifier associated with a message originating from a client. This information may enable gateway 120-1 to match up the original inbound message with a unique sequence number used to ensure that messages in the electronic trading system 100 are properly ordered. Such configuration at the gateway may be necessary for the electronic trading system 100 to achieve state determinism and provide fault tolerance, high availability, and disaster recovery for the gateway's activities. One solution for configuring gateway 120-1 to maintain information about the sequence identifier associated with inbound messages is to have gateway 120-1 wait for a response containing the sequence identifier to return from sequencer 150-1 before forwarding the message to compute node 140-1. Such an approach may lead to additional latency in message processing. In a further example, the sequencer 150-1 may forward the sequence-marked version of the first message received from gateway 120-1 to compute node 140-1, while simultaneously sending the sequence-marked message back to gateway 120-1. This would allow gateway 120-1 to maintain information about the sequence identifier while minimizing latency in the electronic trading system 100.

[0069] Figure 1C is a list of fields in a representative embodiment of a message format 110 for transaction messages, such as transaction messages exchanged between nodes in the previously disclosed electronic trading system 100. In the representative embodiment of Figure 1C, the message format 110 is a normalized message format intended for use in the internal (i.e., within the electronic trading system 100) representation of transaction messages when exchanged between nodes in the electronic trading system 100. In this representative embodiment, the gateway 120 exchanges messages between the participant 130 and the electronic trading system 100 and converts the messages between one or more formats specified by one or more financial trading protocols used by the participant 130 and the normalized transaction format used between nodes in the electronic trading system 100. Fields 110-1 to 110-17 are not examples that limit the invention, and it should be understood that the number of fields in the message format 110 may increase or decrease, other fields may be included, and the order of the fields is not limited to that shown in Figure 1C.

[0070] Although the fields of message format 110 in this example are depicted within only one type of message format, they may be distributed across multiple message formats or encapsulated in a layered protocol. For example, in other embodiments, some fields within message format 110 may be included as part of the header, trailer, or one or more extension fields of a layered protocol, while other fields of message format 100 may be encapsulated within the message payload of the layered protocol. In some exemplary embodiments, message format 110 may define one or more data fields encapsulated in the payload (data) portion of another message format, including (but not limited to) the payload portions of IP datagrams, UDP datagrams, or TCP packets, or the payload portions of message data frame formats, including (but not limited to) Ethernet data frame formats and other data frame formats, including (but not limited to) InfiniBand, Universal Serial Bus (USB), PCI Express (PCI-e), and High Definition Multimedia Interface (HDMI).

[0071] The message format 110 has fields 110-1...110-6 corresponding to information that may be included in a message transmitted or received in accordance with a financial transaction protocol for communication with one or more participant devices 130. Without limiting the present invention, the message type field 110-1 represents the type of transaction message. Some transaction message types (such as "New Order," "Order Change," and "Order Cancellation") correspond to messages received from participant devices 130, while other message types (e.g., "New Order Confirmation," "Order Change Confirmation," "Order Cancellation Confirmation," "Concluded," "Execution Report," "Unsolicited Cancellation," "Trade Bust," and various rejection messages) correspond to messages included in a transaction message generated by the electronic system 100 and transmitted to the participant devices 130.

[0072] The message format 110 also has a symbol field 110-2, which contains an identifier for the financial security being traded, such as a stock ticker or ticker symbol. For example, "IBM" is the stock ticker symbol for "International Business Machines Corporation". The side field 110-3 in the message format 110 may be used to indicate the "side" of the trading message, such as whether the trading message is a "buy", "sell", or "short sell". Similarly, the price field 110-4 may be used to indicate the desired price for buying or selling the security, and the quantity field 110-5 may be used to indicate the desired quantity of the security (e.g., the number of shares). Furthermore, the message format 110 may have an order token field 110-6, in which the order token field 110-6 may be populated with an "order token" or "client order ID" initially provided by the participant device 130 to uniquely identify a new order in the context of a specific trading session (i.e., a "connection" or "flow") established between the participant device 130 and the electronic trading system via the gateway 120.

[0073] Fields 110-1...110-6 are typical fields usually included in most message types by most financial trading protocols, but it should be understood that message format 110 may include additional or alternative fields to support specific message types or specific financial trading protocols. For example, message types such as "order exchange" and "order cancellation" in many financial trading protocols require participant 130 to enter (supply) an additional order token to represent the order to be exchanged or canceled in order to distinguish it from the original order. Similarly, "order exchange" and "order cancellation" may typically have an exchange / cancellation quantity field, and "order exchange" may have an exchange price field. Furthermore, such additional order exchange / cancellation token fields, exchange price fields, and exchange / cancellation quantity fields may also be included in the corresponding acknowledgment message sent by the electronic trading system 100.

[0074] Furthermore, the message format 110 has fields 110-11...110-17 that may be used internally within the electronic trading system 100 and do not necessarily correspond to fields in messages exchanged with participant devices 130. For example, node identifier fields 110-11 may uniquely identify each node in the electronic trading system 100. In some embodiments, a generating node may include its node identifier in messages it introduces into the electronic trading system 100. For example, each gateway 120 may include its node identifier in messages it forwards from participant devices 130 to compute nodes 140 and / or sequencers 150. Similarly, each compute node 140 may include its node identifier in messages it generates to be sent to other nodes in the electronic trading system 100 (e.g., acknowledgments, trades, or asynchronous messages of a kind that are ultimately intended to be forwarded to one or more participant devices 130). Therefore, each message introduced into the electronic trading system 100 can be associated with the node that generated the message by the node identifier fields 110-11 within the message.

[0075] Furthermore, the message format 110 may have flow identifier fields 110-12. In some embodiments, each trading session (i.e., “connection” or “flow”) established between the participant device 130 and the gateway 120 may be identified by a flow identifier intended to be unique throughout the electronic trading system 100. The participant device 130 may be connected to the electronic trading system 100 by one or more flows via one or more gateways 120. In such embodiments, the flow identifier fields 110-12 of the message (used between nodes in the electronic trading system 100) of all messages exchanged between the participant device 130 and the electronic trading system 100 in a particular flow include a unique identifier for that flow. In some embodiments, the flow identifier fields 110-12 are entered by the message generation node. For example, the gateway 120 may enter the identifier of the corresponding flow in the flow identifier field 110-12 of a message it receives from the participant 130 and introduces into the electronic trading system 100. Similarly, the core compute node 140 may input the corresponding flow identifier into the flow identifier fields 110-12 of the messages it generates (i.e., acknowledgment messages, response messages such as successful messages, and other downlink messages including asynchronous messages).

[0076] In some embodiments, the flow identifier fields 110-12 include a value that uniquely identifies a given logical flow that can actually be implemented as multiple (and possibly multiple gateway) redundant trading session connections for high availability purposes. That is, in some embodiments, the same flow ID may be assigned to two or more redundant flows between one or more participant devices 130 and one or more gateways 120. In such embodiments, the redundant flows can be in either an active / standby configuration or an active / active configuration. In an active / active configuration, multiple redundant flows can exchange functionally equivalent messages in parallel between one or more participant devices 130 and one or more gateways 120 simultaneously. That is, a trading client can send multiple functionally equivalent messages in parallel to the electronic trading system 100 using multiple redundant flows and receive multiple functionally equivalent responses from the electronic trading system 100 in parallel using multiple redundant flows, while the electronic trading system 100 may only process one of these functionally equivalent messages. In an active / standby configuration, only one of several redundant flows can be designated as the active flow at a time, while the remaining redundant flows can be designated as standby flows, and transaction messages can only be actually exchanged in the currently active flow. Regardless of whether the redundant flows are configured as active / active or active / standby, messages exchanged in any redundant flow can be identified by the message-generating node using the same or common flow identifier stored in the flow identifier field 110-12 of the normalized message format 110.

[0077] As described above, in some embodiments, messages exchanged between nodes in the electronic system 100 are sent to the sequencer 150 and marked with a sequence identifier. Therefore, the message format 110 has a sequence identifier field 110-14. In some embodiments, an "unmarked message" may be sent with the sequence identifier field 110-14 empty (e.g., zero). In other embodiments, the sequence identifier field 110-14 of an unmarked message may be set to a specific predetermined value that the sequencer would never assign to the message, or to an invalid value. In yet another embodiment, the unmarking of a message may be indicated by a display in another field of the message (not shown), such as a Boolean value or flag value indicating whether the message is sequenced or not. When the sequencer 150 receives an unmarked message, it may generate a "sequence-marked message" by inputting a valid sequence identifier value into the sequence identifier field 110-14 of the unmarked message. A valid sequence identifier value in the sequence identifier field 110-4 of a sequence-marked message uniquely identifies the message and determines its definitive position in the relative ordering of other marked messages in the electronic trading system 100. In this example, a "sequence-marked message" transmitted by the sequencer 150 and a corresponding unmarked message received by the sequencer can be identical, except that the sequence identifier field 110-14 of the sequence-marked message contains a valid sequence identifier value.

[0078] In some embodiments, the message format 110 may further have reference sequence identifier fields 110-15. A generating node may input the value of the sequence number of a past message related to the message being generated into the reference sequence identifier field 110-15 of the new message it is generating. The value in the reference sequence identifier field 110-15 allows a node in the electronic trading system 100 to associate the message with a past related message.

[0079] Past related messages referenced in the reference sequence identifier fields 110-15 may be past messages in the same “order chain” (i.e., “trading order chain”). In most financial trading protocols, messages can be logically grouped into a given “order chain,” which is a series of messages on the same flow that reference or “originate” a common message. A given order chain typically begins with a “new order message” sent by participant device 130. The next message in the order chain is typically a response from the electronic trading system (e.g., a “new order received” message if the trading system has received the message, or a “new order rejected” message if the trading system has rejected the message, perhaps for reasons such as an invalid price or invalid format or parameters). A given order association may also include an "Order Cancellation" message sent by participant device 130 that cancels at least a portion of a new order that has been previously acknowledged (but still remains open, i.e., includes at least a portion of which has not been canceled and / or has not been completed). The electronic trading system may also acknowledge or reject the "Order Cancellation" message using either an "Order Cancellation Confirmation" or an "Order Cancellation Rejection" message, which are also part of the order association. Furthermore, a given order association may also include an "Order Exchange" message sent by participant device 130 that replaces the quantity and / or price of a new order that has been previously acknowledged (but still remains open). The electronic trading system may also acknowledge or reject the "Order Exchange" message using either an "Order Exchange Confirmation" or an "Order Exchange Rejection" message, which are also part of the order association.An open order that has been previously accepted may match with one or more opposite orders on the other side (i.e., a sell or short order on the other side when one side is a buy order), in which case the electronic trading system 100 may generate a full "concluded" message (when the entire quantity of the open order is filled in one match) or one or more partial "concluded" messages (when only a portion of the quantity of the open order is filled in one match), and these "concluded" orders are also part of the order association. Generally, as described above, the reference sequence identifier may identify other preceding messages within the same order association.

[0080] For example, returning to the reference sequence identifier field 110-15, the value of the reference sequence number may be the sequence number assigned by the sequencer to an "upstream" message originating from participant device 130 and introduced into the electronic trading system 100 by gateway 120, and the corresponding "downstream" message, such as a response message generated by compute node 140, may refer to the sequence number value of the upstream message to which it is being responded to. In this example, the reference sequence identifier field 110-15 of the "New Order Confirmation" message or "Confirmed Order" message generated by compute node 140 includes the value of the sequence identifier assigned to the corresponding "New Order" message that compute node 140 is trying to handle with a response via the "New Order Confirmation" message or an order confirmation via the "Confirmed Order" message. However, generally speaking, the value of the reference sequence identifier field 110-15 does not necessarily have to be that of the message that the electronic trading system 100 is trying to respond to directly, but could be that of a preceding message that is part of the same order association, such as the sequence number of "New Order" or "New Order Confirmation".

[0081] Furthermore, in some embodiments, at least for some message types, the gateway 120 may input the value of the sequence identifier of a preceding related message into the reference sequence identifier field 110-15 of the message it introduces into the electronic trading system 100. For example, the gateway 120 may input the value of the sequence identifier assigned to a preceding corresponding "new order" or "new order confirmation" message into the reference sequence identifier field 110-15 of an "order cancellation" or "order exchange" message. Similarly, the core compute node 140 may input the value of the sequence identifier of a "new order" or "new order confirmation" message into the sequence identifier field 110-15 of a corresponding "order cancellation confirmation" or "order exchange confirmation" message, rather than the value of the sequence identifier of the message to which the compute node 140 is directly responding (for example, not the sequence identifier of an "order cancellation" or "order exchange" message). Again, the reference sequence identifier fields 110-15 enable nodes throughout the electronic trading system 100 to associate a message with one or more past messages related to the same order.

[0082] Furthermore, a generating node may include a node-specific timestamp field 110-13 in the messages it introduces into the electronic trading system 100. While the sequence identifiers contained in the sequence identifier fields 110-14 of the sequence-marked messages output by the sequencer 150 are intended to be unique throughout the entire electronic trading system 100, the values ​​in the node-specific timestamp field 110-13 may be unique only among some messages, i.e., among messages introduced into the electronic trading system 100 by a particular generating node. Although referred to herein as "timestamps," the values ​​assigned to the node-specific timestamp field 110-13 may be any appropriate value that is unique among the messages generated by that node. In practice, the node-specific timestamp may be, for example, a timestamp, or any appropriate value that is monotonically increasing or monotonically decreasing.

[0083] In some embodiments, the message format may have another timestamp field. For example, the message format may have a reference timestamp field. This reference timestamp field may be a timestamp value assigned by the generating node of a past related message. In such embodiments, the compute node 140 may include a new timestamp value in the node-specific timestamp fields 110-13 of the message it generates, and may also include the timestamp value of a related message in the reference timestamp field of the same message it generates. For example, a "New Order Confirmation" message generated by the compute node may include the timestamp value of the "New Order" to which it is responding in the reference timestamp field of the "New Order Confirmation Message". In some embodiments, the compute node 140 may not include a new timestamp value in the node-specific timestamp field 110-13 of the message it generates, but may simply input the timestamp value of a past related message into the node-specific timestamp field 110-13.

[0084] Message format 110 may have time-based value (TBV) fields 110-118. As will be explained in more detail elsewhere, the TBV field may correspond to the time when the uplink message is received at gateway 120-1 (referred to here as the reception time or arrival time), or in another implementation, to the desired exit time (referred to herein as the outgoing time or exit time Tex) when the corresponding response message is sent back by the system. In that case, the TBV is calculated from the reception time plus some deterministic time delay value (as will be explained in more detail later). Therefore, in a typical embodiment, the TBV is typically not the same field as the "node-specific timestamp" fields 110-13 or the sequence number mentioned earlier (for example, it is a different field from sequence ID 110-14 or reference sequence ID 110-15).

[0085] Furthermore, the message format 110 may have entity type fields 110-16 and entity count fields 110-17. The entity type of a message may depend on whether it is introduced into the electronic trading system 100 at the gateway 120 or the compute node 140, i.e., whether the message is an uplink message received at the gateway 120 from a participant device 130, or a downlink message generated by the compute node 140 and sent to the participant device 130. For example, in some embodiments, the entity type of an uplink message is considered to be "flow" (and the entity type field 110-16 of the uplink message is populated by the gateway 120 with a value representing the type "flow"), while the entity type of a downlink message is considered to be "symbol" (and the entity type field 110-16 of the downlink message is populated by the compute node 140 with a value representing the type "symbol"). In this embodiment, the count of entities of type "flow" is maintained by the gateway 120, while the count of entities of type "symbol" is maintained by the compute node 140.

[0086] Consider the entity type "flow". Gateway 120 can maintain an uplink message count for each flow by counting the uplink messages received by Gateway 120 for each active flow on Gateway 120. For example, if four non-redundant flows are active on Gateway 120, as described above, a unique flow identifier is assigned to each flow, and Gateway 120 maintains an uplink message count for each flow by counting the number of uplink messages received by each of those four flows. In such an embodiment, Gateway 120 enters the uplink message count for each flow corresponding to the flow of the uplink message (identified throughout the entire electronic trading system 100 by the flow identifier value entered in the flow identifier field 110-12 of the message) into the entity count field 110-17 of the uplink message.

[0087] In the case of redundant flows in an active / active configuration (i.e., where multiple flows receive the same message or at least functionally equivalent messages in parallel from one or more participant devices 130 connected via one or more gateways 120, as described above), the same flow identifier may be assigned to each underlying redundant flow. In particular, if the redundant flows are implemented on separate gateways 120, a flow-specific upstream message count for each redundant flow may be maintained separately for each underlying redundant flow. Since the participant devices 130 are expected to send the same series of messages to the electronic trading system 100 via their respective redundant flows in the same order (i.e., sequence), the entity counts assigned to functionally equivalent messages received in separate redundant flows are also expected to be the same.

[0088] These functionally equivalent upstream messages can be forwarded to the sequencer 150 and compute node 140 by one or more gateways 120. In other words, in such an embodiment, the sequencer 150 and compute node 140 may receive multiple functionally equivalent upstream messages associated with the same flow identifier, but the sequencer 150 and compute node 140 can identify that the messages are functionally equivalent if the entity counts of the multiple messages having the same flow identifier are identical.

[0089] In some embodiments, the sequencer 150 and compute node 140 may track the maximum value of the entity count included in the entity count fields 110-17 of the upstream messages corresponding to each flow, for each flow. This allows the sequencer 150 and compute node 140 to process only the first message that arrives among several functionally equivalent upstream messages received by each node, and ignore other functionally equivalent upstream messages that arrive later. In some embodiments, for example, the sequencer 150 may sequence only the first functionally equivalent upstream message that arrives, and the compute node 140 may initiate processing only for the first functionally equivalent message that arrives. If the entity count of an uplink message received by a node (i.e., sequencer 150 or compute node 140) is the same as or less than the maximum entity count confirmed by that node for the same flow, the node may determine that the uplink message is functionally equivalent to another uplink message previously received, and may simply ignore any subsequent functionally equivalent uplink messages.

[0090] Next, let's consider the case of entity type "symbol". The compute node 140 can maintain a count of downlink messages for each symbol by counting the downlink messages generated and transmitted from the compute node 140 for each symbol that the compute node 140 handles (serviced by). For example, if the compute node 140 handles four types of symbols (e.g., MSFT, GOOG, IBM, ORCL, etc.), a symbol identifier is assigned to each symbol, as described above, and the compute node 140 maintains a count of downlink messages for each of the four types of symbols by counting the number of downlink messages it generates and transmits that handle each of those four types of symbols. In this embodiment, the compute node 140 inputs a downlink message count for each symbol, which is associated with a symbol in the downlink message (identified throughout the electronic trading system 100 by the value entered in the symbol identifier field 110-2 of the message), into the entity count field 110-17 of the uplink message.

[0091] As will be explained further later, in some embodiments, the compute nodes may be configured such that multiple compute nodes handle a particular symbol in parallel for reasons of high availability. Even when multiple compute nodes handle the same given symbol, since the messages are deterministically ordered throughout the entire electronic trading system 100 by the sequencer 150, any compute node can process upstream messages referencing the same symbol in the same order (i.e., sequence) and in the same format, thereby ensuring that functionally equivalent response messages are generated in parallel. Consider the case where downstream messages for a particular symbol are sent through multiple compute nodes 140, and each downstream message referencing that symbol should also be sent from each other actively handling that symbol, with a functionally equivalent message being sent from each other actively handling that symbol. All of these downstream messages can be sent from the compute nodes 140 to the sequencer 150 or the gateway 120. Therefore, in such embodiments, the sequencer 150 and gateway 120 may receive multiple functionally equivalent upstream messages relating to the same symbol, but the sequencer 150 and gateway 120 can identify messages as functionally equivalent if the entity counts of multiple messages having the same symbol identifier are identical. In some embodiments, the sequencer 150 and gateway 120 may track, for each symbol, the maximum entity count included in the entity count fields 110-17 of the downstream messages relating to that symbol. This allows the sequencer 150 and gateway 120 to process only the first message that arrives among multiple functionally equivalent downstream messages received by each node, and ignore other functionally equivalent downstream messages that arrive later. In some embodiments, for example, the sequencer 150 may sequence only the first functionally equivalent downstream message that arrives. Similarly, the gateway 120 may initiate processing only for the first functionally equivalent message that arrives.If the entity count of a downlink message received by a node (i.e., sequencer 150 or gateway 120) is the same as or less than the maximum entity count previously observed for that symbol at that node, the node may determine that the downlink message is functionally equivalent to another downlink message previously received, and may simply ignore any subsequent functionally equivalent downlink messages.

[0092] In an embodiment in which the PLC 150 sequences only the message that arrives first among multiple functionally equivalent messages, there can be various ways in which the PLC does this. For example, other arriving messages that are functionally equivalent to the first arriving functionally equivalent message may simply be ignored by the PLC (in which case, for a series of functionally equivalent messages, only one message with a sequence mark may be output by the PLC). Alternatively, the PLC may track the sequence number it has assigned to the first functionally equivalent message by associating the entity count, flow identifier, or symbol identifier (if the message has entity type "flow" or "symbol") of the first functionally equivalent message with its sequence number. As a result, the sequencer 150 can output each functionally equivalent message in the sequenced version such that the value assigned by the sequencer to the first functionally equivalent message received by the sequencer is the same as the value in the sequence identifier fields 110-14 of all functionally equivalent messages in the sequenced version.

[0093] In other embodiments, the sequencer 150 does not track whether messages are functionally equivalent and may assign a unique sequence number to each unsequenced message arriving at the sequencer 150, regardless of whether the message is one of several functionally equivalent messages. In such embodiments, a different sequence identifier is assigned by the sequencer to each sequenced version of a functionally equivalent message as the value of the sequencer identifier field 110-14. In such embodiments, a receiving node of sequenced functionally equivalent messages may determine the effective sequence identifier within the sequenced set of messages using the sequence identifier of the sequenced version of the message that first arrived at the node among the sequenced functionally equivalent messages. In embodiments where point-to-point direct connections exist between nodes in the electronic trading system 100, the sequenced versions of messages are sent by the sequencer 150 in sequenced order, and accordingly, any node directly connected to the sequencer should receive the sequenced versions of messages in the same sequenced order. Therefore, at any node that receives a sequenced message via a point-to-point direct connection with the sequencer, the value of the sequence identifier field 110-14 of the first sequenced message to arrive among several functionally equivalent sequenced messages should be the same.

[0094] As may become clear from the following explanation, in embodiments with a message format such as message format 110, there may be multiple other methods for uniquely identifying a message throughout the entire electronic trading system 100, in addition to the message sequence identifier. For example, in embodiments where a message has both a node identifier and a node-specific timestamp, the mere presence of these two types of identifiers within the message may be sufficient to uniquely identify the message throughout the entire electronic trading system 100. Such fields may be interpreted as containing metadata, and multiple messages with identical metadata may be interpreted as containing common metadata. Similarly, in embodiments where the flow identifier is unique throughout the entire electronic trading system 100, the combination of the message flow identifier and the node-specific timestamp may be sufficient to uniquely identify the message throughout the entire electronic trading system 100. Furthermore, the combination of the flow identifier and the entity count may be sufficient to uniquely identify a message of entity type "flow," and the combination of the symbol identifier and the entity count may be sufficient to uniquely identify a message of entity type "symbol."

[0095] However, even if there were other ways to uniquely identify a message throughout the entire electronic trading system 100 besides the sequence identifier assigned to the message, it should be noted that the sequence identifier is still necessary to fairly and definitively specify the relative ordering of that message among other messages generated by other nodes in the electronic trading system 100. For example, if node-specific timestamps are actually implemented as timestamp values, even if the system clocks between nodes are perfectly synchronized, it is possible that two different messages, each generated by a different node, could be assigned the same timestamp value, in which case the relative order of these two messages would become ambiguous. Even if it is possible to uniquely identify the messages, the receiving nodes of both messages would need to determine the relative ordering of these two messages by some means before performing any possible processing on the messages.

[0096] One possible method for a receiving node to resolve this ambiguity is to use randomness, for example, by randomly selecting a given message and assuming that it precedes another given message in the relative ordering of messages throughout the electronic trading system 100. However, a configuration that resolves ambiguity using randomness does not support the "state determinism" of the entire electronic trading system 100. If separate receiving nodes randomly determine different relative orderings for a common set of messages, the behavior within the electronic trading system 100 becomes unpredictable and non-deterministic, potentially hindering the proper implementation of critical functions such as fault tolerance, high availability, and disaster recovery.

[0097] Another method for receiving nodes to resolve ordering ambiguity is, for example, to assign a predetermined priority based on the node identifier associated with the message. However, such a method would contradict the important goal of fairness, as it would simply assign higher priority to some messages based on the node identifier of the node that introduced the message into the electronic trading system 100. For example, a given participant device 130 might be favored simply because it happens to be connected to the electronic trading system 100 via a gateway 120 that is considered to have a higher priority according to a predetermined prioritization method.

[0098] If a message is uniquely identified by an entity count and (depending on whether the message's entity type is "symbol" or "flow") a symbol identifier or flow identifier, then ordering can be deterministic between other messages relating to the same symbol (if the entity type is "symbol") or the same flow (if the entity type is "flow"). However, ordering remains non-deterministic between other messages relating to different symbols or flows.

[0099] Therefore, while fields in the message format 100 other than the sequence identifier assigned to the message by the sequencer 150 may be sufficient to uniquely identify a message throughout the entire electronic trading system 100, the sequence identifier may still be necessary to fairly and definitively identify the ordering of a given message relative to other messages within the electronic trading system 100. In such embodiments, the sequencer 150 (or, if there are multiple sequencers 150, the currently active sequencer) functions as an authoritative source with the authority to perform truly definitive ordering of sequence-marked messages throughout the entire electronic trading system 100.

[0100] In some embodiments, a node in the electronic trading system 100 may receive two versions of a message: an unsequenced (unmarked) version of the message introduced into the electronic trading system 100 by the generating node, and a version of the message to which a sequence identifier has been assigned (marked) by the sequencer 150. This can occur in embodiments where the generating node sends the unmarked message to the sequencer 150 as well as to one or more receiving nodes. The sequencer 150 may then send the sequenced version of the same message to a set of nodes, including the same receiving node.

[0101] As mentioned above, while the sequence-marked version of the message is useful in determining the relative processing order (i.e., position within the sequence) of the message among other marked messages in the electronic trading system 100, it may also be useful to have the receiving node receive the unmarked version of the message. For example, since the marked version of the message is transmitted via an intermediary hop, the sequencer 150, it is quite possible, albeit unexpectedly, that the unmarked version of the message is received before the marked version of the message (for example, in embodiments where a direct connection exists between nodes). In other words, in some embodiments, the receiving node has the opportunity to activate the processing of the unmarked message by receiving the unmarked message before receiving the marked version of the message, which has the authority to determine the relative ordering among other marked messages.

[0102] A node that receives both a marked and an unmarked version of the same message can associate these two versions of the message with each other using common identifiers or “common metadata” within both versions of the message. For example, in an embodiment where the marked and unmarked versions of a message are essentially identical except for a sequence identifier assigned by the sequencer 150, the marked message may also contain the same node identifier and node-specific timestamp that are present in the corresponding unmarked message, so that a node receiving both versions of the message can associate the marked and unmarked versions. This allows marked messages to have their own relative ordering relative to other marked messages in the electronic trading system 100, but the association that may be made between the unmarked and marked versions of the same message will (at least using the aforementioned association) determine the relative ordering of the marked message relative to other messages (whether marked or unmarked) in the electronic trading system 100. It should also be understood that nodes in the electronic trading system 100 may associate sequence-marked messages with unmarked versions of messages by other methods described above that uniquely identify the messages. For example, the association between sequence-marked messages and unmarked messages may be made by a combination of a flow identifier and a node-specific timestamp.In addition to or instead of the above, for messages that each have entity types "Symbol" and "Flow," such association may be made by the entity count of the message and the Symbol identifier or Flow identifier within the message.

[0103] In the era of high-speed trading, where microseconds, and even nanoseconds, are inevitable, participant devices 130 that exchange messages with the electronic trading system 100 are often highly susceptible to latency, and predictable low latency is preferred. The configuration shown in Figure 1A addresses this requirement by providing a point-to-point mesh 172 architecture at least between each gateway 120 and each compute node 140. In some embodiments, each gateway 120 within the mesh 172 may have a dedicated high-speed direct connection 180 to the compute node 140 and the sequencer 150.

[0104] For example, a dedicated connection 180-1-1 is established between the first gateway 120-1 and the first core compute node 140-1, a dedicated connection 180-1-2 is established between the first gateway 120-1 and the second core compute node 140-2, and so on. This results in connection example 180-gc being established between gateway 120-g and the cth core compute node 140-c, and connection example 180-sc being established between sequencer 150 and the cth core compute node 140-c.

[0105] It should be understood that in some embodiments, each of the dedicated connections 180 of the point-to-point mesh 172 becomes a point-to-point direct connection that does not utilize a shared switch. Herein, dedicated connections or direct connections may be interchangeably referred to as direct "links" or "dedicated links," which are direct connections dedicated to (e.g., non-shared) communication between two endpoints. Such dedicated / direct links may be any suitable interconnects or interfaces, as will be further disclosed later, and are not limited to network links such as wired Ethernet network connections or other types of wired or wireless network links. Herein, dedicated / direct connections / links may be referred to as end-to-end paths between two endpoints. Such end-to-end paths may be a single connection / link or a series of connections / links. However, the bandwidth of a dedicated / direct connection / link is non-shared overall, i.e., from one endpoint to another, and the bandwidth and latency of a dedicated / direct connection / link are not affected by the resource utilization of one or more elements being traversed. For example, a dedicated / direct connection / link may traverse one or more buffers or other elements whose use does not affect bandwidth or latency. However, such a dedicated / direct connection / link does not traverse a shared network switch whose use could affect bandwidth and / or latency.

[0106] For example, in some embodiments, the dedicated connection 180 in the point-to-point mesh 172 can be provided in a number of ways, such as 10 Gigabit Ethernet (GigE), 25 GigE, 40 GigE, 100 GigE, InfiniBand, Peripheral Component Interconnect-Express (PCIe), RapidIO, Small Computer System Interface (SCSI), FireWire, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), custom serial or parallel buses, etc.

[0107] Therefore, although the compute engine 140, gateway 120, sequencer 150, and other components may be referred to as “nodes” in this specification, other types of interconnections and interfaces are also possible. Thus, the use of terms such as “compute node,” “gateway node,” “sequencer node,” and “mesh node” should not be interpreted as meaning that the connection of a particular component must be made using a network link. Furthermore, “node” as disclosed herein may be any appropriate hardware, software, firmware component, or combination thereof configured to perform the functions described for that node. As will be described in detail later, a node may be a programmed general-purpose processor, but may also be a dedicated hardware device such as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other hardware device or device group, or it may be logic within a hardware device, printed circuit board (PCB), or other hardware component.

[0108] It should be understood that the nodes disclosed herein may be separate elements or may be integrated together within a single element, for example, within a single FPGA, ASIC, or other element configured to implement the logic for performing the functions of the nodes described herein. Furthermore, the nodes may be instantiations of software implementing the logic, which is executed by a general-purpose computer and / or any of the aforementioned devices.

[0109] The traditional approach of connecting components such as the compute engine 140, gateway 120, and sequencer 150 through one or more shared switches does not result in minimal latency. Furthermore, this traditional approach also leads to unpredictable latency spikes during periods of heavy message traffic.

[0110] In one exemplary embodiment, dedicated connections 180 are directly provided between each gateway 120 and each sequencer 150, and between each sequencer 150 and each core compute node 140. In some embodiments, dedicated connections 180 are also provided between all sequencers, so that sequencer example 150-1 has a dedicated connection 180 with each of the other sequencers 150-2, ..., 150-s. Although not shown in Figure 1A, in some embodiments, dedicated connections 180 may also be provided between all gateways 120, so that each gateway 120-1 has a dedicated connection 180 with each of the other gateways 120-2, ..., 120-g. Similarly, in some embodiments, dedicated connections 180 are also provided between all compute nodes 140, so that core compute node example 140-1 has a dedicated connection 180 with each of the other core compute nodes 140-2, ..., 140-c.

[0111] Furthermore, it should be understood that in some embodiments, a dedicated connection 180 between two nodes (for example, between any two nodes 120, 150, and 140) may be implemented as multiple redundant dedicated connections between the same two nodes to enhance redundancy and reliability. For example, a dedicated connection 180-1-1 between gateway 120-1 and core compute node 140-1 (for example, the first core) may actually be implemented as a pair of dedicated connections.

[0112] Furthermore, in some embodiments, all messages sent by a given node are sent in parallel to all nodes directly connected to that node within the point-to-point mesh 172. Each node within the point-to-point mesh 172 can decide for itself, for example based on its configuration, whether to perform any processing upon receiving a message or simply ignore it. In some embodiments, even if a node's configuration is such that it performs virtually no processing upon receiving a message, it may not completely ignore the message but may perform at least minimal processing, such as consuming an arbitrary sequence number assigned to the message by the sequencer 150. That is, in such embodiments, the node may track the most recently received sequence number. This ensures that when the node performs more substantial processing on a message, that processing is carried out in the appropriate sequence order.

[0113] For example, suppose a message containing a trade order to "sell 10 shares of Microsoft stock at $190" is sent from a participant device 130-1, such as a trader's personal computer, and arrives at gateway 120-1 (i.e., the first GW). This message is sent to all core compute nodes 140-1, 140-2, ..., 140-c, even if only core compute node 140-2 is currently performing matching for Microsoft's order. All other core compute nodes 140-1, 140-3, ..., 140-c may either ignore the message upon receipt or perform minimal processing on it. For example, the only processing that 140-1, 140-3, ..., 140-c might perform is to consume the sequence number assigned to the message by sequencer 150-1. Furthermore, this message is sent to all sequencers 150-1, 150-2, ..., 150-s, even if there is only one currently active sequencer engaged in the mesh (in this example, sequencer 150-1). This configuration, where sequencers 150-2, ..., 150-s also receive the message, provides an opportunity for sequencers 150-2, ..., 150-s to take over as the currently active sequencer if sequencer 150-1 (the currently active sequencer) fails or if replacing it with another active sequencer would improve the overall reliability of the electronic trading system 100. Additionally, one or more other sequencers (e.g., sequencer 150-2) may be responsible for transferring the system state to the disaster recovery site 155. Disaster recovery site 155 may have a replica of the electronic trading system 100 located in another physical location, which includes physical or virtual instances of some or all of the individual components of the electronic trading system 100.

[0114] By configuring the system to send each message in parallel to all directly connected nodes, the system 100 reduces complexity while promoting redundancy and high availability. If all directly connected nodes are initially configured to receive all messages, multiple nodes can be configured to redundantly process the same message. Returning to the aforementioned order example of "selling 10 shares of Microsoft stock at $190," in some embodiments, multiple core compute nodes 140 may simultaneously perform matching for Microsoft's order. For example, both core compute node 140-1 and core compute node 140-2 may simultaneously perform matching for Microsoft's message, and after receiving the upward message for the "sell" order, they may each independently generate response messages such as acknowledgment messages and execution messages, and each core compute node 140-1 and core compute node 140-2 may send these messages to the gateway 120 via one or more sequencers 150, which will then pass them on to one or more participant devices 130.

[0115] Because strict ordering and state determinism are ensured by one or more sequencers 150, any corresponding response message generated and transmitted independently by core compute nodes 140-1, 140-2, respectively, can be reliably equivalent in substance and function. In this way, the architecture of the electronic trading system 100 easily supports redundant message processing, increasing the availability and resilience of the system. In such embodiments, the gateway 120 may receive multiple corresponding downlink messages from the core compute node 140 for the same corresponding uplink message. Since these multiple corresponding response messages can be reliably equivalent, the gateway 120 can simply process only the first downlink message received for the same uplink message, ignoring subsequent ones. In some embodiments, the “first” and “subsequent” messages may be sequence-marked messages, which can be identified by their associated sequence numbers. However, in other embodiments in which the sequencer 150 assigns only one sequence identifier to multiple functionally equivalent messages, the functional equivalentity of the messages can be identified based on other identifying information within the messages, such as the values ​​of the entity type fields 110-16 and the entity count fields 110-17, as described in detail in relation to Figure 1C.

[0116] Furthermore, this allows the gateway 120 to process only the first message to arrive at the gateway 120 from among multiple functionally equivalent corresponding messages, thereby potentially improving the overall latency of the electronic trading system 100. It is also easy to configure the electronic trading system 100 so that multiple compute nodes 140 process any uplink message, each of which generates an equivalent response message, and the first response message to arrive is processed by the gateway 120. With such an architecture, even if a compute node 140 does not handle an uplink message for a certain period of time (whether due to system failure, node reconfiguration, or maintenance work), there will be no noticeable impact on latency, resulting in high availability.

[0117] In addition to supporting predictable low latency and redundant processing of messages, the point-to-point mesh 172 architecture of system 100 has multiple redundant paths built into it. It is immediately apparent that there are multiple paths between any gateway 120 and any compute node 140. Even if the direct connection 180-1-1 between gateway 120-1 and compute node 140-1 becomes unavailable, communication between the two elements can continue via another path, such as traversing one of the sequencers 150. In other words, more generally, there are multiple paths between any node in the point-to-point mesh 172 and any other node.

[0118] Furthermore, this point-to-point mesh architecture inherently supports another important objective of financial trading systems: fairness. The point-to-point architecture, with its direct connections between nodes, ensures that the latency of paths between any gateway 120 and any core compute node 140, and between a sequencer 150 and any other node, are all identical or at least very similar. This ensures that two uplink messages sent simultaneously from two different gateways 120 to a sequencer 150 arrive at the sequencer 150 virtually simultaneously. Similarly, downlink messages sent from a core compute node 140 should be sent simultaneously to all gateways 120 and received at each gateway virtually simultaneously. Because the point-to-point mesh topology does not favor any particular gateway 120, the possibility of a participant device 130 connected to a specific gateway 120 being unfairly favored or disadvantaged is minimized.

[0119] Furthermore, the point-to-point mesh architecture of system 100 makes it easy to reconfigure the function of a node, i.e., whether the node is currently functioning as a gateway 120, a core compute node 140, or a sequencer 150. In embodiments where there are direct connections between each node in the point-to-point mesh and each other node, such reconfiguration becomes extremely simple. If each node in the point-to-point mesh 172 is connected to each other node by direct connections, changing the function of a node in the mesh (for example, changing the function of a node from a core compute node 140 to a gateway 120, or from a gateway 120 to a sequencer 150) does not require reconnecting the connections 180 (whether physical or virtual) within the mesh using wiring or cables. In such embodiments, the necessary reconfiguration within the point-to-point mesh 172 can be easily achieved by remotely performing configuration changes. If a node is reconfigured to function as a gateway 120 or reconfigured to function as a gateway 120, some auxiliary network changes may be required outside the point-to-point mesh 172, but the internal wiring of the mesh can be maintained without any changes.

[0120] Therefore, in some embodiments, the reconfiguration of node functionality can be achieved in real time (live) and even dynamically during trading hours. For example, changes in the load characteristics of the electronic trading system 100 or new demands may make it useful to reconfigure one of the core compute nodes 140-1 to function as an additional gateway 120. After some redistribution of state or configuration to another compute node 140, this new gateway 120 may become ready to begin accepting new connections from participant devices 130.

[0121] In some embodiments, slower and potentially higher-latency shared connections 182 may be provided between system components, including between gateways 120 and / or between core compute nodes 140. These shared connections 182 may be used for maintenance, control operations, administrative operations, and / or similar operations that do not require ultra-low-latency communication, such as or in contrast to messages related to trading activity carried over dedicated connections 180 within the point-to-point mesh 172. In contrast to the first direct connections 180-1-1, second direct connections 180-gw1-s1, and third direct connections 180-c1-s1 that carry traffic related to trading activity, shared connections 182-g, 182-c carry traffic of types other than trading activity. Shared connections 182 that carry traffic other than trading activity may be via one or more shared networks and one or more network switches, where nodes in the mesh may exhibit various distribution patterns across these shared networks. For example, in some embodiments, all gateways 120 may be within a gateway-wide shared network 182-g, compute nodes 140 may be within their own computer node-wide shared network 182-c, and sequencers 150 may be within their own sequencer-wide shared network 182-s. On the other hand, in other embodiments, in the case of operations such as those described above, which are less affected by latency, all nodes within the mesh may communicate on the same shared network.

[0122] A distributed computing environment, such as the electronic trading system 100, may rely on high-resolution clocks to maintain strict synchronization between its components. For this purpose, in some embodiments, one or more nodes 120, 140, 150 may be provided with access to a clock, such as a high-resolution global positioning (GPS) clock 195.

[0123] Referring to Figure 1A, for explanatory purposes thereafter, the gateway 120, compute node 140, and sequencer 150 connected within the mesh 172 may be referred to as a "mesh node." Figure 2 shows an exemplary embodiment of a mesh node 200 in the point-to-point mesh 172 architecture of the electronic trading system 100. The mesh node 200 may represent, for example, the gateway 120, sequencer 150, core compute node 140, etc. In this example, the functionality of the mesh node 200 is distributed across both hardware and software, but the mesh node 200 may be implemented in any appropriate combination of hardware and software, including purely hardware implementations or purely software implementations, and in some embodiments, the gateway 120 and / or compute node 140 and / or sequencer 150, or any or all of them, may be implemented with commercial off-the-shelf components.

[0124] In the embodiment shown in Figure 2, to achieve low latency, some functions are implemented in hardware in a Fixed Logic Device 230, while other functions are implemented in software in a device driver 220 and a mesh software application 210. The Fixed Logic Device 230 may be implemented in any suitable form, including application-specific integrated circuits (ASICs), embedded processors, and field-programmable gate arrays (FPGAs). The mesh software application 210 and the device driver 220 may be implemented as instructions executed on one or more programmable data processors, such as a central processing unit (CPU). Different versions or configurations of the mesh software application 210 may be installed on the mesh node 200 depending on its role. For example, different versions or configurations of the mesh software application 210 may be installed based on whether the mesh node 200 functions as a gateway 120, a sequencer 150, or a core compute node 140.

[0125] Any suitable physical communication link layer may be employed (including USB, Peripheral Component Interconnect (PCI)-Express, High Definition Multimedia Interface (HDMI), 10 Gigabit Ethernet (GigE), 25 Gigabit Ethernet, 40 Gigabit Ethernet, 100 Gigabit Ethernet, or InfiniBand (IB) via fiber or copper cables), but the mesh node 200 in this example is equipped with multiple low-latency 10 Gigabit Ethernet SFP+ connectors (interfaces) 270-1, 270-2, 270-3, ..., 270-n (collectively known as connector 270). Connector 270 may be directly connected to another node in the point-to-point mesh via a dedicated connection 180, and / or via a shared connection 182, and / or connected to a participant device 130 via a gateway 120. In this example, these connectors 270 are electronically coupled to 10GigEMAC cores 260-1, 260-2, 260-3, ..., 260-n (collectively known as the GigE core 260). In this embodiment, the 10GigEMAC cores 260 are implemented by fixed logic devices 230 to ensure minimal latency. In other embodiments, the 10GigEMAC cores 260 may be implemented by functions outside of the fixed logic devices 230, such as the functions of a PCI-E network interface card adapter.

[0126] In some embodiments, the fixed logic device 230 may include other components. The fixed logic device 230 in the example in Figure 2 further includes a fixed logic 240 component. In some embodiments, the fixed logic component 240 may implement different functions depending on the role of the mesh node 200, for example, whether the mesh node 200 is a gateway 120, a sequencer 150, or a core compute node 140. The fixed logic device 230 also includes fixed logic memory 250, which may be memory accessed with minimal latency by the fixed logic 240. The fixed logic device 230 further includes a PCI-E core 235 that may implement PCI Express functionality. In this example, PCI Express is used as a conductive mechanism for transferring data between hardware and software, specifically between the fixed logic device 240 and the mesh software application 210, via the PCI Express bus 233 by the device driver 220. However, any appropriate data transfer mechanism between hardware and software may be used, including direct memory access (DMA), shared memory buffers, and memory mapping.

[0127] In some embodiments, the mesh node 200 may also include other hardware components. For example, in some embodiments, depending on its role in the electronic trading system 100, the mesh node 200 may also include a high-resolution clock 195 (illustrated in Figure 1A; explained in relation to Figure 1A) used for high-resolution clock synchronization between nodes within the electronic trading system 100. The mesh node 200 may also include dynamic random-access memory (DRAM) 280 as additional memory that works in conjunction with the fixed logical memory 250. The DRAM 280 may be any suitable volatile or non-volatile memory, including one or more random-access memory banks, one or more hard disks, or one or more solid-state disks, and may be accessed via any suitable memory interface or storage interface.

[0128] (Deterministic latency)

[0129] As mentioned above, the architecture of system 100, by its very nature, supports another important objective of financial trading systems, namely, fairness. As a fundamental concept, the system's latency does not favor any single gateway 120 or core 140 user, thus minimizing the possibility that any participant device 130 connected to a particular gateway 120 could be unfairly advantageous or disadvantageous compared to other devices. This objective is achieved by controlling latency, that is, the time between when a message arrives within the system and when the corresponding message is allowed to leave the system.

[0130] Let's consider Figure 3A. Assume an inbound request message is received at gateway 120-1. Gateway 120-1 processes the request message and generates an internal message IBmsg destined for one or more cores 140. The IBmsg is the request message with one or more fields added, such as a time-based value (TBV). The TBV can be inserted into a given unused field in the standard message (as explained in relation to Figure 1C) or it can be encoded within the IBmsg according to a given proprietary internal protocol.

[0131] In one implementation, the time-based value TBV can be obtained in fixed logic 240 within gateway 120-1. This TBV may correspond to the time the request message was received by gateway 120-1 (referred to here as the reception timestamp or arrival time Tar), but in other implementations, it may correspond to the desired exit time of the corresponding response message sent back by system 100 (referred to here as the transmission time or exit time Tex). In that case, the TBV is obtained by adding some deterministic time delay value (referred to as Td, which will be detailed later) to the reception time Tar. Therefore, in a typical embodiment, this TBV is typically not the same field as the "node-specific timestamp" field 110-13 or the sequence number mentioned earlier (for example, it will be a different field from sequence ID 110-14 or reference sequence ID 110-15).

[0132] The inbound message IBmsg and its corresponding time-based value TBV are then forwarded to core 140 along path 310-1 within the mesh for processing. Typically, a predetermined core, such as core 140-2, then further processes the IBmsg to generate an outbound message OBmsg containing a given response data and TBV (or some other value dependent on TBV). The outbound message is then forwarded to gateway 120-1 along path 310-2. Gateway 120-1 then schedules the exit of the OBmsg from system 100 and sends it to participant 130-1 with an accurate exit time Tex. An element referred to as the outbound quality of service (QoS) shaper 320 controls the exit time of the outbound message OBmsg.

[0133] The exact exit time Tex is determined from the time-based value TBV held by IBmsg and OBmsg and the desired deterministic delay (i.e., latency Td). The deterministic delay Td does not necessarily directly depend on the actual amount of time it takes for any particular response to be sent back from core 140 to gateway 120. As will be detailed later, Td may depend on the maximum expected time required for any inbound message IBmsg to be fully processed in system 100.

[0134] Although it was explained that the exit time Tex can be determined from the TBV (using the arrival time Tar) at gateway 120, this exit time Tex can also be determined by other methods, such as at sequencer 150.

[0135] The time-based value TBV can take various forms. The time-based value TBV may simply be the time Tar at which the corresponding inbound request message is first received at gateway 120-1. In this case, the time delay value (Td) is added to the TBV to determine the departure time Tex at which the outbound message OBmsg leaves gateway 120-1 (arrive at). In configurations where the deterministic delay Td is a fixed time, this fixed value may be stored or implemented in gateway 120 (or sequencer 150) during configuration.

[0136] However, in other implementations, the time-based value TBV held by IBmsg and its corresponding OBmsg may be the actual desired time exit value Tex (i.e., an alternative to the arrival time Tar).

[0137] Furthermore, it should be noted that a given inbound message IBmsg is not necessarily forwarded to core 140. For example, if the checksum is invalid for reasons such as being invalid, or in other situations where gateway 120-1 cannot determine core 140 to which IBmsg is headed, the fixed logic within gateway 120 may reject the IBmsg. Therefore, it should be understood that the outbound message OBmsg may actually be sent from gateway 120-1 rather than from a given core 140. However, even in such situations, it may still be desirable for OBmsg to be sent at an exit time Tex, which corresponds to a deterministic delay Td after the original request message was received at time Tar.

[0138] In some cases, multiple receiving devices 130 may be destinations for an outbound message OBmsg. Even in such cases, OBmsg is sent to all of these multiple response consumers with exactly the same exit time Tex. As mentioned above, since the components of the system are typically implemented as highly synchronized hardware components, there is no doubt that messages can be sent to multiple recipients with exactly the same exit time.

[0139] Continuing to refer to Figure 3A, in one example, the inbound request message may originate from participant device 130-1, which is a buyer in the transaction. Participant device 130-2, connected to gateway 120-2, may be a seller in the same transaction, and participant device 130-3, connected to gateway 120-3, may be a service such as a market data feed reporting such transactions.

[0140] While these buyers, sellers, and market feeds are depicted as being connected to different gateways, it's important to understand that the configuration itself isn't crucial. All three participants might be connected to the same gateway 120-2, or the buyers and sellers might be connected to the same gateway 120-1, with the market data feed located on a different gateway, such as 120-3.

[0141] Depending on the type of message and the current load on the distributed system, the total time it takes for the system to send a response message (or process an inbound message) (i.e., the total time the message remains in system 100 from the time Tar the message enters the gateway until the corresponding response message is ready to be sent through the gateway) can be, for example, in the range of 400 to 800 nanoseconds. However, the outgoing QoS shaper 320 may be configured to send any corresponding response at strict intervals of 1000 nanoseconds (1 microsecond). That is, an outbound OBmsg message enters the outgoing QoS shaper 320-1 after it has reached gateway node 120-1 and is ready to be sent to trading participant 130-1. In one implementation, the OBmsg may remain in the outgoing shaper 320 for several hundred nanoseconds until exactly 1 microsecond has elapsed since its original request message was received at gateway 120-1. Thus, as long as the configured latency interval is set to a sufficiently high value (enough to allow the entire distributed system to comfortably and reliably process any inbound message within that interval), response messages such as OBmsg will always be reliably sent at a precise and definitive time interval from the receipt of the inbound message IBmsg.

[0142] Furthermore, the QoS level (i.e., deterministic latency interval Td) can be adjusted for each individual connection, each gateway, or for the entire system. In some embodiments, the QoS level can be optionally set for each participant or each connection to correspond to different subscriber billing levels. While allowing separate participants 130 to pay according to different QoS levels would mitigate the goal of fairness, such a configuration may be desirable for providers that rely on distributed services as an additional revenue stream.

[0143] Furthermore, the overall system QoS level can be set manually or dynamically. For example, system 100 may dynamically and temporarily increase the deterministic latency interval Td for the entire system 100 in the event of an exceptional event that could prevent the system from being satisfied with the typical latency interval Td. Such an event could be an internal event, such as the failure of one or more components or nodes in the distributed system, which would force the system to temporarily degrade performance until the problematic component can be hot-swapped or a new compute node can be brought online. Exceptional events may originate outside the distributed system, such as news events that cause a surge in activity on the distributed system or a surge in demand for the distributed system. Once the exceptional event is resolved, the latency interval Td can be dynamically reduced to the normal level through adjustment.

[0144] In some embodiments, a message from a single client 130 (market participant) may necessitate the generation and transmission of acknowledgment responses and other types of response messages to multiple participants 130 and / or via multiple participant connections. The content of multiple messages may be substantially identical among multiple participants, or an arbitrary inbound message IBmsg may generate multiple related but distinct response messages OBmsgs, each unique to one participant or to some participants.

[0145] For example, in many protocols, when a trade match occurs, a separate trade message may be generated for each party in the match. Each trade message contains party-specific information, such as an "order token" assigned by the client for order identification. The two trade messages are related to the extent that some information is common, such as a unique market allocation "trade ID" and "match number". The two trade messages may be generated by the same compute core 140, and it is desirable that their transmission time Tex be the same so that the two messages are sent simultaneously.

[0146] In other words, in some embodiments, regardless of whether the connections of multiple participants are on the same gateway or different gateways, two or more related response messages are sent simultaneously to multiple participants 130 at the same precise deterministic time interval. Each related response message should contain a TBV, which is the original timestamp value (or some other time-based value TBV related to the desired transmission time or corresponding reception time) corresponding to the triggered uplink message IBmsg that the response was based on. Thus, assuming that the internal clocks of each gateway are synchronized, the QoS shaper 320 of each gateway ensures that each related response is sent at the same precise time interval, and fairness is ensured among all clients 130 involved in receiving the related responses.

[0147] In some embodiments, the response from core 140 may be a single outbound message OBmsg arriving at one or more gateways 120. However, in this case, one or more gateways 120 will generate two or more related but non-identical messages. These related messages differ in their destination addresses and / or in some other client-specific identifier, such as an order token, within their respective encoding information as a result of the protocol corresponding to participant 130.

[0148] Core 140 may provide a matching engine for a financial trading system that can generate matches between buy orders for securities and corresponding sell orders for the same securities. Even in the typical case where these two orders are placed by two separate parties at different times, a successful match results in the sending of acknowledgment and execution messages to both counterparties in the match (e.g., buyer and seller). The execution messages (OBmsg) sent to each counterparty contain information specific to that particular client's trade, and fairness is ensured by sending these execution messages to both parties exactly simultaneously. In this example, the latency interval between the two resulting related responses may be set based on the upstream timestamp of one of the orders involved in the match (e.g., the order that arrived last as an inbound request message).

[0149] In another example of a related implementation of a matching engine in a financial system, a client device 130 may typically subscribe to a stream of market data messages transmitted from the matching engine, and the subscriber user is informed of real-time market-wide activity, including activities in which it is not a party, such as the latest trades between two third parties. In some embodiments, such market data messages are sent simultaneously to all subscriber users based on a timestamp value in the message processed by the sending QoS shaper 320. In some embodiments, the transmission time of the market data message to the subscriber user is set to coincide with the transmission time of acknowledgment messages or trade messages sent to market participants who are direct parties to the activity reflected in the market data message sent to the subscriber user. Returning to the example of matching two market participants (buyer and seller), a pair of trade messages OBmsg would be sent to both counterparty parties, and this matching activity may also be reported to the market data subscriber user.

[0150] In some embodiments, the system 100 ensures that not only are both acknowledgment messages sent simultaneously to the other parties, but that the acknowledgment message is also sent simultaneously to market data subscribers as a market data message reflecting this execution, thereby ensuring that no market participant gains access to useful financial information prematurely.

[0151] More generally, system 100 has precise control over when outbound messages are sent. In some cases, response messages to multiple recipients may be sent simultaneously, prioritizing fairness to all. In other embodiments, the system may be configured to send responses to participants 130-1 and 130-2, who are parties to a transaction, slightly earlier than the market feed 130-3. Alternatively, the market feed 130-3 may receive responses earlier than participants 130-1 and 130-2. In yet another configuration, system 100 rewards participant 130, who is a "liquidity asset adder," for early notification, while "liquidity removers" are notified later.

[0152] Figure 3B shows an example of an outgoing QoS shaper 320 that may be installed inside one or more gateways 120. Each OBmsg is received along with its corresponding time-based value TBV. The exit time Tex, at which the message is sent out of gateway 120, is then determined by using the TBV together with a desired latency Td. More specifically, when the TBV is the reception time of the corresponding inbound message IBmsg, Tex can be determined (arrive at) by adding the corresponding deterministic delay Td to the TBV.

[0153] The outgoing QoS shaper 320 may be implemented as a "packet scheduler" organized into a series of indexed storage locations 380-1, 380-2, ..., 380-n (i.e., "buckets"), where each location 380 is associated with a discrete high-precision outgoing timing interval Tex1, Tex2, ..., Texn. Each message placed in a particular bucket 380 is emitted at the precise time interval associated with that bucket. Each bucket may contain zero or one or more sets of outbound messages OBmsg.

[0154] As mentioned earlier, in one alternative implementation, instead of assigning an inbound timestamp as a TBV corresponding to each incoming message in the gateway, the TBV may be allocated as a desired outbound indexing location (i.e., bucket) for the packet scheduler 320. Thus, when the desired exit time is used as the TBV, the TBV can be considered to be directly correlated with the exit time Tex.

[0155] The message scheduler 320 can be a winding ring data structure, but other implementations are also possible. For example, the message scheduler 320 may be implemented as a series of linked lists using pointers, for instance, with one linked list for each departure time Tex. Note that the "time" values, whether arrival or departure times, can be based on either absolute or relative time.

[0156] Figure 4 shows an example where an inbound message IBmsg arrives at gateway 120 from market participant 130, and a reply is sent as an outbound message OBmsg with a deterministic latency (e.g., Td) of 1000 hours (where the hour unit can be nanoseconds).

[0157] In the initial state 401, a request message arrives at the gateway 120, a timestamp TBV is added to it, and an internal message D1 (e.g., IBmsg) is generated. This state 401 may occur at time T17356. Next, at time T17358, message D1, including the timestamp TBV, is sent by the gateway 120 to a given core 140. Subsequently, in state 403, such as at time T17918, a response message (R1) returned from the core 140, including the same timestamp TBV, is received by the gateway 120. Next, in state 404 (which may occur at time T17922), the internal response message R1, including the timestamp, is output to the packet scheduler 320 (fed), and an appropriate downstream time slot (in this example, arrival time T17356+1000, i.e., time T18356) is determined. Finally, in state 405, when the time reaches exactly T183568, the packet scheduler allows the response message (R1) from the gateway to exit as a system-level outbound message OBmsg.

[0158] In the example in Figure 4, an inbound message IBmsg and its TBV are depicted as being forwarded directly from the gateway node 120 to a given core 140 (for example, via the activation link 180-1-1 in Figure 1B), but it should be understood that other configurations are also possible. As an alternative example, an IBmsg containing a TBV may travel to a given core 140 via the sequencer 150 (for example, via the ordering path 117 in Figure 1B), or it may travel along both paths (for example, directly from the gateway node 120 to the core 140 and from the gateway node 120 to the core 140 via the sequencer 150). In embodiments where an IBmsg containing a TBV travels via the sequencer 150, the TBV may be included in both unmarked messages and sequence-marked messages. In such embodiments, the TBV included in an unmarked message may be the same value as the TBV included in the corresponding sequence-marked message. Therefore, as mentioned above, the TBV is a field within the message that is different from the sequence identifier of that message (for example, different from sequence ID 110-14 or reference sequence ID 110-15).

[0159] Figure 5 shows an exemplary embodiment in which an outbound message OBmsg arises from an internal event in System 100, rather than being a response to any inbound "request" message. In this example, OBmsg is "asynchronous" with respect to any corresponding inbound message. It is desirable that such asynchronous messages also be marked and processed with a scheduled delivery time Tex.

[0160] An example of asynchronous messaging in a trading system might be a timer-triggered order cancellation. The matching engine within the core may generate such a cancellation message when the "expiration date" associated with an inactive order expires. Since this cancellation message is delivered to multiple clients / participants (e.g., both the order initiator and the market data feed), it needs to exit simultaneously from all gateways to all destination participants.

[0161] In this case, the exit time Tex may be determined by the core 140 that generated the message. In other implementations, the exit time Tex may be determined by one of the sequencers 150 through which the OBmsg passes.

[0162] In other words, outbound messages that need to be sent to multiple participants are sent with a common exit time Tex, as part of ensuring deterministic latency, regardless of whether the outbound message is generated as part of a request-response or is an asynchronous message generated within the mesh.

[0163] Furthermore, it should be understood that the concept of deterministic latency includes more than just fixed time. In other implementations, deterministic latency may consist of a set of fixed time values ​​uniformly distributed within a range that conforms to a fairness standard. The specific series of time values ​​may be statistically uniform and randomly distributed within that range.

[0164] In other words, system 100 also supports deterministic latency changes, which involve randomizing the order of latency values ​​Td within a uniform-length latency range. In such embodiments, while timestamps are associated with inbound messages IBmsg in roughly the same way as described above, the difference is that the outgoing QoS shaper assigns the message to a time slot that depends on the latency values ​​randomized within the boundary range.

[0165] In such embodiments, the outgoing QoS shaper 320 can be thought of as assigning downlink time slots in a manner similar to dealing cards from a deck of playing cards, and once a latency value is assigned within a range, that particular latency value will not be reassigned until all other values ​​within that range have been assigned and the "deck of cards" has been shuffled again. In one embodiment, the QoS shaper 320 is responsible for assigning sequentially randomized latency values, while in other embodiments, other components such as the gateway 120 or sequencer 150 may be responsible for assigning these values ​​when an inbound message is received.

[0166] In one implementation where the deterministic delay Td, which is variable but also deterministic, follows some predetermined pattern, the Td for each inbound message IBmsg may not necessarily be the same. Therefore, it may be desirable that the Td value assigned to each message, along with its corresponding acceptance time Tar, be carried as a "time-based value" TBV for each message. This allows the gateway 120 to determine the exact exit time Tex at the transmission time of one or more corresponding outbound messages OBmsg. However, in other embodiments, the variable but deterministic delay Td may be added to the arrival time Tar just at the acceptance time, so that the TBV is a single value Tex. That is, in the case of a variable delay, using the exit time Tex for the TBV eliminates the need to maintain both the delay value Td and the acceptance time Tar for each message.

[0167] Figure 6A shows an example in which the system assigns a deterministic latency (Td) in 2000-hour increments, selected from a deterministic range of 1000 to 5000 hours, to a specific message. Similar to the example in Figure 4, the inbound message arrives at gateway 120-1 at time T17356, resulting in state 602. This message is timestamped with TBV=17356. In state 603, the inbound message containing this timestamp is sent by the gateway to the core (time T17358). In state 604 (time T17918), a response message returned from core 140, containing the same timestamp, is received by the gateway. In state 605 (time T17922), this response message containing the timestamp is queued in the packet scheduler within gateway 120-1. Then, in state 606 (time T19356), one or more response messages are sent from the gateway, with a latency of 2000 hours assigned to this particular message from a range of values ​​within the range of 1000 to 5000 hours. The specific latency value assigned to the message (i.e., its location in the packet scheduler) may be determined by the gateway when the message is received, or by other means as described elsewhere.

[0168] Figure 6B shows a similar example, but here the system determines that the deterministic delay assigned to the message is 3500 hours (selected from the same range of 1000 to 5000 units). Although the deterministic delay is different, the other processing is the same as in Figure 6A. In starting state 612, the message arrives at the gateway at time T17356 and is timestamped. In state 614 (time T17358), the message leaves the gateway and is forwarded to the core. In state 616 (time T17918), the core sends a response message containing the timestamp back to the gateway. Next, in state 618 (time T17922), the response message containing the timestamp is placed in the packet scheduler queue and does not leave the gateway until state 620 (time T20856).

[0169] Figure 7 shows an example of how a pattern, such as a sequence of randomized values, is assigned to deterministic latency. The boundary range is 1000 to 5000 hours, and the increment between slots is 500 hours. Suppose the QoS shaper generates a sequence of nine uniformly distributed Tex values: 2000, 3500, 2500, 1500, 4000, 5000, 4500, 3000, and 1000. Once the first sequence of numbers is exhausted, the QoS shaper repeats this process, preferably generating another uniformly distributed random pattern of the nine values.

[0170] However, it is also possible to arrive at a deterministic delay using other randomization methods, including concepts similar to multiple decks of playing cards, or even by using a standard pseudorandom number generator.

[0171] Latency value ranges for messages can be assigned system-wide, per matching engine, per account / participant, or per flow / connection. In some cases, it may be desirable that the latency distribution is not completely random or that the intervals are not perfectly uniform across the entire latency range. For example, it may be desirable to limit the number of consecutive low latency values ​​and / or consecutive high latency values. In other words, the latency assignment method within the QOS shaper 320 may aim to distribute latency uniformly even within a relatively small number of latencies, for example, by ensuring that for every five consecutive latencies assigned to a participant's response, at least two latencies are in the high latency range and the remaining two latencies are in the low latency range. Depending on the application, during extremely congested periods, it may be highly desirable to limit the number of consecutive similar (i.e., all high or all low) latencies. For example, activity levels tend to be high around market openings and closings, as well as after news reports that could have a financial impact, such as interest rate adjustment announcements. In such situations, it may be necessary to control the latency distribution more strictly than usual.

[0172] In selecting the numerical range, the inherent delays expected for the components of the system, including at least the gateway 120 and core 140, must be taken into consideration. For example, depending on the type of message and the current load on the distributed system, the time it takes to process a message may be in the range of 400 to 800 nanoseconds, while the outgoing QoS shaper may be configured to send any corresponding response at randomized intervals in the range of 1000 to 5000 nanoseconds. That is, the response message enters the outgoing QoS shaper after reaching the gateway node and being ready to be sent to the trading participant, and remains there for a period until the elapsed time since the original message was timestamped upon entering the gateway node reaches exactly the randomized interval. The configuration of adding randomized but uniformly distributed jitter to the latency ensures fairness because any randomized jitter within the set latency boundary is assigned to each trading message with equal probability. Furthermore, the addition of jitter helps deter participants from exploiting the predictability of perfectly consistent latency.

[0173] Furthermore, it should be understood that many of the concepts previously described regarding fixed latency can also be applied to this embodiment. For example, it may be desirable to send all related response messages triggered by a single upstream participant message simultaneously. Also, it may be desirable to be able to adjust the set of QoS levels for each participant / connection or for the entire system.

[0174] Figure 8 illustrates some of the considerations that may be taken into account when determining an appropriate fixed value for Td. It is desirable that Td be selected to be greater than the sum of the worst-case delay through core 140 (Tcore) and the worst-case delay of the path between gateway 120 and core 140 (Tpath) (for both inbound and outbound messages).

[0175] Specifically, some of the considerations for determining the Td for a synchronous message (i.e., when an outbound message is a response to an inbound message) may include the following: - Gateway acceptance time - Transmission time from gateway to core - Time spent inside the core - Transmission time between core and gateway, or - Gateway transmission time.

[0176] Similarly, considerations for determining the Td for asynchronous messages (messages that are not specific responses to inbound messages) may include the following: - Time spent inside the core - Transmission time between core and gateway, or - Gateway transmission time.

[0177] In one exemplary embodiment, since the core 140 is expected to be largely implemented in hardware (e.g., a fixed-logic FPGA-based system), the "time spent in the core" for task execution will be relatively constant and predictable. However, it should be understood that some tasks performed on the core 140 may take longer than others. For example, the response of the core 140 to a cancellation request may be much faster than a response to an order addition. Even between different order addition requests, the processing time required may differ depending on whether the order is for a frequently traded security (and therefore the data necessary to execute the trade is held in a local cache of the core 140 dedicated to that security) or for a less frequently traded security (in which case the data necessary to execute the order is retrieved from a location outside the local cache of the core 140). Therefore, the type of message to be sent may also be considered in the selection of Td.

[0178] In an architecture using a mesh where all gateways 120 and cores 140 are fully connected, the path-related latency Tpath is also expected to cluster around a relatively small and predictable value, as mentioned above.

[0179] Therefore, with the goal of selecting time Td in a way that "hides" variability such as Tcore and Tpath from the market via participant devices 130, it is desirable that Td be selected to be greater than the sum of such internal system delays. As long as the worst-case Td set is greater than or equal to the assumed (possible) maximum value of system delay, system 100 can ensure deterministic latency for all participant devices 130.

[0180] Td may be determined during the design of the system. However, in an alternative approach, Td can be dynamically adjusted by monitoring the time difference between an inbound message and its corresponding outbound message, ensuring that Td is always greater than the maximum time difference (Δ). In this embodiment, the maximum latency time of the system can be determined by placing the system 100 in a special mode to receive a series of test request messages (various additional orders, cancellations, etc.), and recording (noting) the response time for each of the participant connections, gateways, and core sets.

[0181] Figure 9 is similar to Figure 8, but depicts a situation where deterministic latency is uniformly distributed within a certain numerical range. In this example, there are multiple possible delay time candidates (Td1, Td2, Td3, ..., Tdn). It is preferable that all of these selection times are greater than the expected worst-case internal delay, for example, the sum of Tcore and Tpath.

[0182] As mentioned above, some implementations of system 100 can support participants using different financial transaction protocols. That is, the processing of such messages will be faster or slower depending on the protocol chosen. For example, inbound request messages encoded using a binary protocol will be processed faster on core 140 than requests using a text-based protocol.

[0183] In such a system, to provide definitive and fair operation for all participants, it is desirable that the latency be the same regardless of the protocol used. Therefore, it is desirable that Td be selected to accommodate such variability.

[0184] In the example in Figure 10, gateway 120-1 receives an IBmsg from a first participant 130-1 (e.g., a buyer) using a text-based financial transaction protocol. A different participant 130-2 (e.g., a seller) may use a binary financial transaction protocol when sending an IBmsg to gateway 120-2. The time (P1) required for the system to process the first participant's IBmsg and send back a corresponding response (OBmsg) may be longer than the time (P2) required for the system to process the IBmsg for the second participant and send back a response. Therefore, system 100 may be set to a sufficiently long time Td, as described above, to take into account the processing time required for the slowest financial transaction protocol. This ensures that the response time is the same regardless of which protocol the message is exchanged according to. Alternatively, the system may be configured to process different protocols differently and to encourage or discourage the use of a particular protocol by giving the message a deterministic time boost or deterministic time penalty depending on the protocol of the message.

[0185] In other words, by adjusting Td (presumably per connection) with a value that depends on further parameters (e.g., P1, P2, etc.) such as the financial transaction protocol used, the outbound message OBmsg being sent back is scheduled to exit system 100 at a time that depends on both the reception time (e.g., timestamp, TBV, etc.) and a protocol-dependent value. In this case, (Such) IBmsg is forwarded to the core along with a protocol-dependent time value (P) carried with the message and TBV. This allows the gateway to determine the time for the outbound message OBmsg to exit, taking into account the message protocol. In this way, gateways 120-1, 120-2 can calculate separate transmission times for market participants 130-1, 130-2, depending on the message protocol used by each market participant 130-1, 130-2.

[0186] However, parameters such as P1 and P2 may be based on considerations other than the protocol being used. For example, these parameters may be used to differentiate service tiers, such as ensuring that participants paying for "Tier 1" services experience similar latency, but not as much as "Tier 2" users who pay lower rates. In other words, the concept that system 100 "uses deterministic latency to be a fair system for participants" can have various meanings. It may mean "treating all participants exactly the same," it may mean encouraging certain transactional behaviors or protocols, or it may be used for security reasons.

[0187] Furthermore, deterministic latency may be temporarily adjusted based on current system conditions. For example, deterministic latency may be increased due to the failure of one or more components, or as a result of an unusual burst of activity within the system.

[0188] (Other use cases)

[0189] The aforementioned architecture may be used for purposes other than electronic trading systems. For example, in addition to processing securities trading orders, it can be used for monitoring data streams flowing through a network, capturing packets, decoding raw packet data, real-time analysis of packet content, and providing responses.

[0190] (Summary of the embodiments)

[0191] Those skilled in the art will understand that various implementations are possible for the aforementioned methods and systems.

[0192] For example, one approach may involve multiple gateways connected to receive inbound messages from two or more participant devices. Each of these gateways is further configured to determine a time-based value (TBV) for a given inbound message. The gateway forwards the given inbound message, along with its TBV, to one or more compute nodes, which then receive a response message from the compute nodes containing information derivable from the TBV. The gateway then sends the response message to one or more of the participant devices as an outbound message, transmitted with a deterministic delivery time that depends on both the information derivable from the TBV and deterministic latency.

[0193] One embodiment of the method involves sending a response message with deterministic latency. Multiple uplink messages are received by one or more gateways from multiple participant devices. A predetermined gateway then determines a corresponding time base value for a predetermined uplink message among the multiple uplink messages. The predetermined uplink message relates to an electronic trading function. Next, the predetermined gateway transmits the predetermined uplink message and information that can be derived from the corresponding time base value to a sequencer node. Subsequently, the sequencer node transmits a sequence-marked message and the information that can be derived from the corresponding time base value to at least one predetermined compute engine among a plurality of compute engines. The predetermined compute engine among the plurality of compute engines receives the sequence-marked message and the information that can be derived from the corresponding time base value and determines a predetermined compute response message. The predetermined compute response message is based on the sequence-marked message and is further configured to complete an electronic trading matching function. Subsequently, the predetermined compute engine sends back to at least the predetermined gateway the predetermined compute response message and the information that can be derived from the corresponding time base value. The predetermined gateway then receives the predetermined compute response message from the predetermined compute engine among the plurality of compute engines and the information that can be derived from the corresponding time base value, and transmits the predetermined compute response message to one or more participant devices among the plurality of participant devices at a definitive transmission time that depends at least on the information that can be derived from the corresponding time base value for the predetermined uplink message.

[0194] In some embodiments, the predetermined gateway among the plurality of gateways may receive another uplink message from another participant device. In the same embodiment, the predetermined gateway further forwards another transfer message to one or more of the plurality of compute engines, which includes the other uplink message and information derivable from a different time base value. The predetermined gateway among the plurality of gateways receives another compute response message from one or more of the plurality of compute engines, which includes the information derivable from the different time base value. The predetermined gateway then delays the transmission of the other compute response message until it reaches a different deterministic transmission time that depends at least on the information derivable from the different time base value. The predetermined gateway then transmits the other compute response message to the other participant device at the different deterministic transmission time, such that the predetermined compute response message and the other compute response message are each delayed by the deterministic latency.

[0195] The time base value may depend on the time relating to at least one of the reception time of the predetermined uplink message and the desired transmission time of the predetermined compute response message.

[0196] The deterministic latency may depend on the maximum time it takes for a predetermined compute engine among the plurality of compute engines to return the compute response message. The deterministic latency may follow a pattern that is both variable and deterministic. The deterministic latency may be selected from a set of latencies uniformly distributed within a predetermined range. Furthermore, the deterministic latency may be set per gateway, per connection, or for the entire system.

[0197] In some embodiments, the transfer from the predetermined gateway to the compute engine may be performed via one or more dedicated direct connections from the predetermined gateway to the plurality of compute engines.

[0198] A given asynchronous message from the compute engine may be received by at least the predetermined gateway and subsequently sent simultaneously as an outbound message to two or more participant devices.

[0199] The elapsed time (transit time) obtained in the process (step) of transferring the predetermined uplink message and the process (step) of receiving the predetermined compute response message can be determined, and the definitive transmission time also depends on this elapsed time.

[0200] The transmission of the predetermined compute response message from the predetermined gateway may further include transmitting the predetermined compute response message to one or more other participant devices at the definitive transmission time.

[0201] The aforementioned definitive transmission time may be one of a plurality of definitive transmission times. In that case, the predetermined compute response message may be stored in one of a series of indexing locations associated with each of the plurality of definitive transmission times.

[0202] The information, which can be derived from the corresponding time base value, is inserted into an unused field in the predetermined uplink message, after which the predetermined uplink message can be forwarded to the plurality of compute engines.

[0203] The corresponding time base value may be part of the system's internal protocol field within the predetermined uplink message.

[0204] In some embodiments, the deterministic latency can change dynamically.

[0205] Two or more of the aforementioned gateways may each receive the predetermined compute response message from the predetermined compute engine.

[0206] In some embodiments, the predetermined gateway may forward the predetermined uplink message containing the information derivable from the corresponding time base value to the plurality of compute engines. The predetermined compute engine among the plurality of compute engines then receives the predetermined uplink message containing the information derivable from the corresponding time base value. Here, the determination of the predetermined compute response message by the predetermined compute engine among the plurality of compute engines may further be based on at least one of the predetermined uplink message and the sequence-marked message. In this embodiment as well, the forwarding of the predetermined uplink message and the reception of the predetermined compute response message may be performed by a plurality of direct connections provided between each of the plurality of gateways and each of the plurality of compute engines.

[0207] Optionally, the plurality of gateways may be further configured to receive asynchronous messages from at least one of the plurality of compute engines, and then simultaneously send the asynchronous messages as outbound messages to two or more of the participant devices.

[0208] The predetermined compute response message may relate to a transaction match event between two participant devices and two matching parties associated with each device. In this embodiment, the predetermined gateway may simultaneously send the predetermined compute response message to the two matching parties and the two participant devices corresponding to them.

[0209] The predetermined compute response message may be transmitted as a market data event message to a third device associated with a subscriber to the market data stream.

[0210] In some embodiments, the definitive transmission time may depend on at least one of the message path delay and the compute engine delay.

[0211] Other embodiments may include a system comprising a plurality of compute engines, one or more gateways, and a sequencer node. The one or more gateways are configured to (a) receive a plurality of uplink messages from a plurality of participant devices, (b) determine a corresponding time base value for a predetermined uplink message among the plurality of uplink messages, (c) forward the predetermined uplink message to the plurality of compute engines along with information that can be derived from the corresponding time base value, and (d) transmit the predetermined uplink message to the sequencer node. The sequencer node may be configured to (e) receive the predetermined uplink message and (f) transmit a sequence-marked message to a predetermined compute engine. The predetermined compute engine may be configured to (g) receive the predetermined uplink message and the information derivable from the time base value from at least the predetermined gateway, (h) receive the sequence marked message from the sequencer node, (i) determine a predetermined compute response message configured to complete the electronic transaction matching function based on at least one of the predetermined uplink message and the sequence marked message, and (j) send back the predetermined compute response message and the information derivable from the corresponding time base value to one or more of the gateways. At least the predetermined gateway may be further configured to (k) receive the predetermined compute response message and the information derivable from the time base value from the predetermined compute engine, and (l) send the predetermined compute response message to one or more of the plurality of participant devices in a deterministic transmission time that depends at least on the information derivable from the corresponding time base value for the predetermined uplink message.

[0212] In such a system, the deterministic transmission time may follow a pattern that is both variable and deterministic.

[0213] In such a system, the predetermined gateway may be further configured to forward the uplink messages and receive the predetermined compute response messages via a plurality of direct connections provided between each of the plurality of gateways and each of the plurality of compute engines.

[0214] Furthermore, such a system may be configured such that the predetermined compute response message relates to a transaction match event between two participant devices and two matching parties associated with each of them, and the predetermined gateway may be further configured to simultaneously send the predetermined compute response message to the two matching parties and the two participant devices corresponding to them.

[0215] Furthermore, a possible configuration for sending response messages with deterministic latency may include the following:

[0216] The process of receiving multiple inbound messages from multiple participant devices at multiple gateways;

[0217] A process for determining a corresponding time base value for a predetermined uplink message among the plurality of uplink messages using a predetermined gateway,

[0218] The process in which the aforementioned predetermined uplink message relates to the electronic transaction matching function;

[0219] The process of transferring the predetermined uplink message, along with information derivable from the corresponding time base value, to multiple compute engines via the predetermined gateway;

[0220] The process of transmitting the predetermined uplink message to the sequencer node via the predetermined gateway;

[0221] The process by which the sequencer node sends a sequence-marked message to one or more of the compute engines;

[0222] A process in which the plurality of compute engines receive, at least from the predetermined gateway, the predetermined uplink message and the information that can be derived from the corresponding time base value;

[0223] The process of receiving the sequence-marked message using a predetermined compute engine among the plurality of compute engines;

[0224] A process of determining a predetermined compute response message, configured to complete the electronic transaction matching function, using one of the plurality of compute engines, based on at least one of the predetermined uplink message and the sequence-marked message;

[0225] A process by which at least one of the plurality of compute engines sends back to at least one of the predetermined gateways the predetermined compute response message and the information that can be derived from the corresponding time base value;

[0226] A process of receiving, at least at the predetermined gateway, the predetermined compute response message from at least one of the plurality of compute engines, and the information that can be derived from the corresponding time base value;

[0227] A process of delaying the predetermined compute response by at least the predetermined gateway until a deterministic transmission time is reached that depends at least on the deterministic latency and the information that can be derived from the corresponding time base value; and

[0228] A process of transmitting the predetermined compute response message to one or more participant devices among the plurality of participant devices at a predetermined time using at least the predetermined gateway.

[0229] Furthermore, the system may have the following configuration:

[0230] Multiple compute engines;

[0231] It comprises multiple gateways configured to receive multiple uplink messages from multiple participant devices;

[0232] The specified gateway is

[0233] For a predetermined uplink message among the aforementioned multiple uplink messages that relates to the electronic transaction matching function, the corresponding time base value is determined.

[0234] The predetermined uplink message is transmitted to the plurality of compute engines along with information that can be derived from the corresponding time base value.

[0235] It is further configured to forward the predetermined uplink message to the sequencer node,

[0236] The aforementioned sequencer node is

[0237] It is further configured to send a sequence-marked message to one or more of the compute engines,

[0238] A predetermined compute engine among the plurality of compute engines is:

[0239] At least the predetermined gateway receives the predetermined uplink message and the information that can be derived from the corresponding time base value,

[0240] The sequencer receives the sequence-marked message from the sequence node,

[0241] A predetermined compute response message configured to complete the electronic transaction matching function is determined based on at least one of the predetermined uplink message and the sequence-marked message.

[0242] The system is configured to send back at least the predetermined gateway the predetermined compute response message and the information that can be derived from the corresponding time base value.

[0243] The aforementioned predetermined gateway is

[0244] At least the predetermined compute engine receives the predetermined compute response message and the information that can be derived from the corresponding time base value,

[0245] The predetermined compute response message is delayed until a deterministic transmission time is reached that depends at least on the deterministic latency and the information that can be derived from the corresponding time base value for the predetermined uplink message.

[0246] The system is further configured to send the predetermined compute response message to one or more of the plurality of participant devices at the definitive transmission time.

[0247] (Other implementation / realization options)

[0248] It should be understood that the exemplary embodiments described so far can be realized in a number of different ways. Depending on the context, various "data processors" can be realized by a physical or virtual general-purpose computer, each comprising a central processor, memory, disk or other mass storage device, one or more communication interfaces, one or more input / output (I / O) devices, and other peripheral devices. For example, by loading software instructions into the processing unit and executing those instructions to perform the aforementioned functions, the general-purpose computer transforms into a processor and performs the aforementioned processing.

[0249] As is known in the art, such a computer may have a system bus, which is a set of hardware lines used for data transmission between components of the computer or processing system. One or more buses are essentially one or more shared conduits that connect different components of the computer system (e.g., one or more central processing units, disks, various types of memory, input / output ports, network ports, etc.) to enable the transmission of information between those components. One or more central processing units are attached to the system bus to execute computer instructions. Typically, input / output device interfaces are also attached to the system bus to connect the disks, memory and various types of input / output devices. One or more network interfaces enable connections to various other devices attached to the network. One or more memories store volatile and / or non-volatile computer software instructions and data used to implement embodiments. Disks and other mass storage devices store non-volatile computer software instructions and data used, for example, to implement the procedures described herein.

[0250] Therefore, embodiments can typically be implemented using hardware, custom-designed semiconductor logic, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, software, or any combination thereof.

[0251] In certain embodiments, the procedures, apparatus, and methods described herein constitute a computer program product. The computer program product includes a computer-readable medium (e.g., a removable storage medium such as one or more DVD-ROMs, CD-ROMs, diskettes, tapes, etc.) that provides at least a portion of the software instructions of the system. The computer program product can be installed by any suitable software installation method, as is well known in the art. In other embodiments, at least a portion of the software instructions may be downloaded via cable and / or communication and / or wireless connections.

[0252] Furthermore, embodiments may be implemented as instructions stored in a non-transient machine-readable medium that can be read and executed in one or more steps. A non-transient machine-readable medium may have any mechanism for storing or transmitting information in a format readable by a machine (e.g., a computing device). For example, non-transient machine-readable media may include read-only memory (ROM), random-access memory (RAM), storage consisting of magnetic disk storage media, optical storage media, flash memory devices, and the like.

[0253] Furthermore, this specification may describe firmware, software, routines, and instructions as if they were performing specific processes and / or functions. However, such descriptions included herein are for convenience only, and it should be understood that in reality, such processes occur when computing devices, processors, controllers, and other equipment execute the firmware, software, routines, and instructions.

[0254] Furthermore, please understand that block diagrams and system diagrams may have an increased or decreased number of components, different layouts, and can be represented in different ways. However, please also understand that depending on the implementation, the block diagrams and network diagrams may be predetermined, and the number of block diagrams and network diagrams representing the implementation of the embodiment may also be predetermined.

[0255] In other words, further embodiments may be realized by various computer architectures and / or physical computers and / or virtual computers and / or cloud computers and / or given combinations thereof. Therefore, the computer systems described herein are intended for illustrative purposes only and do not limit the embodiments.

[0256] In the above description, exemplary embodiments have been illustrated in detail, but those skilled in the art will understand that various modifications to the form and details may be made, as long as they do not exceed the legal scope of this patent as encompassed in the attached claims. Furthermore, the present invention includes the following embodiments. [Aspect 1] A system comprising multiple gateways connected to receive inbound messages from two or more participant devices, wherein one or more of the gateways are: For a given inbound message, calculate the time-based value (TBV), The predetermined inbound message, along with its TBV, is forwarded to one or more compute nodes. A response message containing information that can be derived from the TBV is received from one or more compute nodes. To one or more of the participant devices, a response message is sent as an outbound message with a deterministic transmission time that depends on both the information derivable from the TBV and the deterministic latency. The system is further structured in this way. [Aspect 2] A system according to Embodiment 1, wherein the TBV depends on a timestamp relating to the reception time of the predetermined inbound message. [Aspect 3] A system according to Embodiment 1, wherein the TBV depends on a desired transmission time of the response message. [Aspect 4] In the system described in Embodiment 3, further, A packet scheduler configured to receive the aforementioned response message, having a series of indexed locations, each associated with a desired transmission time, A system comprising, wherein the TBV is a value that depends on the value of the associated indexed location for the desired delivery time. [Aspect 5] A system according to Embodiment 1, wherein the TBV is inserted into an unused field in the inbound message and then forwarded to a given compute node. [Aspect 6] A system according to Embodiment 1, wherein the TBV is added as part of a field of the system's internal protocol within the inbound message. [Aspect 7] A system according to Embodiment 1, wherein the information that can be derived from the TBV includes the TBV. [Aspect 8] A system according to Embodiment 1, wherein the deterministic latency depends on the maximum time it takes for the gateway to receive the response message from one or more compute nodes. [Aspect 9] A system according to Embodiment 1, wherein the deterministic latency follows a pattern that is variable but deterministic. [Aspect 10] A system according to Embodiment 1, wherein the deterministic latency is selected from a series of latencies uniformly distributed within a predetermined range. [Aspect 11] A system according to Embodiment 1, wherein the deterministic latency is set for each gateway, for each connection, or for the entire system. [Aspect 12] A system according to Embodiment 1, wherein the deterministic latency changes dynamically depending on system conditions. [Aspect 13] A system according to Embodiment 1, wherein two or more gateways each receive the response message from the compute node. [Aspect 14] A system according to Embodiment 1, wherein the predetermined inbound message is forwarded to one or more compute nodes, and the response message is received from one or more compute nodes via a plurality of direct connections provided between each of the one or more gateways and each of the one or more compute nodes. [Aspect 15] In the system described in Embodiment 1, the response message relates to a transaction match event between two participant devices and two matching parties associated with each device, and the gateway is The response message is sent simultaneously to the two participant devices as an outbound message within the specified transmission time. The system is further configured in this way. [Aspect 16] A system according to embodiment 15, wherein the outbound message is also simultaneously transmitted at the same time as a market data event message to a device associated with a subscriber to the market data stream. [Aspect 17] In the system described in Embodiment 1, the one or more gateways are A given compute node receives an asynchronous message, The asynchronous message is sent simultaneously to two or more participant devices as an outbound message within the specified transmission time. The system is further configured in this way. [Aspect 18] A system according to Embodiment 1, wherein the TBV further depends on a time value relating to at least one of the message path delay and compute node delay. [Aspect 19] In the system described in Embodiment 1, the one or more gateways The predetermined inbound message, along with its TBV, is forwarded to one or more sequencer nodes. The system is further configured in this way. [Aspect 20] In the system described in Embodiment 1, further, The one or more compute nodes mentioned above are: The system is configured to receive the predetermined inbound message, along with its TBV, from one or more gateways. A system in which one or more compute nodes are further configured to send the response message back to one or more gateways along with the information that can be derived from the TBV. [Aspect 21] In the system described in Embodiment 1, the one or more gateways are further configured to forward the predetermined inbound message, along with its TBV, to one or more sequencer nodes. A system in which one or more sequencer nodes are configured to forward the predetermined inbound message, along with its TBV, as a sequence-marked message to one or more compute nodes. [Aspect 22] The system according to embodiment 2, wherein one or more compute nodes are further configured to determine the response message based on at least one of the predetermined inbound message and the sequence-marked message. [Aspect 23] The system according to Embodiment 1, wherein the predetermined inbound message relates to an electronic transaction matching function, and the response message is configured to complete the electronic transaction matching function. [Aspect 24] In a method for transmitting response messages from an electronic device with deterministic latency, The process of receiving uplink messages from participant devices, Regarding the aforementioned uplink message, the process for determining the time-based value (TBV) is as follows: A process of sending the uplink message and the transfer message including the TBV to one or more compute engines, The process of receiving a compute response message from one or more compute engines, which includes information that can be derived from the TBV, A process of delaying the compute response message until it reaches a deterministic transmission time that depends on both the TBV and the deterministic latency, The process of sending the compute response message to the participant device within the definitive transmission time, A method that includes [a certain feature].

Claims

1. A system comprising multiple gateways connected to receive inbound messages from two or more participant devices, wherein one or more of the gateways are: For a predetermined inbound message, a time-based value (TBV) is determined according to the arrival time of the predetermined inbound message. The predetermined inbound message, along with its TBV, is forwarded to one or more compute nodes. A response message containing the same TBV as the predetermined inbound message is received from one or more compute nodes. The desired transmission time is determined, which depends on the computational combination of deterministic latency and the TBV. To one or more of the participant devices, a reply message dependent on the response message is sent as an outbound message transmitted at the desired transmission time. The system is further structured in this way.

2. The system according to claim 1, wherein the TBV includes a timestamp corresponding to the reception time of the predetermined inbound message, and the desired transmission time is determined after receiving the response message from one or more compute nodes.

3. The system according to claim 1, wherein the TBV includes the desired transmission time of the reply message, and the desired transmission time is determined before the predetermined inbound message is forwarded to one or more compute nodes.

4. In the system described in claim 1, further, A packet scheduler configured to receive the aforementioned response message, having a series of indexed locations, each associated with a desired transmission time, A system comprising, wherein the TBV is a value that depends on the value of the associated indexed location for the desired transmission time.

5. The system according to claim 1, wherein the TBV is inserted into an unused field in the predetermined inbound message and then forwarded to one or more compute nodes.

6. The system according to claim 1, wherein the TBV is added as part of a field of the system's internal protocol within the predetermined inbound message.

7. The system according to claim 1, wherein the deterministic latency depends on the maximum time it takes for one or more gateways to receive a reply message from one or more compute nodes.

8. The system according to claim 1, wherein the deterministic latency follows a pattern that is variable but deterministic.

9. The system according to claim 1, wherein the deterministic latency is selected from a series of latencies uniformly distributed within a predetermined range.

10. A system according to claim 1, wherein the deterministic latency is set for each gateway, each connection, or for the entire system.

11. A system according to claim 1, wherein the deterministic latency changes dynamically depending on system conditions.

12. A system according to claim 1, wherein two or more gateways each receive the response message from one or more compute nodes.

13. The system according to claim 1, further comprising the following: the predetermined inbound message is forwarded to one or more compute nodes, and the response message is received from one or more compute nodes via a plurality of direct connections provided between each of the one or more gateways and each of the one or more compute nodes.

14. In the system according to claim 1, the response message relates to a transaction match event between two participant devices and two matching parties associated with each of them, and the gateway is The reply message is sent simultaneously to the two participant devices as an outbound message at the desired transmission time. The system is further configured in this way.

15. The system according to claim 14, wherein the outbound message is also simultaneously transmitted at the desired transmission time as a market data event message to a device associated with a subscriber to the market data stream.

16. In the system according to claim 1, the one or more gateways are Asynchronous messages are received from at least one of the aforementioned compute nodes. The asynchronous message is sent as an outbound message to two or more participant devices. The system is further configured in this way.

17. The system according to claim 1, wherein the TBV further depends on a time value relating to at least one of message path delay and compute node delay.

18. In the system according to claim 1, the one or more gateways The predetermined inbound message is forwarded to one or more sequencer nodes along with its TBV. The system is further configured in this way.

19. In the system described in claim 1, further, The one or more compute nodes mentioned above are: The system is configured to receive the predetermined inbound message, along with its TBV, from one or more gateways. A system in which one or more compute nodes are further configured to send the response message, along with the TBV, back to one or more gateways.

20. In the system according to claim 1, the one or more gateways are further configured to forward the predetermined inbound message, along with its TBV, to one or more sequencer nodes. A system in which one or more sequencer nodes are configured to forward the predetermined inbound message, along with its TBV, as a sequence-marked message to one or more compute nodes.

21. The system according to claim 20, wherein one or more compute nodes are further configured to determine the response message based on at least one of the predetermined inbound message and the sequence-marked message.

22. The system according to claim 1, wherein the predetermined inbound message relates to an electronic transaction matching function, and the response message is configured to complete the electronic transaction matching function.

23. In a method for transmitting response messages from an electronic device with deterministic latency, The process of receiving uplink messages from participant devices, The process of determining the time-based value (TBV) for the aforementioned uplink message according to the arrival time of the uplink message, A process of sending the uplink message and the transfer message including the TBV to one or more compute engines, The process of receiving a compute response message from one or more compute engines that includes the same TBV as that included in the transfer message, A process for determining a desired transmission time that depends on the computational combination of the TBV and the deterministic latency, The process of sending a reply message dependent on the compute response message to the participant device at the desired transmission time, A method that includes [a certain feature].

24. A method for transmitting a response message from an electronic device with deterministic latency, The process of receiving uplink messages from participant devices, The process for determining the time-based value (TBV) for the aforementioned uplink message is based on a computational combination of the arrival time of the uplink message and the deterministic latency, A process of sending the uplink message and the transfer message including the TBV to one or more compute engines, The process of receiving a compute response message from one or more compute engines that includes the same TBV as that included in the transfer message, The process of determining a desired transmission time that depends on the aforementioned TBV, The process of sending a reply message dependent on the compute response message to the participant device at the desired transmission time, A method that includes [a certain feature].

25. In the system according to claim 1, the one or more gateways are Convert the predetermined inbound message into a normalized message format to be used for subsequent communication between the one or more gateways and the one or more compute nodes. The system is further configured in this way.

Images