Electronic trading system and method based on a point-to-point mesh architecture

Abstract

This provides an electronic trading system and method based on a point-to-point mesh architecture. [Solution] The electronic trading system is based on a point-to-point mesh architecture and comprises a gateway, a core compute node, and a sequencer. The core compute node performs the electronic trading matching function. The gateway sends messages to the core compute node via a first direct connection and to the sequencer via a second direct connection. The sequencer sends the ordered message to the core compute node via a third direct connection. The core compute node completes the electronic trading matching function by determining the relative ranking among other messages based on the ordered message.

Description

Technical Field

[0001] (Cross - reference to related applications) This application is a continuation - in - part of U.S. Patent Application No. 16 / 988510, filed on August 7, 2020, and a continuation - in - part of U.S. Patent Application No. 16 / 988491, filed on August 7, 2020. The entire teachings of the above applications are incorporated herein by reference.

Background Art

[0002] According to current financial product trading systems, traders can electronically submit orders via a communication network and receive confirmations, market data, and other information. The "electronic" market realized by this, also known as an "electronic trading system", has largely replaced the pit - based "open outcry" trading system where all previous traders or their agents physically stood at a designated location, i.e., the trading pit, and traded with each other through oral and visual / hand - gesture communication, and has become the dominant means of trading most financial products.

[0003] When trading stocks and / or other financial products in an electronic trading system, the electronic trading system generally matches a trading order, i.e., a sell order and a corresponding buy order, to generate a trade. Depending on the conditions related to the trade, the process of matching a sell order and a buy order may become complicated.

[0004] One example of an electronic trading system is a computerized exchange, which typically has a central matching engine residing in a central server and multiple distributed servers or gateways. In such a computerized exchange, the normal process may be as follows: Order entry messages, e.g., sell orders and / or buy orders, are sent from a client or participant device, e.g., a trader terminal, to the computerized exchange, which processes the order entry messages. These order entry messages are received by the central server via gateways. For example, the processing in the computerized exchange at the central server may include, among other things, matching orders based on the received order entry messages.

[0005] The order processing approval message generated by the central server is then typically sent back to the participant device via a gateway that forwards the transaction. The gateway may perform additional processing before the order processing approval message is sent back to the participant device. The central server may also disseminate information about the order processing approval message to one or more other gateways that process the order processing approval message, either in the same format as it was received or in other formats, to generate market data output via the market data stream. The market data output is typically forwarded to participant devices or other subscribers to the market data stream through various communication mechanisms, requiring additional processing at the gateway. [Overview of the Initiative]

[0006] According to an exemplary embodiment, the electronic trading system comprises a gateway, a core compute node (also interchangeably referred to here as a core compute engine or compute engine) configured to perform electronic trading matching functions, and a sequencer. The gateway and the core compute node are connected via a first direct connection. The gateway and the sequencer are connected via a second direct connection. The sequencer and the core compute node are connected via a third direct connection. The first, second, and third direct connections each have their own non-shared bandwidth. The gateway is configured to send a message representing an electronic trading request with a limit price to buy or sell a financial instrument to the core compute node via the first direct connection. The message is received by the core compute node. The gateway is further configured to send a message to the sequencer via a second direct connection, and the sequencer is configured to send a sequenced message to the core compute node via a third direct connection. The sequencer is inserted between the gateway and the core compute node via the second and third direct connections. The sequenced messages transmitted by the sequencer are ordered versions of the messages transmitted by the gateway. In contrast, the sequenced messages are received by the core compute node. The core compute node is configured to determine the relative ranking (i.e., order) of the sequenced messages among the sequenced versions of other messages received by the core compute node in the electronic trading system. The core compute node is further configured to complete the electronic trading matching function for electronic trading requests according to the determined relative ranking, matching limit orders with counterparty limit orders for financial instruments, and enabling electronic trading of financial instruments.

[0007] Messages and sequentially marked messages may contain the same user data. This user data is associated with electronic transaction requests.

[0008] The message may be a gateway message sent by the gateway in response to the reception of an incoming message received by the gateway from a participant device. The sequencer is further configured to send a sequenced message to the gateway via a second direct connection. The sequenced message is received by the gateway. The sequenced message is the first sequenced message. The core compute node is further configured to send a core compute node message to the gateway via a first direct connection in response to the reception of the gateway message. The core compute node is further configured to send a core compute node message to the sequencer via a third direct connection, and the sequencer is further configured to send a second sequenced message to the gateway via a second direct connection. The second sequenced message is an ordered version of the core compute node message. The gateway is further configured to determine the relative ranking of the second sequenced message and the sequenced versions of other messages sent from the core compute node to the gateway. The gateway is further configured to send outgoing messages to participant devices. Outgoing messages are sent according to the determined relative ranking. The sequencer is further configured to send a second ordered message in response to this to the core compute node via a third direct connection.

[0009] A gateway can be a given gateway among multiple gateways. A core compute node can be a given core compute node among multiple core compute nodes. Each gateway among multiple gateways can be connected to each core compute node among multiple core compute nodes via its respective first direct connection. A sequencer can be connected to each gateway among multiple gateways via its respective second direct connection, and to each core compute node among multiple core compute nodes via its respective third direct connection. Multiple gateways, multiple core compute nodes, sequencers, and their respective direct connections can constitute at least a part of a point-to-point mesh system.

[0010] Within a point-to-point mesh system, each gateway in a group of gateways may be configured to send its respective compute node-bound messages to all compute nodes and sequencers of the group of core compute nodes. It should be understood that messages addressed to compute nodes may, for compatibility purposes, be referred to here as "comput node-bound" messages. Similarly, messages addressed to gateways may, for compatibility purposes, be referred to here as "gateway-bound" messages. Each core compute node in a group of core compute nodes may be configured to send its respective gateway-bound messages to all gateways and sequencers of the group of gateways. The sequencers may further be configured to send their respective ordered messages to the group of gateways and the group of core compute nodes upon receiving their respective compute node-bound messages or their respective gateway-bound messages.

[0011] A sequencer can be a given sequencer among several sequencers in a point-to-point mesh system. Each gateway in a plurality of gateways can be connected to each sequencer in the plurality of sequencers via a second direct connection. Each core compute node in a plurality of core compute nodes can be connected to each sequencer in the plurality of sequencers via a third direct connection. A given sequencer can be the currently active sequencer providing service to the point-to-point mesh system. Each other sequencer in the plurality of sequencers can be a standby sequencer waiting to take over from the currently active sequencer. Each sequencer in a plurality of sequencers can be connected to each other sequencer in the plurality of sequencers via a fourth direct connection. Each gateway in a plurality of gateways can further be configured to send messages destined for its respective compute node to a given sequencer among the plurality of sequencers. Each core compute node in a plurality of core compute nodes can further be configured to send messages destined for its respective gateway to a given sequencer among the plurality of sequencers. A given PLC may also be configured to send sequence-marked messages to each of the other PLCs in a group of PLCs via a fourth direct connection to each of them, so that a standby PLC can take over from the currently active PLC if the currently active PLC fails.

[0012] Each message sent by a given gateway to a compute node may be the same message received by multiple core compute nodes. Multiple core compute nodes may be configured to generate response messages upon receiving the same message. These response messages may be received by the given gateway from among the multiple core compute nodes. The given gateway may further be configured to act based on a given response message among those generated in response to the same message. A given response message may arrive at the given gateway first, relative to other response messages generated in response to the same message. The gateway may further be configured to ignore other response messages that arrive after the given response message.

[0013] Multiple compute node-bound messages representing the same message may be received by a given compute node from among multiple gateways. The given compute node may further be configured to act based on a given compute node-bound message among multiple compute node-bound messages. A given compute node-bound message may arrive at the given compute node first, relative to other compute node-bound messages representing the same message. A given core compute node may further be configured to ignore other compute node-bound messages arriving after the given compute node-bound message.

[0014] The electronic trading system may further include an order book accessible by the core compute node. The core compute node may further be configured to match trading orders for financial instruments using an electronic trading matching function. The core compute node may further be configured to maintain the remaining positions of financial instruments in the order book. A discrepancy in the value of financial instruments may result from the execution of the electronic trading matching function. The remaining positions may include a discrepancy in the value of financial instruments. It should be understood that the remaining positions can convey more information than quantity. For example, positions may be bullish and bearish (sides). According to the exemplary embodiment, the remaining positions convey both price and quantity.

[0015] The electronic trading system may also be equipped with a clock. Gateways, core compute nodes, and sequencers may be synchronized based on the clock.

[0016] The gateway may further be configured to serve at least one participant device and to send messages to the sequencer and core compute nodes upon receiving incoming messages at the gateway. Incoming messages may be sent by at least one participant device. The sequencer may further be configured to generate sequentially marked messages by marking messages with a unique sequential identifier, or by creating a representation of an received message, marking the representation with a unique sequential identifier, and transmitting the marked representation. The representation to be marked may be a sequentially marked message.

[0017] The electronic trading system may further include at least one redundant direct connection for each of the first direct connection, the second direct connection, and the third direct connection, or any set of these.

[0018] A gateway can be a given gateway among a plurality of gateways connected to each other in a communicative manner via a shared gateway network. A core compute node can be a given core compute node among a plurality of core compute nodes connected to each other in a communicative manner via a shared core compute node network. A sequencer can be a given sequencer among a plurality of sequencers connected to each other in a communicative manner via a shared sequencer network or via their respective fourth direct connections.

[0019] The electronic trading system may further include a system status log. A given PLC may be configured to transmit the system status log to at least one other PLC among a plurality of PLCs via a shared PLC network, or to store the system status log in a data store. The data store may be accessible to multiple PLCs via a shared PLC network.

[0020] The electronic trading system may be an active electronic trading system, and at least one of the sequencers may be communicatively coupled to a disaster recovery site. The disaster recovery site may include a standby electronic trading system. The standby electronic trading system may be a replica of the active electronic trading system and may be configured to allow electronic trading to continue in the event of a failure of the active electronic trading system.

[0021] A gateway, core compute nodes, sequencers, and first, second, and third direct connections may constitute a first point-to-point mesh system. An electronic trading system may be a first electronic trading system communicatively coupled to a proxy node. The proxy node may further be communicatively coupled to at least one participant device and a second electronic trading system. The second electronic trading system may include a second point-to-point mesh system. The proxy node may be configured to transmit messages to the first and second electronic trading systems upon receipt of an incoming message from at least one participant device. The first and second electronic trading systems may be configured to generate their respective responses to messages transmitted by the proxy node. The proxy node may further be configured to transmit a response to at least one participant device upon receipt of the first arriving response among those generated and received from the first or second electronic trading system.

[0022] According to another exemplary embodiment, a method for executing an electronic transaction comprises the step of sending a message representing an electronic transaction request with a limit price for buying or selling a financial instrument from a gateway to a core compute node via a first direct connection. The method further comprises the step of receiving the message at the core compute node in an electronic trading system in order to execute an electronic transaction function. The method further comprises the step of sending the message from the gateway to a sequencer in the electronic trading system via a second direct connection, and sending a sequenced message from the sequencer to the core compute node via a third direct connection. The first, second, and third direct connections each have their own non-shared bandwidth. The sequencer is inserted between the gateway and the core compute node via the second and third direct connections. The sequenced message sent by the sequencer is an ordered version of the message sent from the gateway via the second direct connection. The method further comprises the step of receiving the sequenced message at the core compute node. The method further comprises the steps of: receiving other messages from a gateway at the core compute node; and receiving an ordered version of other messages from a sequencer at the core compute node. The method further comprises the step of: determining the relative ranking of ordered messages among the ordered versions of other messages received by the core compute node in an electronic trading system at the core compute node. The method further comprises the step of: completing an electronic trading matching function for an electronic trading request at the core compute node according to the determined relative ranking, the completion step matching a limit order with a counterparty limit order for a financial instrument, thereby enabling electronic trading of the financial instrument.

[0023] Embodiments of alternative methods are similar to those described above in relation to the embodiment of the illustrated electronic trading system.

[0024] According to another exemplary embodiment, a sequencer of an electronic trading system comprises a non-temporary computer-readable medium encoding a series of instructions, which, once read and executed by the sequencer, cause the sequencer to communicate directly with a gateway via a first direct connection in the point-to-point mesh system of the electronic trading system. The series of instructions further causes the sequencer to communicate directly with a core compute node via a second direct connection in the point-to-point mesh system of the electronic trading system. The sequencer is inserted between the gateway and the core compute node via the first and second direct connections. The first and second direct connections each have their own non-shared bandwidth. The series of instructions further causes the sequencer to generate a sequentially marked message by marking the message or its representation with a unique sequential identifier, the message being received by the sequencer via the first or second direct connection, respectively. The series of instructions further causes the sequencer to transmit the sequentially marked message to the gateway and the core compute node.

[0025] According to another exemplary embodiment, the electronic trading system comprises a gateway coupled to a core compute node via an activation link, a ranking path, and a sequencer electronically located within the ranking path. The gateway is configured to transmit messages to the core compute node via the activation link and the ranking path. The core compute node is configured to receive messages and sequentially marked versions of messages from the gateway and the sequencer, respectively. The sequentially marked version includes a sequence identifier indicating the definitive position of the message among a plurality of messages communicated via the activation link and received by the sequencer via the ranking path. The message and the sequentially marked version include common metadata.

[0026] The core computing node may be configured to (i) initiate a matching function activity for an electronic transaction in response to receiving a message via an activation link, and (ii) prioritize completion of the matching function activity towards providing a service for the electronic transaction using an order identifier in response to receiving an ordered marked version via a ranking path.

[0027] The message and the ordered marked version may include common metadata. The core computing node may further be configured to correlate the message to the ordered marked version based on the common metadata in response to receiving the ordered marked version via the ranking path.

[0028] According to yet another exemplary embodiment, an electronic transaction system may include a gateway, a sequencer, and core computing nodes arranged in a point-to-point mesh topology. The core computing nodes may be configured to perform a matching function towards providing a service for a transaction request received from a participant device and introduced into the point-to-point mesh topology via the gateway. The point-to-point mesh topology may include a first direct connection, a second direct connection, and a third direct connection. The sequencer may be configured to (i) determine a definitive rank (i.e., order) for a message received by the sequencer via the first direct connection between the gateway and the core computing node and from the gateway or the core computing node via the second or third direct connection, respectively. The sequencer may further be configured to (ii) convey the position of the message within the definitive rank by transmitting an ordered marked version of the message representing the transaction request or a response thereto to the gateway and the core computing nodes via the second and third direct connections, respectively.

[0029] It should be understood that the exemplary embodiments disclosed herein may be implemented in the form of a method, an apparatus, a system, or a computer-readable medium having program code embodied therein.

[0030] As will become apparent from the following more detailed description of the exemplary embodiments, as shown in the accompanying drawings where like reference numerals refer to like parts throughout the various figures. The drawings are not necessarily to scale and emphasis is instead placed upon illustrating the embodiments being depicted.

Brief Description of the Drawings

[0031] [Figure 1A] FIG. 1A is a block diagram of an exemplary embodiment of a market for trading financial products. [Figure 1B-1] FIG. 1B-1 is a block diagram of an exemplary embodiment of an electronic trading system. [Figure 1B-2] FIG. 1B-2 is a block diagram of an exemplary embodiment of another electronic trading system. [Figure 1C] FIG. 1C is a block diagram of an exemplary embodiment of a point-to-point mesh system. [Figure 1D] FIG. 1D is a block diagram of another exemplary embodiment of an electronic trading system. [Figure 1E] FIG. 1E is a table of an exemplary embodiment of fields of a message format for trading messages. [Figure 1F] FIG. 1F is a flowchart showing an exemplary embodiment of the operation of an electronic trading system. [Figure 1G] FIG. 1G is a flowchart showing another exemplary embodiment of the operation of an electronic trading system. [Figure 2] FIG. 2 is a block diagram of an exemplary embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system. [Figure 3] FIG. 3 is a block diagram of another exemplary embodiment of a point-to-point mesh system. [Figure 4] FIG. 4 is a block diagram of another exemplary embodiment of a point-to-point mesh system. [Figure 5] FIG. 5 is a flowchart of an exemplary embodiment of a method for executing an electronic trade. [Figure 6] Figure 6 is a block diagram of an exemplary embodiment of a sequencer for an electronic trading system. [Figure 7] Figure 7 is a block diagram of another exemplary embodiment of the electronic trading system. [Modes for carrying out the invention]

[0032] The following is a description of the exemplary embodiment.

[0033] It should be understood that the dedicated / direct connections disclosed herein are point-to-point connections that do not pass through a shared network switch.

[0034] Current electronic trading systems attempt to offer performance advantages, but many suffer from performance degradation due to the increasing transaction volume resulting from the growing number of market participants. Some market participants employ high-frequency trading methods, often relying on high-speed computers to automatically monitor the market and react to market events, typically in an overwhelming manner. Furthermore, the ongoing demand for further reductions in processing latency and response time has created a need for increased capacity and performance to maintain the performance experienced by each market participant while avoiding harmful consequences such as capacity depletion and unfair access.

[0035] The increasing speed at which market participants can evaluate and respond to changes in market data, such as responding to market events, necessitates a greater need for sophisticated identification to increase the speed at which transactions are received by electronic trading systems, narrow the time intervals between receipts, and determine the order in which those transactions are received. For example, the deterministic operation of electronic trading systems, such as order allocation, is based on these factors. Furthermore, to increase capacity and opportunities along with increasing the bandwidth of each channel, the addition of communication channels to electronic trading systems allows more transactions to be submitted to the electronic trading system via multiple parallel paths.

[0036] Therefore, it is useful for electronic trading systems to identify incoming transactions received in a short period of time. It is even more useful for such systems to mediate between transactions received simultaneously or so close in time that they appear to be received simultaneously. In addition to increased capacity and lower latency, the global nature of business further increases the need for fault tolerance to enhance the availability and reliability of electronic trading systems.

[0037] A business transaction can be defined as one or more actions or operations undertaken in accordance with one or more relevant business rules (including industry, legal, or regulatory requirements or practices) to achieve a business or commercial objective, which may include compliance with industry, regulatory, or legal requirements. A business transaction can be performed by one or more computer operations and / or database operations / program actions, which themselves may also be called transactions. A business transaction can be characterized as deterministic in the following respects, as defined by the relevant business rules: It can be characterized by interdependences or relationships in which business transactions affect their outcomes, such as a dependency on the order in which they are processed (i.e., sequence), including temporal order, and / or a dependency on real-time processing, as defined by the business rules, in order to enable a business / commercial objective and / or meet the expectations of participants. This is referred to here as "transaction determinism." Generally, a set of deterministic transactions will produce a specific result when executed in one order (i.e., sequence), and a different result when executed in a different order (i.e., sequence). In some applications, deterministic processing may be preferred / preferred over real-time processing.

[0038] High-performance electronic trading systems are useful in ensuring transaction determinism under increasing loads while providing enhanced trading opportunities, fault tolerance, low latency processing, high capacity (e.g., processing a large number of messages per second), risk mitigation and market protection with minimal impact, and equitable access to information and opportunities.

[0039] The exemplary embodiments disclosed herein relate to a high-speed electronic trading system that provides a market in which orders to buy and sell financial instruments such as stocks, bonds, commodities, futures, and options are traded among market participants such as traders and brokers. The exemplary embodiments of the electronic trading system disclosed herein exhibit low latency, fairness, fault tolerance, and other features more fully described below.

[0040] Figure 1A is a block diagram of an exemplary embodiment of a market 90, including an exemplary embodiment of an electronic trading system 100 used for trading financial instruments (not shown). The electronic trading system 100 is primarily responsible for "matching" trading orders with each other and employs a point-to-point mesh system 102 for this matching. In one example, a limit order to "buy" a financial instrument is matched by the matching engine of the point-to-point mesh system 102 with a corresponding counter limit order to "sell". The matched limit and counter limit orders must satisfy at least partially the desired price, and any remaining unsatisfied quantities are passed on to other appropriate counter orders. The matched trading orders are then paired and the trade is executed.

[0041] Orders that are not fully satisfied or are partially satisfied are held in a data structure called the "Order Book" (not illustrated). The pending information regarding mismatched trading orders is available to the matching engine to satisfy subsequent trading orders. The Order Book is typically maintained for each financial instrument and generally defines or represents the state of the market for that particular instrument, i.e., for that particular financial instrument. The Order Book may, for example, include recent prices and quantities at which market participants have expressed their intention to buy or sell.

[0042] The matching results may be made visible to market participants via a streaming data service (not shown) called a market data feed (not shown). A market data feed typically includes individual messages that convey relevant information such as pricing for each financial instrument traded, as well as volume and other statistics.

[0043] In market 90, market participants include two traders, namely the first trader 104a and the second trader 104b. It should be understood that market 90 is not limited to market participants who are traders, nor is it limited to two traders. In market 90, market participants such as the first trader 104a and the second trader 104b may electronically submit and confirm trading orders, receive market data and other information via a communication network (not shown).

[0044] In the exemplary embodiment shown in Figure 1A, the first trader 104a submits a first trade order (not shown) to buy a financial instrument (not shown) via a first incoming message 3a transmitted to the electronic trading system 100 through a first participant device 130a. The electronic trading system 100 employs a point-to-point mesh system 102 to match the first trade order against a second trade order (not shown) to sell the financial instrument. The second trade order to sell the financial instrument is submitted by the second trader 104b via a second incoming message 3b transmitted to the electronic trading system 100 through a second participant device 130b.

[0045] In an exemplary embodiment, the electronic trading system 100 transmits a first outgoing message 5a and a second outgoing message 5b to the first participant device 130a and the second participant device 130b, respectively, to notify the first trader 104a and the second trader 104b that their respective trade orders have been successfully executed. The point-to-point mesh system 102 enables the electronic trading system 100 to execute high-speed, definitive electronic trading of financial instruments. An exemplary embodiment of the point-to-point mesh system 102 is disclosed below with reference to Figures 1B-1 and 1B-2.

[0046] Figure 1B-1 is a block diagram of an exemplary embodiment of the electronic trading system 100 of Figure 1A disclosed above. In a particular embodiment, the electronic trading system 100 comprises a gateway 120-1 coupled to a core compute node 140-1 via an activation link 180-1-1 and a ranking (i.e., ordering) path 117. The electronic trading system 100 further comprises a sequencer 150-1 electronically located within the ranking path 117. The gateway 120-1 is configured to transmit messages (not shown) to the core compute node 140-1 via the activation link 180-1-1 and the ranking path 117. The messages may be messages 106 disclosed below with respect to Figure 1B-2. The core compute node 140-1 is configured to receive messages and ordered versions of messages (not shown) from the gateway 120-1 and the sequencer 150-1, respectively. A sequenced version of a message may be a sequenced message 106', as further disclosed below in Figure 1B-2. The sequenced version (i.e., sequenced message 106') includes a sequence identifier (ID), which may be contained in the sequence ID field 110-14 of a sequenced message 106', as further disclosed below with respect to Figure 1E for non-limiting examples. The sequence ID indicates the definitive position of the sequenced version of that message among sequenced versions of several other messages communicated via activation link 180-1-1 and received by sequencer 150-1 via ranking path 117. The several messages for which the sequence ID indicates a definitive position also include sequenced versions of other messages received by core compute node 140-1 via ranking path 117. Messages (e.g., unordered messages or unsequenced messages) and their sequenced versions include common metadata (not shown). The sequence ID of a message is identified by correlating the message with its sequenced version via the common metadata, as further disclosed below.The sequence ID further indicates the definitive position of that message among all messages communicated throughout the entire electronic trading system 100, which have passed through sequencer 150-1 and have been sequentially marked by sequencer 150-1.

[0047] It should be understood that while the elements of the electronic trading system 100 timestamp the messages communicated there, the sequence ID determined by the sequencer 150-1 determines the position (rank / priority) of the messages communicated in the electronic trading system 100. Multiple systems can timestamp messages with the same timestamp, and as a result, the rank / priority of those messages may have to be determined by their recipients. Such a thing does not happen in the electronic trading system 100 because the sequencer 150-1 can be the sole entity that determines the rank / priority of messages communicated throughout the entire electronic trading system 100.

[0048] The core compute node 140-1 may be configured to (i) initiate matching function activity for electronic transactions in response to the receipt of a message via activation link 180-1-1, and (ii) prioritize the completion of the matching function activity toward servicing electronic transactions using the order identifier in response to the receipt of an ordered marked version via ranking path 117.

[0049] Messages and ordered versions may contain common metadata. Core compute node 140-1 may further be configured to correlate messages to ordered versions based on common metadata upon receiving ordered versions via the ranking path 117.

[0050] In the exemplary embodiment of Figure 1B-1, the message is transmitted to the core compute node 140-1 via activation link 180-1-1 in the activation link forward direction, i.e., at act-link-fwd-dir113a, and to the core compute node 140-1 via the order path 117 in the order path forward direction, i.e., at order-path-fwd-dir115a. Following the completion of the matching function activity, the core compute node 140-1 may transmit a response (not shown) to the gateway 120-1 via activation link 180-1-1 and the order path 117 in the activation link reverse direction (i.e., act-link-rev-dir113b) and the order path reverse direction (i.e., order-path-rev-dir115b). The response transmitted to the gateway 120-1 via activation link 180-1-1 and the order path 117 may be response 107, which is further disclosed below with respect to Figure 1B-2.

[0051] The activation link 180-1-1 is a single direct connection, while the ranking path 117 includes multiple direct connections. For example, the activation link 180-1-1, also called the first direct connection 180-1-1, is a single direct connection, while the ranking path 117 may include a second direct connection 180-gw1-s1 and a third direct connection 180-c1-s1, which are further disclosed below with respect to Figure 1B-2.

[0052] The gateway 120-1, the sequencer 150-1, and the core compute node 140-1 are arranged in a point-to-point mesh topology. The core compute node 140-1 may be configured to perform a matching function (i.e., an electronic transaction matching function) toward serving transaction requests received from participant devices and introduced into the point-to-point mesh topology via the gateway 120-1. This participant device is further disclosed below with respect to Figure 1B-2. The point-to-point mesh topology includes a first direct connection, a second direct connection, and a third direct connection, as disclosed above and further disclosed below with respect to Figure 1B-2. The sequencer 150-1 may be configured to (i) determine a definitive rank (i.e., order) of messages communicated between the gateway 120-1 and the core compute node 140-1 via the first direct connection and received by the sequencer 150-1 from the gateway 120-1 or the core compute node 140-1 via the second or third direct connection, respectively. The sequencer 150-1 may further be configured to communicate the position of a message within a deterministic order by (ii) transmitting an ordered version of the message to the gateway 120-1 and the core compute node 140-1 via second and third direct connections, respectively. The message represents a transaction request or a response thereto, as disclosed below with respect to Figure 1B-2.

[0053] Figure 1B-2 is another block diagram of an exemplary embodiment of the electronic trading system 100 of Figure 1A disclosed above. The electronic trading system 100 comprises a gateway 120-1, a core compute node 140-1 configured to perform electronic trading matching functions, and a sequencer 150-1. The gateway 120-1 and the core compute node 140-1 are connected via a first direct connection 180-1-1. The gateway 120-1 and the sequencer 150-1 are connected via a second direct connection 180-gw1-s1. The sequencer 150-1 and the core compute node 140-1 are connected via a third direct connection 180-c1-s1. The first direct connection 180-1-1, the second direct connection 180-gw1-s1, and the third direct connection 180-c1-s1 each have their own non-shared bandwidth.

[0054] In some of the drawings in the disclosure, arrows such as those for the first direct connection 180-1-1, the second direct connection 180-gw1-s1, and the third direct connection 180-c1-s1 shown in Figure 1B-2 are included on direct connections (i.e., direct links). Such arrows on direct connections / links indicate that data can flow bidirectionally through the direct connections / links. Other drawings in the disclosure, such as Figure 1B-1 disclosed above and Figure 1D further disclosed below, may not have the above arrows applied to direct connections / links, but it should be understood that data can flow bidirectionally along such connections / links, and that such connections / links may be a single communication link or two parallel links, each of which is unidirectional and data flows in opposite directions through these two links.

[0055] Continuing with reference to Figure 1B-2, gateway 120-1 is configured to send message 106 (i.e., B), representing an electronic trading request with a limit price to buy or sell a financial instrument, to core compute node 140-1 via a first direct connection 180-1-1, to which message 106 is received by core compute node 140-1. Gateway 120-1 is further configured to send message 106 (i.e., B) to sequencer 150-1 via a second direct connection 180-gw1-s1, to which sequencer 150-1 is configured to send a sequenced message 106' (i.e., C) to core compute node 140-1 via a third direct connection 180-c1-s1, as further disclosed below with reference to Figure 1D. Sequencer 150-1 is inserted between gateway 120-1 and core compute node 140 via the second and third direct connections. The ordered message 106' transmitted by the sequencer 150-1 is an ordered version of the message transmitted by the gateway 120-1. The ordered message is received by the core compute node 140-1. The core compute node 140-1 is configured to determine the relative ranking of the ordered message 106' among ordered versions (not shown) of other messages received by the core compute node 140-1 in the electronic trading system 100. The core compute node 140-1 is further configured to complete the electronic trading matching function for electronic trading requests according to the determined relative ranking, enabling electronic trading of financial instruments by matching limit orders with counterparty limit orders for those financial instruments.

[0056] It should be understood that the electronic trading matching function may include more than just matching trading orders. For example, the electronic trading matching function may include sending approval messages, as further disclosed below with respect to Figure 1D. Also, core compute node 140-1 may perform part of the electronic trading matching function before receiving an ordered message 106', which enables core compute node 140-1 to facilitate it. For example, following the receipt of message 106 (e.g., an unordered message), core compute node 140-1 may initiate the electronic trading matching function by loading data relating to the ticker symbol identified in message 106 into high-speed memory for later access, such as the ticker symbol in the ticker symbol field 110-2, as further disclosed below with respect to Figure 1E, in non-limiting examples. The ticker symbol may represent a traded security, such as a stock ticker or stock listing symbol. However, it should be understood that core compute node 140-1 may initiate (trigger) the electronic trading matching function by performing any type of activity related to the content of message 106.

[0057] The sequencer 150-1 may be further configured to generate a sequence-marked message 106' by marking message 106 or its representation with a unique sequence identifier (not shown). The unique sequence identifier may have a value corresponding to the arrival time of message 106 in the sequencer 150-1, or it may indicate the relative sequence position of message 106 among a plurality of messages (not shown) received in the sequencer 150-1.

[0058] As described above, core compute node 140-1 may initiate electronic trading functionality upon receiving unordered message 106, thereby beginning processing of the unordered message 106. However, core compute node 140-1 may not complete processing of message 106 and / or guarantee the results of processing message 106 until core compute node 140-1 receives ordered message 106'. If there is no definitive prioritization for processing the message, such as that specified via the order identifier in ordered message 106', then, for example, the processing of the message by compute node 140-1 may be unpredictable. A non-limiting example of possible unpredictable outcomes is a group of unprocessed unordered messages, each representing a potential match for a counterparty in a securities exchange. It would be useful to have a definitive mode for mediating between these potential matches, presumably because only a subset of the potential matches will be executable for a given trading order from the counterparty.

[0059] According to some embodiments, after receiving both the unordered message 106 and the ordered message 106', the compute node 140-1 may correlate the unordered message 106 with the ordered message 106' via identification information in both versions of the message, as will be described later in relation to Figure 1E. When the compute node 140-1 receives the ordered message 106', it may then determine the appropriate order in which the messages 106 / 106' should be processed relative to other messages throughout the electronic trading system 100, and may complete the processing of the messages 106 / 106' by sending appropriate response messages, or, in some cases, referring to the order identifier assigned to the ordered message 106' by the sequencer. Returning to the non-limiting example of multiple messages representing potential matches for counterparties in the exchange of securities, when compute node 140-1 receives an ordered version of the messages representing potential matches, compute node 140-1 can strictly determine the order in which possible matches occur and complete the electronic transaction matching function.

[0060] According to the exemplary embodiment, in addition to sending the sequence-marked message 106' to compute node 140-1 via a third direct connection 180-c1-s1, sequencer 150-1 may further send the sequence-marked message 106' (i.e., C) to gateway 120-1 via a second direct connection 180-gw1-s1. By providing the sequence-marked message 106' (i.e., C) to the sender of message 106, the sender, i.e., gateway 120-1, can correlate the sequence number assigned to the message, i.e., message 106, with other identifying information in the message (as described later in relation to Figure 1E). This allows the sender to easily deal with subsequent messages that refer to that sequence number, as further disclosed below in relation to Figure 1D.

[0061] The gateway 120-1, core compute node 140-1, sequencer 150-1, first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1 constitute a point-to-point mesh system 102. According to an exemplary embodiment, in the point-to-point mesh system 102, the first direct connection 180-1-1, the second direct connection 180-gw1-s1, and the third direct connection 180-c1-s1, or any set thereof, may be protected by at least one respective redundant direct connection (not shown). If this direct connection fails, the respective redundant direct connection may be adopted in its place. Thus, the electronic trading system 100 may further include at least one respective redundant direct connection to the first, second, and third direct connections, or any set thereof.

[0062] According to an exemplary embodiment, the electronic trading system 100 may further include a clock, and the gateway 120-1, core compute node 140-1 and sequencer 150-1 may be synchronized based on a clock such as clock 195 in Figure 1D, as further disclosed below.

[0063] Message 106 may be called a “gateway” message because it originates from a gateway, i.e., gateway 120-1. Message 106 may also be called a “comput node” message because, in the exemplary embodiment, it is addressed to a compute node, i.e., core compute node 140-1. Message 106 is a gateway message sent by gateway 120-1 in response to the receipt of incoming message 103 (i.e., A) received by gateway 120-1 from a participant device (not shown). Sequentially marked message 106' may be a first sequentially marked message. The core compute node 140-1 may be configured to send a core compute node message, i.e., response 107 (i.e., D), to gateway 120-1 via a first direct connection 180-1-1 in response to receiving message 106, i.e., a gateway message, and to send the core compute node message (i.e., response 107) to sequencer 150-1 via a third direct connection 180-c1-s1. The sequencer 150-1 may be configured to send a second ordered message, i.e., ordered response 107' (i.e., E), to gateway 120-1 via a second direct connection 180-gw1-s1. The second ordered message is an ordered version of the core compute node message. Gateway 120-1 may further be configured to determine the relative ranking of the second ordered message (i.e., ordered response 107') and the ordered versions of other messages sent from core compute node 140-1 to gateway 120-1. Gateway 120-1 may further be configured to send outgoing message 105 (i.e., F) to participant devices according to the determined relative ranking. It should be understood that messages 106 and response 107 relate to trading activity.

[0064] Sequencer 150-1 may further send a second sequenced message, i.e., a sequenced response 107' (i.e., E), to core compute node 140-1 via a third direct connection 180-c1-s1. By providing the sequenced response 107' (i.e., E) to the sender of response 107, the sender, i.e., core compute node 140-1, can correlate the sequence number assigned to the message, i.e., response 107, with other identifying information in the message (as described later in relation to Figure 1E). This allows the sender to easily handle subsequent messages that refer to that sequence number, as further disclosed below in relation to Figure 1D.

[0065] Similarly to the above in relation to messages 106 and 106' received by compute node 140-1, in response to the receipt of an unordered response message 107, gateway 120-1 may activate processing of the response message even before gateway 120-1 receives an ordered response 107'. In a non-limiting example, activating (starting) processing may include updating the state of gateway 120-1's open trade order database and / or accumulating outgoing messages 105 that are ready to be sent to participant devices. However, in some embodiments, gateway 120-1 may not complete processing of the ordered response message 107 (which includes sending outgoing messages 105 to participant devices) until gateway 120-1 receives an ordered response message 107' that includes an order identifier specifying the definitive position of the response message 107 in a series of messages including other messages in the electronic trading system 100. In some embodiments, after receiving both the unordered message 107 and the ordered response message 107', the gateway 120-1 may correlate the unordered response message 107 with the ordered response message 107' via identification information in both versions of the message, as described later in relation to Figure 1E. This ensures that the definitive position of the response messages 107 / 107' is determined upon receipt of the ordered response message 107'. In some embodiments, the processing of the response messages may then be completed, which includes ensuring that the outgoing message 105 is transmitted to the participant device.

[0066] The electronic trading system 100 may further include an order ledger (not shown) accessible by the core compute node 140-1. The core compute node 140-1 may further be configured to match trading orders for financial instruments (not shown) based on an electronic trading matching function being performed. For example, the core compute node 140-1 may further be configured to match trading orders for financial instruments using an electronic trading matching function. The core compute node 140-1 may further be configured to maintain the remaining positions (not shown) of financial instruments in the order ledger. A monetary mismatch of financial instruments may result from performing the electronic trading matching function. The remaining positions may include monetary mismatches of financial instruments. It should be understood that the remaining positions can convey more information than quantity. For example, positions may be bullish and bearish (sides). According to the exemplary embodiment, the remaining positions convey both price and quantity.

[0067] Gateway 120-1 may further be configured to serve at least one participant device (not shown) and, upon receiving an incoming message 103 at Gateway 120-1, send message 106 to Sequencer 150-1 and Core Compute Node 140-1. The incoming message 103 is sent by at least one participant device. Sequencer 150-1 may further be configured to generate sequentially marked messages by marking the message with a unique sequential identifier, or by creating a display of the received message, marking the display with a unique sequential identifier, and transmitting the marked display. The display to be marked may be a sequentially marked message.

[0068] According to the exemplary embodiment, gateway 120-1 may be a given gateway among a plurality of gateways, such as the plurality of gateways shown in Figures 1C, 1D, 3, and 4, which are further disclosed below. Similarly, core compute node 140-1 may be a given core compute node among a plurality of core compute nodes, such as the plurality of core compute nodes shown in Figures 1C, 1D, 3, and 4, which are further disclosed below.

[0069] Figure 1C is a block diagram of an exemplary embodiment of a point-to-point mesh system 122. The point-to-point mesh system 122 includes a plurality of gateways 120, a plurality of core compute nodes 140, and a sequencer 150-1. Each gateway 120-1, 120-2...120-g of the plurality of gateways 120 is connected to each core compute node 140-1, 140-2...140-c of the plurality of core compute nodes 140 via a first direct connection, i.e., a first direct connection 180-a. The sequencer 150-1 is connected to each gateway of the plurality of gateways via a second direct connection, i.e., a second direct connection 180-b, and to each core compute node of the plurality of core compute nodes via a third direct connection, i.e., a third direct connection 180-c. Multiple gateways, multiple core compute nodes, sequencer 150-1, and their respective direct connections form at least a portion of the point-to-point mesh system 122.

[0070] Within the point-to-point mesh system 122, each of the multiple gateways 120 is configured to send its respective compute node-bound messages to all of the multiple core compute nodes 140 and to the sequencer 150-1. Within the point-to-point mesh system 122, each of the multiple core compute nodes 140 is configured to send its respective gateway-bound messages to all of the multiple gateways 120 and to the sequencer 150-1. Within the point-to-point mesh system 122, the sequencer 150-1 is further configured to send its respective ordered messages to the multiple gateways 120 and the multiple core compute nodes 140 upon receiving their respective compute node-bound messages or gateway-bound messages.

[0071] Each message destined for a compute node, sent by a given gateway among a plurality of gateways 120, is the same message received by a plurality of core compute nodes 140. At least two of the plurality of core compute nodes 140 may be configured to generate response messages in response to receiving the same message. The response messages may be received at a given core compute node among the plurality of core compute nodes. A given gateway may further be configured to act on a given response message among the response messages generated in response to receiving the same message. A given response message may arrive at the given gateway first, relative to other response messages generated in response to receiving the same message. A given gateway may further be configured to ignore other response messages that arrive after the given response message. Such response messages are sometimes called functionally equivalent messages. Functionally equivalent messages represent the same message that causes the recipient of the functionally equivalent message to perform the same function (e.g., one or more activities). Functionally equivalent messages represent the same message and, in some embodiments, may actually be identical. However, in other embodiments, functionally equivalent messages may not necessarily be identical, as they may, in some cases, include different metadata such as different timestamps, different source identifiers, etc., as non-limiting examples. For clarity, the term “functionally equivalent messages” does not refer to unordered messages related to the corresponding ordered message. Rather, while in some embodiments there may be multiple ordered messages representing the same ordered message, functionally equivalent messages generally refer to multiple unordered messages representing the same unordered message, and thus we may sometimes refer to functionally equivalent unordered messages or functionally equivalent ordered messages.The following description of Figure 1E illustrates at least one example of how a receiving node may determine that two or more such response messages are functionally equivalent and represent the same message.

[0072] Multiple functionally equivalent messages (not shown) may be received at a given gateway from among multiple core compute nodes 140, multiple sequencers 150, or a combination thereof. The given gateway may further be configured to act on a given functionally equivalent message among the multiple functionally equivalent messages, the given functionally equivalent message being the first to arrive at the given gateway. The given gateway may further be configured to ignore other functionally equivalent messages among the multiple functionally equivalent messages that arrive after the given functionally equivalent message. Such messages may be understood as "functionally" equivalent because, for the same message received at multiple core compute nodes, each of the multiple core compute nodes independently generates a response message, and each of the responses, even if not strictly identical, functionally produces the same result. For example, such a response message may uniquely identify a particular core compute node sending the response by having at least different source core identifiers contained therein.

[0073] According to an exemplary embodiment, these at least two "functionally equivalent messages" may arrive at a given core compute node, such as core compute node 140-1, from a gateway / sequencer. The given core compute node may be configured to process only the first of such functionally equivalent messages to arrive. In this way, multiple functionally equivalent messages may be received at a given compute node from multiple gateways, multiple sequencers, or a combination thereof. The given compute node may further be configured to act on a given functionally equivalent message among multiple functionally equivalent messages, the given functionally equivalent message being the first to arrive at the given compute node. The given compute node may further be configured to ignore other functionally equivalent messages among multiple functionally equivalent messages that arrive after the given functionally equivalent message.

[0074] Thus, multiple compute node-bound messages representing the same message may be received by a given compute node, such as compute node 140-1, from among multiple gateways 120. This may be the case when the electronic trading system 100 is configured for high availability (HA). For example, redundant flows from participant devices may be received between multiple gateways, and thus multiple compute node-bound messages representing the same message may be sent by that gateway to compute node 140. According to an exemplary embodiment, a given core compute node may be configured to act on a given compute node-bound message among multiple compute node-bound messages, and the given compute node-bound message may be the first to arrive at the given compute node relative to other compute node-bound messages representing the same message. The given core compute node may further be configured to ignore other compute node-bound messages that arrive after the given compute node-bound message. Messages addressed to multiple compute nodes that represent the same message are sometimes referred to as functionally equivalent messages. The description of Figure 1E below illustrates at least one example of how a receiving node can determine that two or more such messages addressed to compute nodes are functionally equivalent and represent the same message.

[0075] The electronic trading system 100 may be an active electronic trading system in which at least one of a plurality of sequencers can be communicably coupled to a disaster recovery site, including a standby electronic trading system, such as the disaster recovery site 155 shown in Figure 1D below.

[0076] Figure 1D is a block diagram of another exemplary embodiment of the electronic trading system. Figure 1D shows an exemplary electronic trading system 100 including a number of gateways 120-1, 120-2, ..., 120-g (collectively referred to as gateway 120), a set of core compute nodes 140-1, 140-2, ..., 140-c (collectively referred to as core compute node 140 or compute node 140), and one or more sequencers 150-1, 150-2, ..., 150-s (collectively referred to as sequencer 150). Thus, in some embodiments, gateways 120, core compute nodes 140, and sequencers 150 are considered nodes in the electronic trading system 100. As will be described in more detail later, in one embodiment, gateways 120, compute nodes 140, and sequencers 150 are directly connected to each other, preferably via a low-latency dedicated connection 180.

[0077] In relation to the description of the electronic trading system 100, the term "peer" refers to other devices that generally provide the same functionality in the electronic trading system 100 (e.g., "gateway" vs. "core compute node" vs. "sequencer"). For example, gateways 120-2, ..., 120-g are peers of gateway 120-1, core compute nodes 140-2, ..., 140-c are peers of core compute node 140-1, and sequencers 150-2, ..., 150-s are peers of sequencer 150-1.

[0078] In relation to the description of System 100, the terms "active" and "standby" may refer to the high availability (HA) role / state / mode of a system / component. Generally, a standby system / component is a redundant (backup) system / component that, when powered on, can take over the functions performed by the active system / component. Such a switchover / failover, i.e., switching from the standby role / state / mode to the active role / state / mode, may, in non-limiting examples, occur automatically in response to a failure of the currently active system / component.

[0079] The electronic trading system 100 processes trading orders from one or more participant computing devices 130-1, 130-2, ..., 130-p (collectively referred to as participant device 130) and provides them with relevant information. Participant devices 130 interact with the electronic trading system 100 and may be one or more personal computers, tablets, smartphones, servers, or other data processing devices configured to display and receive trading order information. Participant devices 130 may be operated by a human via a graphical user interface (GUI) or via a high-speed automated trading method running on a physical or virtual data processing platform. Each participant device 130 may exchange messages with the electronic trading system 100 (i.e., send and receive messages to and from it) via a connection established by the gateway 120. While Figure 1D shows each participant device 130 connected to the electronic trading system 100 via a single connection to the gateway 120, it should be understood that participant devices 130 may be connected to the electronic trading system 100 via multiple connections to one or more gateway devices 120.

[0080] Although each gateway 120-1 can provide service to a single participant device 130, it typically provides service to multiple participant devices 130.

[0081] Compute nodes 140-1, 140-2, ..., 140-c (hereinafter also referred to as Matching Engine 140 or Compute Engine 140) provide the matching functions described above and may further generate outgoing messages to be delivered to one or more participant devices 130. Each compute node 140 is a high-performance data processor and typically holds one or more data structures that retrieve and hold one or more order books 145-1, 145-2, ..., 145-b. An order book 145-1 may be held, for example, for each product for which the core compute node 140-1 is responsible. One or more compute nodes 140 and / or one or more gateways 120 may provide a market data feed 147. The market data feed 147 may be broadcast (e.g., multicast) to participants, which may be participant devices 130 or any other appropriate computing devices.

[0082] Some outgoing messages generated by the core compute node 140 are synchronous, meaning they may be generated directly by one core compute node 140 in response to one or more incoming messages received from one or more participant devices 130, such as outgoing "approval messages" or "execution messages" corresponding to incoming "new order" messages. However, in some embodiments, at least some outgoing messages may be asynchronous and initiated by the trading system 100, for example, certain "unaccepted" cancellation messages and "transaction cancellation" or "transaction failure" messages.

[0083] A distributed computing environment such as the electronic trading system 100 can be configured using multiple matching engines that operate in parallel on multiple compute nodes 140.

[0084] The sequencer 150 ensures that the appropriate order of any priority-dependent operations is maintained. To ensure that operations on incoming messages are not performed out of order, incoming messages received at one or more gateways 120, for example, a new trade order message from one of the participant devices 130, will typically then pass through at least one sequencer 150 (e.g., a single currently active sequencer and possibly one or more standby sequencers), and the incoming messages are marked with a sequence identifier (by the single currently active sequencer if multiple sequencers exist). This identifier may be a unique monotonically increasing value used throughout the distributed system 100 (e.g., the electronic trading system 100) in subsequent processing to determine relative priority among messages and to uniquely identify the message throughout the electronic trading system 100. In some embodiments, the sequence identifier may indicate the order in which the message arrived at the sequencer (i.e., the sequence). For example, the sequence identifier may be a value that is monotonically incremented or decremented by the sequencer according to a fixed interval for each incoming message; for example, the sequence identifier may be incremented by 1 for each incoming message. However, it should be understood that the sequence identifier is unique but not limited to a monotonically increasing or decreasing value. In some embodiments, the initial unmarked message and the sequence-marked message may be essentially identical, except for the value of the sequence identifier contained in the marked version of the message. Once a marked incoming message, i.e., a sequence-marked message, is sequenced, it is usually then forwarded by the sequencer 150 to other downstream compute nodes 140 to perform potentially rank-dependent processing on the message. Thus, in addition to uniquely identifying the message throughout the electronic trading system 100, the sequence identifier assigned by the sequencer 150 can also determine the relative ranking of each marked message among other marked messages in the electronic trading system 100.

[0085] Thus, in contrast to other purposes for which sequence identifiers may be employed, the unique sequence identifiers disclosed herein may be used to ensure a definitive order (i.e., sequence) for processing electronic transaction messages. A unique sequence identifier represents an indicative, unique, definitive ranking (i.e., sequence) for the processing of a given electronic transaction message relative to other transaction messages in an electronic transaction system. According to an exemplary embodiment, a sequence identifier may be added to the sequence ID fields 110-14 of a message, as further disclosed below with respect to Figure 1E as a non-limiting example.

[0086] In some embodiments, messages may flow in other directions, i.e., from the core compute node 140 to one or more participant devices 130, passing through one or more gateways 120. Outgoing messages generated by one such core compute node 140 may also be rank-dependent (i.e., order-rank-dependent) and therefore may first pass through a sequencer 150, which is typically further marked with an order identifier. The sequencer 150 may then forward the marked response messages to the gateways 120 to deliver them to the participant devices 130 in a appropriately deterministic order.

[0087] The use of the sequencer 150 to generate unique sequence numbers and thereby mark messages or their displays, i.e., to generate sequentially marked messages, ensures that the correct prioritization of operations is maintained throughout the distributed system, i.e., the electronic trading system 100, and regardless, the compute nodes or sets of compute nodes 140 process the messages. This approach provides "state determinism," for example, that the entire state of the system is deterministic and reproducible (potentially at any other location, such as a disaster recovery site), providing fault tolerance, high availability, and disaster recovery capability.

[0088] It may also be important for the generating node (i.e., a node that introduces a new message into the electronic trading system 100, for example, by generating a new message and / or by forwarding a message received from the participant device 130) and its peer node to receive a sequence number assigned to that message. Receiving a sequence number for a message it generates can be useful for the generating node and its peer node not only for processing messages sequentially according to those sequence numbers, but also for correlating the messages generated by the node with message sequence identifiers used throughout the rest of the electronic trading system 100. Such correlation between an unmarked version of a message introduced into the electronic trading system by a generating node and a sequence-marked version of the same message output by a sequencer can be done via identification information in both versions of the message, as will be further discussed in relation to Figure 1E. Subsequent messages generated within the electronic trading system 100 are also assigned their own sequence numbers, but may still be based on one or more sequence numbers of related preceding messages. Therefore, a node may need to quickly refer to messages it has previously generated (by their sequence number). This is because, for example, the sequence number of a message generated by a node is used as a reference for subsequent messages.

[0089] In some embodiments, the generating node may first send the message to the sequencer 150 and wait for the sequencer to receive a sequence number for the message before forwarding the message to other nodes in the electronic trading system 100.

[0090] In an alternative exemplary embodiment, to avoid at least one hop that could add unnecessary latency within the electronic trading system 100, after receiving an unordered message from the generating node, the sequencer 150 may not only send an ordered version of the message (i.e., an ordered message) to the destination node, but also return the ordered version of the message to the sending node and its peers substantially simultaneously. For example, after assigning a sequence number to an incoming message sent from gateway 120 to core compute node 140, the sequencer 150 may not only forward the ordered version of the message to core compute node 140, but also return the ordered version of the message to gateway 120-1 and other gateways 120. Thus, if any subsequent message generated at core compute node 140 is based on that sequence number, any gateway 120 can easily identify the relevant message originally generated by gateway 120-1 by its sequence number.

[0091] Similarly, in some further embodiments, an ordered version of an outgoing message generated by the core compute node 140 and sent from there to the gateway 120, and ordered by the sequencer 150, may be forwarded by the sequencer 150 to the gateway 120 and also sent back to the core compute node 140.

[0092] Some embodiments, as further disclosed below with respect to Figure 4, may include multiple sequencers 150 for high availability, for example, to ensure that other sequencers are available if a first sequencer fails. In embodiments having multiple sequencers 150 (e.g., a currently active sequencer 150-1 and one or more standby sequencers 150-2, ..., 150-s), the currently active sequencer 150-1 may maintain a system state log (not shown) of all messages that have passed through sequencer 150-1 and the associated sequence numbers of the messages. This system state log may be sent to the standby sequencers continuously or periodically to provide them with essential system states so that they can take over the active sequencer as needed. Alternatively, the system state log may be stored in a data store accessible to the multiple sequencers 150.

[0093] The system status log may be continuously or periodically replicated to one or more sequencers in a standby replica electronic trading system (not shown in detail) at the disaster recovery site 155, thereby enabling electronic trading to continue in exactly the same state at the disaster recovery site 155 if the primary site of the electronic trading system 100 experiences a catastrophic failure.

[0094] According to an exemplary embodiment, the currently active sequencer among a group of sequencers may store a system state log in a data store (not shown). The data store may be accessible to the group of sequencers via a shared sequencer network, such as the sequencer-wide shared network 182-s, which is further disclosed below with respect to Figure 1D. When a given sequencer among the group of sequencers switches its role (state) from standby to active, it may retrieve the system state log from the data store and synchronize its state with the state of the previously active sequencer.

[0095] In some embodiments, the system status log may be provided to a dropcopy service 152, which may be performed by one or more sequencers and / or one or more other nodes in the electronic trading system 100. The dropcopy service 152 may provide a record of daily trading activity through the electronic trading system 100, which may be distributed, for example, to regulatory authorities and / or clients connected via participant devices 130. In alternative embodiments, the dropcopy service 152 may be performed by one or more gateways 120. Furthermore, in addition to or instead of referring to the system status log, the dropcopy service 152 may provide a record of trading activity based on the content of incoming and outgoing messages transmitted throughout the electronic trading system 100. For example, in some embodiments, a gateway 120 performing the dropcopy service 152 may receive all messages exchanged throughout the electronic trading system 100 from the sequencer 150 (and / or from the core compute node 140 and other gateways 120). A participant device 130 configured to receive daily trading activity records from the drop copy service 152 does not necessarily have to send or utilize the matching function of the electronic trading system 100 to place trading orders.

[0096] Messages exchanged between participant devices 130 and gateway 120 shall conform to any appropriate protocol that may be used for financial transactions (for convenience, referred to as "financial transaction protocols"). For example, messages may be exchanged according to custom protocols or established standard protocols that include both binary protocols (such as Nasdaq Ouch and NYSE UTP) and text-based protocols (such as NYSE FIX CCG). In some embodiments, the electronic trading system 100 may support the simultaneous exchange of messages according to multiple financial transaction protocols that include multiple protocols simultaneously at the same gateway 120. For example, participant devices 130-1, 130-2, and 130-3 may simultaneously establish trading connections according to Nasdaq Ouch, NYSE UTP, and NYSE FIX CCG, respectively, or they may be exchanging messages with gateway 120-1.

[0097] Furthermore, in some embodiments, the gateway 120 may translate messages received from participant devices 130 that conform to financial transaction protocols into a standardized message format used for exchanging messages between nodes in the electronic trading system 100. The standardized transaction format may be an existing protocol, or it may generally be of a different size and data format from any financial transaction protocol used to exchange messages with participant devices 130. For example, the standardized transaction format may include one or more additional fields or parameters, or omit one or more fields or parameters, and / or each field or parameter of the message in the standardized format may be of a different data type or size from the corresponding message received from participant devices 130 at the gateway 120. Similarly, in the reverse direction, the gateway 120 may translate outgoing messages generated in a standardized format by the electronic trading system 100 into messages in the format of one or more financial transaction protocols used by participant devices 130 to communicate with the gateway 120.

[0098] Figure 1E is a table of exemplary embodiments of fields in a message format 110 for transaction messages, such as transaction messages exchanged between nodes in the electronic trading system 100 disclosed above. In the exemplary embodiment of Figure 1E, the message format 110 is a standardized message format intended to be used for the internal (i.e., within the electronic trading system 100) representation of transaction messages when they are exchanged between nodes in the electronic trading system 100. In this exemplary embodiment, the gateway 120 exchanges messages between the participant 130 and the electronic trading system 100 and translates the messages between a format specified by one or more financial transaction protocols used by the participant 130 and a standardized transaction format used between nodes in the electronic trading system 100. It should be understood that fields 110-1 to 110-17 are for non-limiting examples, the message format 110 may contain more, fewer, or different fields, and the order of such fields is not limited to that shown in Figure 1E.

[0099] Although the fields in message format 110 are shown in a single message format in this example, they may be distributed across multiple message formats or encapsulated in a layered protocol. For example, in other embodiments, some sets of fields in message format 110 may be included as part of a header, trailer, or extension field in a layered protocol that encapsulates other fields of message format 110 in the message payload. According to some exemplary embodiments, message format 110 may define one or more fields of data encapsulated in the payload (data) portion of other message formats, including, but not limited to, the payload portions of IP datagrams, UDP datagrams, and TCP packets, or, in non-limiting examples, message data frame formats such as Ethernet data frame formats, or other data frame formats including InfiniBand, Universal Serial Bus (USB), PCI Express (PCI-e), and High-Definition Multimedia Interface (HDMI).

[0100] The message format 110 includes fields 110-1...110-6 corresponding to information that may be contained in a message sent or received in accordance with a financial transaction protocol for communication with one or more participant devices 130. As a non-limiting example, the message type field 110-1 indicates the transaction message type. Certain transaction message types (such as message types "New Order," "Replacement Order," or "Cancellation Order") correspond to messages received from participant devices 130, while other message types ("New Order Approval," "Replacement Order Approval," "Cancellation Order Approval," "Execution," "Execution Report," "Unaccepted Cancellation," "Transaction Failure," or various rejection messages) correspond to messages contained in transaction messages generated by the electronic trading system 100 and sent to participant devices 130.

[0101] Message format 110 also includes a ticker symbol field 110-2 containing an identifier for the securities being traded, such as a stock ticker or stock ticker symbol. For example, "IBM" is the stock ticker symbol for "International Business Machines Corporation". A side field 110-3 in message format 110 may be used to indicate the "side" of the trading message, such as whether the trading message is a "buy", "sell", or "short sell". Similarly, a price field 110-4 may be used to indicate the desired price for buying or selling the security. A quantity field 110-5 may be used to indicate the desired quantity of the security (e.g., the number of shares). Message format 110 may also include an order token field 110-6, which is added to the "order token" or "client order ID" initially provided by the participant device 130 to uniquely identify the new order in the context of a particular trading session (i.e., a "connection" or "flow") established between the participant device 130 and the electronic trading system via the gateway 120.

[0102] Fields 110-1...110-6 are typical fields usually included for most message types that conform to most financial trading protocols, but it should be understood that message format 110 may also include additional or alternative fields to support specific message types or specific financial trading protocols. For example, according to many financial trading protocols, “replacement order” and “cancel order” message types require participant 130 to supply additional order tokens to represent the replaced or canceled order and distinguish it from the original order. Similarly, “replacement order” and “cancel order” usually also include a replacement / cancel quantity field, and “replacement order” may include a replacement price field. These additional replacement / cancel order token fields, replacement price field, and replacement / cancel quantity field may be included in the corresponding approval message transmitted by the electronic trading system 100.

[0103] Furthermore, the message format 110 includes fields 110-11...110-17 that may be used internally within the electronic trading system 100 but do not necessarily correspond to fields in the message exchanged with the participant device 130. For example, the node identifier field 110-11 can uniquely identify each node in the electronic trading system 100. In some embodiments, a generating node may include its identifier in the message it introduces into the electronic trading system 100. For example, each gateway 120 may include its node identifier in the message it forwards from the participant device 130 to the compute node 140 and / or the sequencer 150. Similarly, each compute node 140 may include its node identifier in the message it generates that is sent to other nodes in the electronic trading system 100 (e.g., approval, execution, or type of an asynchronous message intended to be ultimately forwarded to one or more participant devices 130). Thus, each message introduced into the electronic trading system 100 can be associated with the generating node of the message via the node identifier field 110-11 in the message.

[0104] Message format 110 may also include flow identifier fields 110-12. In some embodiments, each trading session (i.e., “connection” or “flow”) established between participant device 130 and gateway 120 may be identified by a flow identifier intended to be unique throughout the electronic trading system 100. As described above in relation to Figure 1D, participant device 130 may be connected to the electronic trading system 100 via one or more flows and via one or more gateways 120. In such embodiments, the message version of all messages exchanged between participant device 130 and electronic trading system 100 via a particular flow, according to the standardized message format 110 (used between nodes in electronic trading system 100), will include a unique identifier for that flow in the flow identifier fields 110-12. In some embodiments, flow identifier fields 110-12 are added by the message generation node. For example, gateway 120 may add an identifier to the flow identifier field 110-12 of the flow associated with the message it receives from participant 130, which gateway 120 introduces into the electronic trading system 100. Similarly, the core compute node 140 may add flow identifiers related to the messages it generates (i.e., response messages such as acknowledgment or agreements, or other outgoing messages including asynchronous messages) to the flow identifier fields 110-12.

[0105] In some embodiments, the flow identifier fields 110-12 include a value that uniquely identifies a logical flow, which may actually be implemented for high availability purposes as multiple redundant trading session connections, possibly via multiple gateways. That is, in some embodiments, the same flow ID may be assigned to two or more redundant flows between participant device 130 and gateway 120. In such embodiments, the redundant flows may be in either an active / standby configuration or an active / active configuration. In an active / active configuration, functionally equivalent messages may be exchanged simultaneously and in parallel across multiple redundant flows between participant device 130 and gateway 120. That is, a trading client may send functionally equivalent messages to the electronic trading system 100 simultaneously and in parallel across multiple redundant flows, and receive multiple functionally equivalent responses from the electronic trading system 100 in parallel across multiple redundant flows. However, the electronic trading system 100 may only act on a single such functionally equivalent message. In an active / standby configuration, one flow may be designated as the active flow at a time among multiple redundant flows, while the other flow among multiple redundant flows may be designated as the standby flow. Transaction messages are, in practice, only exchanged through the currently active flow. Regardless of whether the redundant flow is configured in an active / active or active / standby configuration, messages exchanged through any of the redundant flows can be identified by the same flow identifier stored by the message generation node in the flow identifier field 110-12 of the standardized message format 110.

[0106] As described above, in some embodiments, messages exchanged between nodes in the electronic trading system 100 are sent to the sequencer 150 to be marked with a sequence identifier. Therefore, the message format 110 includes a sequence identifier field 110-14. In some embodiments, an "unmarked message" may be sent with a sequence identifier field 110-14 of an empty blank (e.g., zero) value. In other embodiments, the sequence identifier field 110-14 of an unmarked message may be set to a specific predetermined value that the sequencer does not assign to the message, or an invalid value. In yet another embodiment, the fact that a message is unmarked may be indicated via an indicator in another field of the message (not shown), such as a Boolean value or flag value indicating whether the message is ordered or not. When the sequencer 150 receives an unmarked message, it may subsequently generate an "ordered marked message" by adding a valid sequence identifier value to the sequence identifier field 110-14 of the unmarked message. The valid sequence identifier values ​​in the sequence identifier fields 110-14 of the sequence-marked message uniquely identify the message and further specify the definitive position of the marked message in the relative ranking of marked messages among other marked messages throughout the electronic trading system 100. In this example, the “sequence-marked message” transmitted by the sequencer 150 may subsequently be identical to the corresponding unmarked message received by the sequencer, except that the sequence identifier field 110-14 of the sequence-marked message contains a valid sequence identifier value.

[0107] In some embodiments, the message format 110 may also include a reference sequence identifier field 110-15. A generating node may add the value of the sequence number of the previous message related to the message being generated to the reference sequence identifier field 110-15 of the new message it generates. The value of the reference sequence identifier field 110-15 allows a node in the electronic trading system 100 to correlate the message to a previously related message.

[0108] The preceding related messages referenced in the reference sequence identifier fields 110-15 may be preceding messages in the same “order chain” (i.e., “trading order chain”). According to most financial trading protocols, messages can be logically grouped into an “order chain,” which is a set of messages via a single flow that references, i.e., “derives from,” a common message. An order chain typically begins with a “new order message” sent by participant device 130. The next message in the order chain is typically a response from the electronic trading system (e.g., a “new order accepted” message if the message is accepted by the trading system, or possibly a “new order rejected” message if the message is rejected by the trading system for having an invalid format or invalid parameters, such as an invalid price, in a non-limiting example). The order chain may also include a “cancellation order” message sent by participant device 130 that cancels at least a portion of the quantity of a previously accepted new order (but which still remains unsettled, i.e., including at least some quantities that have not been canceled and / or executed). A “Cancel Order” message can also be approved or rejected by the electronic trading system via a “Cancel Order Approval” or “Cancel Order Rejection” message, and may be part of an order chain. The order chain may also include “Replacement Order” messages sent from participant device 130 that replace the quantity and / or price of a previously approved (but still open) new order. A “Replacement Order” message can also be approved or rejected by the electronic trading system via a “Replacement Order Approval” or “Replacement Order Rejection” message, and may be part of an order chain. An order approved before it is still open may be matched with one or more opposing orders on the opposite side (i.e., a “buy” on one side and a “sell” or “short sell” on the other side).The electronic trading system 100 will then generate a full "execution" message (when the entire quantity of an open order is executed in a single match) or one or more partial "execution" messages (when only a portion of the quantity of an open order is executed in a single match), and these "execution" messages may also be part of the order chain. As mentioned above, the reference order identifier can generally identify other previous messages in the same order chain.

[0109] For example, returning to the base sequence identifier field 110-15, the value for the base sequence number would be the sequence number assigned by the sequencer to the “Incoming” message originating from participant device 130 and introduced to the electronic trading system 100 by gateway 120, so that the corresponding “Outgoing” message, such as a response message generated by compute node 140, can refer to the sequence number value of the incoming message it is responding to. In this example, a “New Order Approval” message or “Executed” message generated by compute node 140 would include in the base sequence identifier field 110-15 the value for the sequence identifier assigned to the corresponding “New Order” message to which compute node 140 responded with a “New Order Approval” message or executed an order with a “Executed” message. However, generally, the value for the base sequence identifier field 110-15 does not necessarily have to be that of a message directly responded to by the electronic trading system 100, but could be that of a previous message that is part of the same order chain, such as the sequence number of a “New Order” or “New Order Approval”.

[0110] In some embodiments, for at least some message types, the gateway 120 may add the value of the sequence identifier for the relevant previous message to the base sequence identifier field 110-15 in the messages they introduce to the electronic trading system 100. For example, the gateway 120 may add the value of the sequence identifier assigned to the previous corresponding "new order" or "new order approval" message to the base sequence identifier field 110-15 in a "cancel order" or "replace order" message. Similarly, the core compute node 140 also adds the value of the sequence identifier for "new order" or "new order approval" to the sequence identifier field 110-15 for the corresponding "cancel order approval" or "replace order approval" message, rather than for the message to which the compute node 140 is directly responding (e.g., not the sequence identifier for the "cancel order" or "replace order" message). Again, the base sequence identifier field 110-15 allows nodes in the electronic trading system 100 to generally correlate messages to one or more previous messages in the same order chain.

[0111] A generating node may include a node-specific timestamp field 110-13 in the messages it introduces to the electronic trading system 100. While the sequence identifiers in the sequence identifier fields 110-14 of the sequence-marked messages output by the sequencer 150 are intended to be unique throughout the electronic trading system 100, the value in the node-specific timestamp field 110-13 may be unique among a subset of messages introduced to the electronic trading system 100 by a particular generating node. Here, we refer to it as a "timestamp," but the value entered in the node-specific timestamp field 110-13 can be any appropriate value unique among the messages generated by that node. For example, the node-specific timestamp can actually be a timestamp or any appropriate monotonically increasing or decreasing value.

[0112] In some embodiments, the message format may include other timestamp fields. For example, some message formats may include a base timestamp field, which may be a timestamp value assigned by the generating node of the previous related message. In such embodiments, the compute node 140 may include a new timestamp value in the node-specific timestamp fields 110-13 for the messages it generates, and may include a timestamp value from the related message in the base timestamp field of the messages generated by the compute node. For example, a "New Order Approval" message generated by a compute node may include a timestamp value of the "New Order" to which it is responding in the base timestamp field of the "New Order Approval Message". Furthermore, in some embodiments, the compute node 140 may not include a new timestamp value in the node-specific timestamp fields 110-13 in the messages they generate, but may simply add a timestamp value from the previous related message to the node-specific timestamp fields 110-13.

[0113] Message format 110 may also include entity type fields 110-16 and entity count fields 110-17. The entity type of a message may depend on whether it is introduced into the electronic trading system 100 by gateway 120 or by compute node 140; in other words, whether the message is an incoming message received by gateway 120 from participant device 130 or an outgoing message generated by compute node 140 and sent to participant device 130. For example, in some embodiments, an incoming message is considered to be of entity type "flow" (and the entity type field 110-16 is added to the incoming message by gateway 120 to represent type "flow"), while an outgoing message is considered to be of entity type "signature" (and the entity type field 110-16 is added to the outgoing message by compute node 140 to represent type "signature"). In such embodiments, the entity count for type "flow" is maintained by gateway 120, and the entity count for type "signature" is maintained by compute node 140.

[0114] Considering the entity type "flow," gateway 120 maintains an incoming message count per flow, counting incoming messages received by gateway 120 through each flow active on gateway 120. For example, if four non-redundant flows are active on gateway 120, each flow will be assigned a unique flow identifier as described above, and gateway 120 will maintain an incoming message count per flow, counting the number of incoming messages received through each of those four flows. In such an embodiment, gateway 120 adds to the entity count field 110-17 of the incoming message the incoming message count per flow related to the flow of the incoming message (which is identified throughout the electronic trading system 100 by the value of the flow identifier added to the message's flow identifier field 110-12).

[0115] In the case of redundant flows in an active / active configuration (i.e., a configuration in which multiple flows receive the same message or at least functionally equivalent messages in parallel from participant devices 130 connected via one or more gateways 120, as described above), each inherent redundant flow will be assigned the same flow identifier, and furthermore, especially if the redundant flows are running on separate gateways 120, the incoming message count per flow may still be maintained separately for each redundant flow. Since the participant devices 130 are expected to send the same set of messages in the same order (i.e., sequence) to the electronic trading system 100 via each of the redundant flows, it is also expected that the entity counts assigned to functionally equivalent messages received via separate redundant flows should be identical. These functionally equivalent incoming messages can be forwarded by the gateways 120 to the sequencer 150 and compute node 140. Therefore, in such embodiments, the sequencer 150 and compute node 140 may receive multiple functionally equivalent incoming messages associated with the same flow identifier, but if the entity count is identical for multiple messages having the same flow identifier, the sequencer 150 and compute node 140 may identify such messages as functionally equivalent. In some embodiments, the sequencer 150 and compute node 140 may track the highest entity count included in the entity count fields 110-17 of the incoming messages associated with each flow, on a flow-by-flow basis. This allows the sequencer 150 and compute node 140 to act only on the first of multiple functionally equivalent incoming messages received by each node, ignoring other functionally equivalent incoming messages that arrive later. For example, in some embodiments, the sequencer 150 may order only the first of such functionally equivalent incoming messages, and the compute node 140 may begin processing only the first of such functionally equivalent messages.If an incoming message received by a node (i.e., sequencer 150 or compute node 140) has an entity count less than or equal to the highest entity count the node has for that flow, the node can infer that the incoming message is functionally equivalent to other previously received incoming messages and may simply ignore any subsequently received functionally equivalent incoming messages.

[0116] Considering the case of entity type "stock symbol," compute node 140 may maintain an outgoing message count per stock symbol and count the outgoing messages generated and transmitted by compute node 140 for each stock symbol handled by compute node 140. For example, if four stock symbols (e.g., MSFT, GOOG, IBM, ORCL) are handled by compute node 140, each stock symbol is assigned a stock symbol identifier added to the stock symbol field 110-2 of the message, as described above, and compute node 140 maintains an outgoing message count per stock symbol and counts the number of outgoing messages generated and transmitted for each of these four stock symbols it handled. In such an embodiment, compute node 140 adds the outgoing message count per stock symbol (associated with the stock symbol of the outgoing message, which is identified throughout the electronic trading system 100 by the value added to the stock symbol field 110-2 of the message) to the entity count field 110-17 of the incoming message.

[0117] In some embodiments, as will be further described later, the compute nodes may be configured such that multiple compute nodes handle a particular tick mark in parallel for reasons of high availability. Even if multiple compute nodes handle a given tick mark, for the definitive prioritization of messages throughout the electronic trading system 100 provided by the sequencer 150, they will be processing incoming messages that refer to the same tick mark in the same order (i.e., sequence), thereby ensuring that functionally equivalent response messages are generated in parallel. When considering outgoing messages sent for a particular tick mark across multiple compute nodes 140, each outgoing message that refers to that tick mark should have a functionally equivalent message sent by each other compute node 140 that actively handles that tick mark. All of these outgoing messages can be sent by the compute nodes 140 to the sequencer 150 and the gateway 120. Therefore, in such embodiments, the sequencer 150 and gateway 120 may receive multiple functionally equivalent incoming messages associated with the same brand symbol, but the sequencer 150 and gateway 120 may identify messages as functionally equivalent if the entity counts for multiple messages having the same brand symbol identifier are identical. In some embodiments, the sequencer 150 and gateway 120 may track the highest entity count included in the entity count fields 110-17 of the outgoing messages associated with the brand symbol for each brand symbol. This allows the sequencer 150 and gateway 120 to act only on the first of multiple functionally equivalent outgoing messages received by each node, ignoring other functionally equivalent outgoing messages that arrive later. For example, in some embodiments, the sequencer 150 may order only the first of such functionally equivalent outgoing messages that arrive. Similarly, the gateway 120 may begin processing only the first of such functionally equivalent messages that arrive.If an outgoing message received by a node (i.e., sequencer 150 or gateway 120) has an entity count less than or equal to the highest entity count previously known by that node for that stock symbol, the node can infer that the outgoing message is functionally equivalent to other outgoing messages previously received and may simply ignore any subsequently received functionally equivalent outgoing messages.

[0118] In an embodiment in which the sequencer 150 orders only the first message of a set of functionally equivalent messages arriving at the sequencer, the sequencer can do so in various ways. For example, other subsequent arriving messages that are functionally equivalent to the first arriving functionally equivalent message may simply be ignored by the sequencer (in this case, only a single ordered message may be output by the sequencer for a set of functionally equivalent messages). Another possibility is that the sequencer tracks an ordered number to assign to the first functionally equivalent message by associating, for example, the entity count of the message, its flow identifier or brand identifier (for messages having the entity types "flow" and "brand identifier," respectively), and its ordered number. This allows the sequencer to output an ordered version of each functionally equivalent message in which the value of the ordered identifier field 110-14 is the same as the one assigned by the sequencer to the first arriving message among the functionally equivalent messages received by the sequencer 150 for all ordered versions of the functionally equivalent messages.

[0119] In other embodiments, the PLC 150 does not need to track whether messages are functionally equivalent, and may assign a unique sequence number to each unordered message arriving at the PLC 150, regardless of whether the message is one of several functionally equivalent messages. In such embodiments, each of the ordered versions of several functionally equivalent messages is assigned a different sequence identifier by the PLC as a value in the sequence identifier fields 110-14. To determine a valid sequence identifier for a set of functionally equivalent messages, a receiving node of ordered functionally equivalent messages in such embodiments may use the sequence identifier of the ordered version of the ordered functionally equivalent message that first arrives at the node. In embodiments where there are direct point-to-point connections between nodes in the electronic trading system 100, the ordered versions of messages are sent out in an ordered order by the PLC 150, so they should be received in the same ordered order among all nodes directly connected to the PLC. Therefore, for all nodes that receive an ordered message via their respective direct point-to-point connection to the sequencer, the first ordered message to arrive among several functionally equivalent ordered messages should have the same value in the order identifier field 110-14.

[0120] As will be apparent from the above, in embodiments having a message format such as message format 110 in addition to a message sequence identifier, there may be a number of other ways to uniquely identify a message throughout the electronic trading system 100. For example, in embodiments where a message includes both a node identifier and a node-specific timestamp, the presence of these two identifiers in the message may be sufficient to uniquely identify the message throughout the electronic trading system 100. Such fields may be understood as containing metadata. Multiple messages containing such identical metadata may be understood as containing common metadata. Similarly, in embodiments where a flow identifier is unique throughout the electronic trading system 100, the combination of the message's flow identifier and node-specific timestamp may be sufficient to uniquely identify the message throughout the electronic trading system 100. Furthermore, the combination of a flow identifier and an entity count may be sufficient to uniquely identify a message of entity type "flow," and the combination of a tick mark identifier and an entity count may be sufficient to uniquely identify a message of entity type "tick mark."

[0121] However, it should be noted that, in addition to the sequence identifier assigned to a message, there may be other ways to uniquely identify a message throughout the electronic trading system 100, but the sequence identifier is still necessary to fairly and definitively specify the relative ranking of messages among other messages generated by other nodes throughout the electronic trading system 100. For example, if node-specific timestamps are actually implemented as timestamp values, even if the system clocks between nodes are perfectly synchronized, each of two different messages generated by different nodes may be assigned the same timestamp value by their respective generating nodes, and the relative ranking between these two messages becomes ambiguous. Even if messages are uniquely identifiable, the receiving nodes of both messages still need a way to determine the relative ranking of the two messages before taking any possible actions on the messages.

[0122] One possible approach for recipient nodes to resolve this ambiguity is to use randomness, for example, by randomly selecting one message to precede another in the relative ranking of messages throughout the electronic trading system 100. However, using randomness to resolve ambiguity does not support "state determinism" throughout the electronic trading system 100. Different recipient nodes would randomly determine different relative rankings among the same set of messages, resulting in unpredictable and non-deterministic behavior within the electronic trading system 100, hindering the proper implementation of important features such as fault tolerance, high availability, and disaster recovery.

[0123] Another approach for recipient nodes to resolve ambiguity in prioritization could be, for example, a predetermined prioritization method based on the node identifier associated with the message. However, such an approach would act against the important goal of fairness by giving higher priority to some messages simply based on the node identifier of the node that introduced the message into the electronic trading system 100. For example, some participant devices 130 would be given priority simply because they happened to connect to the electronic trading system 100 via a gateway 120 that is considered high in the predetermined prioritization method.

[0124] If messages are uniquely identified through an entity count and either a ticker symbol identifier or a flow identifier, depending on whether each message has a "ticker symbol" or "flow" entity type, then a deterministic ranking may exist among other messages associated with that ticker symbol (for messages with the "ticker symbol" entity type) or flow (for messages with the "flow" entity type). However, the ranking among other messages associated with different ticker symbols and flows remains non-deterministic.

[0125] Therefore, even if other fields in the message format 110 may be sufficient to uniquely identify a message throughout the electronic trading system 100, a sequence identifier assigned to the message by the sequencer 150 may still be necessary to fairly and definitively specify the ranking of the message relative to other messages in the electronic trading system 100. In such embodiments, the sequencer 150 (or a single currently active sequencer if multiple sequencers 150 exist) acts as a reliable source of truly definitive ranking among sequence-marked messages throughout the electronic trading system 100.

[0126] In some embodiments, a node in the electronic trading system 100 may receive two versions of a message: an unordered (unmarked) version of the message, as introduced into the electronic trading system 100 by the generating node, and a (marked) version of the message that includes an order identifier assigned by the sequencer 150. This can occur in embodiments where the generating node sends the unmarked message to one or more receiver nodes and the sequencer 150. The sequencer 150 may then send an ordered, marked version of the same message to a set of nodes including the same receiver nodes.

[0127] As described above, the marked version of a message is useful for determining the relative processing order (i.e., position in order) of a message among other marked messages in the electronic trading system 100, but it can also be useful for a receiving node to receive the unmarked version of a message. For example, in unexpected cases (e.g., in embodiments where there are direct connections between nodes), it is certainly possible for the unmarked version of a message to be received before the marked version of a message, because the marked version of a message is transmitted through the sequencer 150 via an intervening hop. Therefore, in some embodiments, a receiving node may have the opportunity to activate the processing of an unmarked message in response to its receipt of an unmarked message, even before it receives a marked version of a message that reliably indicates the relative ranking of a marked message among other marked messages.

[0128] Nodes receiving both marked and unmarked versions of the same message can correlate the two versions of the message through the same identification information or “common metadata” in both versions of the message. For example, as described above, a generating node may include in the messages it generates (i.e., unmarked messages) a node identifier and a node-specific timestamp that together uniquely identify each message throughout the electronic trading system 100. In embodiments where the marked and unmarked versions of a message are essentially identical except for the sequence identifier assigned by the sequencer 150, the marked message will also include the same node identifier and node-specific timestamp that are also included in the corresponding unmarked message, thereby allowing recipient nodes of both versions of the message to correlate the marked and unmarked versions. Thus, a marked message indicates the relative ranking of a marked message to other marked messages throughout the electronic trading system 100, but for correlations that may occur between the unmarked and marked versions of the same message, the marked message indicates its relative ranking to other messages (marked or unmarked) throughout the electronic trading system 100 (at least indirectly through the correlations described above). It should be understood that a node in the electronic trading system 100 may correlate the ordered version of a message with the unordered version in another manner that uniquely identifies the message described above. For example, the correlation between ordered and unordered messages may be performed by a combination of a flow identifier and a node-specific timestamp. Such correlation may be additionally or alternatively performed by the entity count of the message along with the ticker symbol identifier or flow identifier in the message for messages having entity types "ticker symbol" and "flow," respectively.

[0129] In the era of high-speed trading where microseconds or even nanoseconds are critical, participant devices 130 that exchange messages with the electronic trading system 100 are often highly susceptible to latency, and low, predictable latency is preferred. The configuration shown in Figure 1D addresses this requirement by providing a point-to-point mesh 172 architecture between at least each of the gateways 120 and each of the compute nodes 140. In some embodiments, each gateway 120 in the mesh 172 may have a dedicated high-speed direct connection 180 to the compute nodes 140 and the sequencer 150.

[0130] For example, dedicated connection 180-1-1 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-1 (i.e., Core1), dedicated connection 180-1-2 is provided between gateway 120-1 (i.e., GW1) and core compute node 140-2 (i.e., Core2), and so on. Then, example connection 180-gc is provided between gateway 120-g and compute node 140-c, example connection 180-sc is provided between sequencer 150 and core compute node 140-c (i.e., Core c), example connection 180-gw1-s1 is provided between gateway 120-1 (i.e., GW g) and sequencer 150-1, and example connection 180-c1-s1 is provided between core compute node 140-1 (i.e., Core1) and sequencer 150-1.

[0131] It should be understood that each dedicated connection 180 in the point-to-point mesh 172 is, in some embodiments, a point-to-point direct connection that does not utilize a shared switch. A dedicated or direct connection may here be interchangeably referred to as a direct or dedicated “link,” which is a direct connection dedicated to (i.e., non-shared) communication between two endpoints. Such a dedicated / direct link may be any appropriate interconnection or interface as further disclosed below, and is not limited to network links such as wired Ethernet network connections or other types of wired or wireless network links. A dedicated / direct connection / link may here also be referred to as an end-to-end path between two endpoints. Such an end-to-end path may be a single connection / link or may include serial connections / links. However, the bandwidth of the dedicated / direct connection / link as a whole, i.e., from one endpoint to the other endpoint, is non-shared, and the bandwidth and latency of the dedicated / direct connection / link are not affected by resource utilization, even if elements are traversed. For example, a dedicated / direct connection / link may traverse one or more buffers or other elements whose bandwidth or latency is not affected by utilization. However, dedicated / direct connections / links do not traverse shared network switches because such switches can impact bandwidth and / or latency due to their shared use.

[0132] For example, in some embodiments, the dedicated connection 180 in the point-to-point mesh 172 may be provided in a number of forms, such as 10 Gigabit Ethernet (GigE), 25GigE, 40GigE, 100GigE, InfiniBand, Peripheral Component Interconnect-Express (PCIe), RapidIO, Small Computer System Interface (SCSI), FireWire, Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or custom serial or parallel buses. Therefore, although the compute engine 140, gateway 120, sequencer 150, and other components may be referred to as “nodes” here, the use of terms such as “compute node,” “gateway node,” “sequencer node,” or “mesh node” should not be interpreted as meaning that a particular component is necessarily connected using a network link, as other types of interconnections or interfaces are possible. Furthermore, the “nodes” disclosed herein may be any appropriate hardware components, software components, firmware components, or combinations thereof configured to perform the respective functions described above for a node. As will be described in more detail below, a node may be a programmed general-purpose processor, a dedicated hardware device such as a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other hardware devices or groups of devices, logic within a hardware device, a printed circuit board (PCB), or other hardware components.

[0133] It should be understood that the nodes disclosed herein may be independent elements or may be integrated with each other within a single element, such as a single FPGA, ASIC, or other element, configured to execute logic and perform the functions of the nodes described herein. Furthermore, the nodes may be entities of software execution logic executed by a general-purpose computer and / or one of the aforementioned devices.

[0134] Traditional approaches that connect components such as the compute engine 140, gateway 120, and sequencer 150 through one or more shared switches do not provide the lowest possible latency. These traditional approaches also result in unpredictable spikes in latency during periods of increased message traffic.

[0135] In the exemplary embodiment, dedicated connections 180 are also directly provided between each gateway 120 and each sequencer 150, and between each sequencer 150 and each core compute node 140. Furthermore, in some embodiments, dedicated connections 180 are provided between all sequencers so that exemplary sequencer 150-1 has a dedicated connection 180 to each of the other sequencers 150-2, ..., 150-s. Although not shown in Figure 1D, in some embodiments, dedicated connections 180 may also be provided between all gateways 120 so that each gateway 120-1 has a dedicated connection 180 to each of the other gateways 120-2, ..., 120-g as further disclosed below with respect to Figure 3. Similarly, in some embodiments, the dedicated connection 180 is also provided between all compute nodes 140 such that the exemplary core compute node 140-1 has a dedicated connection 180 to each of the other core compute nodes 140-2, ..., 140-c as further disclosed below with respect to Figure 3.

[0136] It should also be understood that, in some embodiments, a dedicated connection 180 between two nodes (for example, between any two nodes 120, 150, or 140) may be implemented as multiple redundant dedicated connections between those two nodes for increased redundancy and reliability. For example, the dedicated connection 180-1-1 between gateway 120-1 and core compute node 140-1 may actually be implemented as a pair of dedicated connections.

[0137] Furthermore, according to some embodiments, any message sent by a node is sent in parallel to all nodes directly connected to it in the point-to-point mesh 172. Each node in the point-to-point mesh 172 may decide, for example, based on its configuration, whether to take any action upon receiving a message or simply ignore it. In some embodiments, a node does not have to completely ignore a message. Even if a node does not take any substantial action upon receiving a message due to its configuration, it may take at least minimal action, such as consuming one of the sequence numbers assigned to the message by the sequencer 150. That is, in such embodiments, a node may track the last received sequence number to ensure that if the node takes a more substantial action on a message, it does so in an appropriate ordering order.

[0138] For example, suppose a message containing a trade order to "sell 10 shares of Microsoft at $190.00" originates from a participant device 130-1, such as a trader's personal computer, and arrives at gateway 120-1 (i.e., GW1). This message is sent to all core compute nodes 140-1, 140-2, ..., 140-c, even though only core compute node 140-2 is currently performing matching for orders for Microsoft shares. All other core compute nodes 140-1, 140-3, ..., 140-c may ignore the message upon receipt or take only minimal action on it. For example, the only action taken by 140-1, 140-3, ..., 140-c may be to consume the sequence number assigned to the message by sequencer 150-1. The message is also sent to all sequencers 150-1, 150-2, ..., 150-s, even if a single sequencer (in this example, sequencer 150-1) is the currently active sequencer serving the mesh. The other sequencers 150-2, ..., 150-s also receive the message and are given the opportunity to take over as the currently active sequencer if sequencer 150-1 (the currently active sequencer) fails, or if moving to a different active sequencer would increase the overall reliability of the electronic trading system 100. One or more of the other sequencers (e.g., sequencer 150-2) are also responsible for relaying the system state to the disaster recovery site 155. The disaster recovery site 155 may include a copy of the electronic trading system 100 at another physical location, which comprises physical or virtual entities of some or all of the individual components of the electronic trading system 100.

[0139] By sending each message in parallel to all directly connected nodes, the electronic trading system 100 reduces complexity and further promotes redundancy and high availability. If all directly connected nodes receive all messages by default, multiple nodes may be configured to act redundantly on the same message. Returning to the above example of an order to "sell 10 shares of Microsoft at $190.00", in some embodiments, multiple core compute nodes 140 may simultaneously perform matching for the Microsoft stock order. For example, both core compute node 140-1 and core compute node 140-2 would simultaneously perform matching for the Microsoft stock message, and after receiving the incoming message for the "sell" order, each core compute node 140-1 and core compute node 140-2 may independently generate response messages, such as acknowledgment or execution messages, which are sent through the sequencer 150 to the gateway 120 and passed to one or more participant devices 130.

[0140] Due to the strict ranking and state determination guaranteed by the sequencer 150, it can be ensured that each of the related response messages independently generated and transmitted from the core compute nodes 140-1 and 140-2 is substantially and functionally equivalent. Therefore, the architecture of the electronic trading system 100 immediately supports redundant message processing, increasing the availability and resilience of the system. In such embodiments, the gateway 120 may receive multiple related outgoing messages from the core compute node 140 for the same corresponding incoming message. Because it can be ensured that these multiple related response messages are equivalent, the gateway 120 only needs to process the first outgoing message received and ignores subsequent related outgoing messages corresponding to the same incoming message. In some embodiments, the “first” and “subsequent” messages may be identified by their related sequence numbers, since the messages may be sequence-marked messages. However, in other embodiments, such as when the sequencer 150 assigns a single sequence identifier among multiple functionally equivalent messages, messages may be identified as functionally equivalent based on other identifying information in the message, such as the values ​​in the entity type fields 110-16 and the entity count fields 110-17, as further described in relation to Figure 1E.

[0141] Therefore, enabling the gateway 120 to act on the first of several functionally equivalent related response messages to arrive at them can also improve the overall latency of the electronic trading system 100. Furthermore, the electronic trading system 100 can be easily configured so that any incoming message is processed by multiple compute nodes 140, each of which generates equivalent response messages that can be processed by the gateway 120 on a first-come, first-served basis. Such an architecture provides high availability without a noticeable impact on latency if a compute node 140 has not handled an incoming message for a given period of time (whether due to system failure, node reconfiguration, or maintenance work).

[0142] The point-to-point mesh 172 architecture of such an electronic trading system 100 provides multiple built-in redundant paths in addition to maintaining low predictable latency and redundant message processing. As can be seen from the diagram, there are multiple paths between any gateway 120 and any compute node 140. Even if the direct connection 180-1-1 between gateway 120-1 and compute node 140-1 becomes unavailable, communication is still possible between these two elements via alternative paths, such as by traversing one of the sequencers 150 instead. Therefore, more generally, in a point-to-point mesh 172, there are multiple paths between any node and any other node.

[0143] Furthermore, this point-to-point mesh architecture inherently maintains another important objective of the financial trading system, namely fairness. The point-to-point architecture, with direct connections between nodes, ensures that paths between any gateway 120 and any core compute node 140, or between the sequencer 150 and any other node, have identical or at least very similar latencies. Thus, two incoming messages sent simultaneously to the sequencer 150 from two different gateways 120 should arrive at the sequencer 150 substantially simultaneously. Similarly, an outgoing message sent from the core compute node 140 should be sent simultaneously to all gateways 120 and received by each gateway substantially simultaneously. Because the point-to-point mesh topology does not favor any one gateway 120, the opportunity for connecting to a particular gateway 120 to give a participant device 130 an unfair advantage is minimized.

[0144] Furthermore, the point-to-point mesh architecture of the electronic trading system 100 allows for easy reconfiguration of the function of a node, i.e., whether the node is currently acting as a gateway 120, a core compute node 140, or a sequencer 150. Performing such reconfiguration is particularly easy in embodiments where each node has a direct connection to itself and to each other node in the point-to-point mesh. When each node is connected to each other node in the mesh via a direct connection, changing the function of a node within the mesh (for example, changing the function of a node from a core compute node 140 to a gateway 120, or from a gateway 120 to a sequencer 150) would require rewiring or recabling of the connections 180 (whether physical or virtual) within the point-to-point mesh 172. In such embodiments, the internally required reconfiguration of the point-to-point mesh 172 can be easily achieved through configuration changes performed remotely. If a node is reconfigured to act as a new gateway 120, or reconfigured from acting as a gateway 120 to another function, some incidental external networking changes to the point-to-point mesh 172 may be required, but the internal wiring of the mesh may remain unchanged.

[0145] Therefore, in some embodiments, the reconfiguration of node functionality can be achieved live and even dynamically during trading hours. For example, due to changes in the load characteristics or new demands of the electronic trading system 100, it may be useful to reconfigure the core compute node 140-1 to act as an additional gateway 120 instead. After some possible redistribution of state or configuration to other compute nodes 140, the new gateway 120 becomes available and begins accepting new connections from participant devices 130.

[0146] In some embodiments, slower and potentially higher-latency shared connections 182 may be provided between system components, such as between the gateway 120 and / or the core compute node 140. These shared connections 182 may be used for maintenance, control, management, and / or similar tasks that do not require very low-latency communication, in contrast to messages related to trading activities performed over dedicated connections 180 in the point-to-point mesh 172, such as messages 106 and responses 107 disclosed above with respect to Figure 1B-2. In contrast to the first direct connections 180-a, second direct connections 180-b, and third direct connections 180-c that carry traffic related to trading activities, shared connections 182g and 182c carry non-trading activity type traffic. The shared connections 182 carrying non-trading traffic may be via one or more shared networks and one or more network switches, and the nodes in the mesh may be distributed across these shared networks in different ways. For example, in some embodiments, all gateways 120 may be in the gateway-wide shared network 182-g, compute nodes 140 may be in their own respective compute node-wide shared network 182-c, and sequencers 150 may be in their own separate sequencer-wide shared network 182-s, while in other embodiments, all nodes in the mesh may communicate via the same shared network for work that is not affected by their latency.

[0147] A distributed computing environment, such as an electronic trading system 100, may follow a high-resolution clock to maintain tight synchronization among its various components. For this purpose, one or more of the nodes 120, 140, and 150 may, in some embodiments, be given access to a clock such as a high-resolution Global Positioning System (GPS) clock 195. For the purposes described below, the gateway 120, compute node 140, and sequencer 150 connected to the point-to-point mesh 172 are referred to as a “mesh node,” which may have an architecture as further disclosed below with respect to Figure 2.

[0148] Figure 1F is a flowchart of an exemplary embodiment of a process 125 that can be executed by the electronic trading system 100 to process a message. Referring to Figure 1B-2, gateway 120-1 receives a message from a participant device (131) and may send message 106 to sequencer 150-1 and compute node 140-1 (132). Gateway 120-1 may optionally include an identifier generated by gateway 120-1 in message 106, and in an electronic trading system having multiple compute nodes and / or sequencers, gateway 120-1 may send message 106 to all such nodes. Upon receiving message 106, compute node 140-1 first determines whether message 106 is one that compute node 140-1 has assigned to a process, and may make this determination based on one or more values ​​or identifiers in message 106 (for example, values ​​of common parameters such as a ticker symbol representing a given financial instrument or a transaction type). If message 106 is assigned to processing, compute node 140-1 may perform preprocessing on message 106 (134) to generate preliminary results. For example, compute node 140-1 may read into memory information related to the values ​​referenced in message 106 (e.g., the ticker symbol of a financial instrument). Alternatively or additionally, in some embodiments, preprocessing may include, in non-limiting examples, performing matching functions such as generating a response message including an acknowledgment or execution message, or performing a preliminary update of the order book for the values ​​referenced in message 106 (e.g., the ticker symbol for a financial instrument). Such preprocessing may be performed in a manner that does not have harmful side effects, or may be “rolled back” atomically (i.e., per transaction) if it is later determined that the message on which preprocessing was performed was not received in order, as necessary.

[0149] Simultaneously, the sequencer 150-1 may order message 106 by associating it with a unique order identifier that allows its position within a series of messages that may need to be processed in a given order (i.e., sequence) (133), thereby generating an ordered version of message 106, i.e., an ordered message 106'. Upon completion, the sequencer 150-1 may send the ordered message 106', which includes message 106 (or its representation) and the order identifier, to the compute node 140-1 (135). Upon receiving the ordered message 106', the compute node 140-1 may, in some embodiments, determine whether the (unordered) messages 106 from which it performed its preprocessing were received in order, i.e., verify the order (136). As part of verifying the order, compute node 140-1 may correlate message 106 to the ordered message 106' by common metadata or common identification information in both versions of the message. In processing 125 of the exemplary embodiment, message 106 is received in order and compute node 140-1 can proceed to continue the step of processing message 106 (137). As a result of the preprocessing, compute node 140-1 has already completed some operations (e.g., looking up values ​​in cache memory) that it would have performed after receiving the ordered message 106', thereby allowing compute node 140-1 to complete the step of processing message 106 faster than it would have if the preprocessing operations had not been performed.

[0150] Figure 1G is a flowchart of an exemplary embodiment of process 165, illustrating another exemplary embodiment of the operation of the electronic trading system. Process 165 may be performed by the electronic trading system 100 to process a message. In the embodiment of Figure 1G, the electronic trading system 100 includes at least a second gateway, gateway 120-2. Referring to Figure 1B-2, gateway 120-1 may receive a first message (M1) from a participant device (166) and, in response, send a message 106 corresponding to the first message M1 to sequencer 150-1 and compute node 140-1 (167). Message 106 may also be referred to as the first message M1, but it should be understood that message 106 may not be identical to M1 because, in non-exclusive examples, the gateway may modify the message to include different metadata, such as an identifier for gateway 120-1. Gateway 120-2 receives a second message (M2) from another participant device (168) and may send a message corresponding to the second message M2 to sequencer 150-1 and compute node 140-1 (169). The message corresponding to the second message M2 may also be referred to here as the second message M2. In this example, the first and second messages belong to a common set of messages, and the relative order (i.e., sequence) in which the first and second messages are processed may affect the state of the electronic trading system 100. In the exemplary embodiment of Figure 1G, compute node 140-1 receives the second message before receiving the first message, and if compute node 140-1 confirms that the second message is one that it has assigned to process, compute node 140-1 may then perform preprocessing on the message (171). Upon receiving the first message (M1), compute node 140-1 may also perform preprocessing on the first message (M1) (173).

[0151] The sequencer 150-1 may receive both the first and second messages. The sequencer 150-1 may assign to each of the first and second messages a unique sequence identifier that can be used to determine the relative ranking, i.e., order, of the first and second messages (170, 174). In some embodiments, the sequence identifier assigned to a message by the sequencer may, as an unrestricted example, be a monotonically increasing sequence number. Furthermore, in some embodiments, the value of the sequence identifier assigned to a message by the sequencer may be based on the time of arrival of the message in the sequencer 150-1. That is, in some embodiments, as an unrestricted example, messages may be placed in a first-in, first-out (FIFO) queue each time they are received by the sequencer 150-1, and a sequence identifier may be assigned each time they are removed from the FIFO queue. In the exemplary embodiment of Figure 1G, since message M1 is received before message M2 in sequencer 150-1, in embodiments where the sequence identifier is a monotonically increasing sequence number, sequencer 150-1 will assign message M1 a lower sequence number relative to the other sequence number assigned to message M2. Once the process of assigning a sequence identifier to each message is complete, sequencer 150-1 may send the first and second messages or the marked messages (i.e., messages containing the sequence identifier) ​​corresponding to their representations to compute node 140-1 (175, 177). In some embodiments, when compute node 140-1 receives the marked messages, it may determine whether the (unordered) message 106 that it preprocessed was received in order, i.e., verify the order (176). As part of verifying the order, compute node 140-1 may correlate the unordered message 106 with the order-marked message 106' by common metadata or common identification information in both versions of the message. In process 165, compute node 140-1 did not receive the second message in the correct order.In this result, some or all of the preprocessing artifacts (for example, a cache reference for the values ​​of common parameters of the messages, as an unrestricted example) may be optionally discarded, or, in the case of other types of preprocessing, may be atomically rolled back. The compute node 140-1 may then proceed to process both the first and second messages in order (i.e., sequence) as indicated by the sequencer 150-1 (178).

[0152] Even if compute node 140-1 determines that an unordered message was not received in order, node 140-1 still benefits from preprocessing the message. By preprocessing the message, compute node 140-1 determines that it will likely need to process the message in the future, thereby taking proactive time-saving action. For example, information necessary to fully process a message, such as information about current open trading orders for the same stock symbol, must be present in faster memory (e.g., FPGA memory such as the fixed logic memory 250 shown in Figure 2, or other cache memory) before the message is fully processable. If it is not yet present in faster memory, it may first need to be retrieved from slower memory (e.g., a hard disk, or DRAM such as the DRAM 280 shown in Figure 2). Regarding the values ​​of common parameters such as stock symbols for messages that have not been recently referenced, this information may not currently exist in high-speed memory. However, compute node 140-1 can pre-copy memory even for non-sequential messages, thereby saving time if ordered messages arrive promptly afterward in the correct order (i.e., sequence). Provided that high-speed memory is large enough to hold information for multiple values, it may even be beneficial to receive a sequence of multiple consecutive non-sequential messages by allowing the compute node to pre-fetch information for a large number of values. Therefore, compute node 140-1 may temporarily store the results of pre-processing operations for reference during subsequent processing operations, rather than ignoring them.

[0153] In some exemplary embodiments, the results of preprocessing by compute nodes may still be valid in instances where there is no ordering. For example, in embodiments where each compute node handles a subset of operations, such as a subset of values ​​for common parameters of a message (e.g., a limited number of stock symbols or other indicators of a financial instrument), a compute node may still receive messages that reference other stock symbols that it does not handle, but the ordering may only be relevant to the compute node insofar as it affects the values ​​that the compute node handles. Furthermore, the ordering of messages that reference different values ​​may be considered independent of each other. For example, if a message is not received in order relative to other messages that reference stock symbols of different values, the order in which these two messages are processed will not affect the outcome of the matching function operation, no further action regarding preprocessing (e.g., rollback) will be required, and the preprocessed results may ultimately be used without modification and / or constrained to the state of the distributed system.

[0154] It should be understood that in some embodiments, verifying the order (136 in Figure 1F and 176 in Figure 1G) may not be necessary depending on the nature of the preprocessing performed by compute node 140-1. For example, if the preprocessing performed on an unordered message does not have harmful side effects and does not affect the results of processing other messages (e.g., reading data related to the stock symbols referenced in the message into high-speed memory), then it may not be necessary to verify that the preprocessing of the unordered message was performed in the order determined by sequencer 150-1, and indeed, it may not be necessary to discard or atomically roll back the results of the preprocessing. In such embodiments, compute node 140-1 may simply then complete the message processing in accordance with the order in which it received the ordered message 106', in the order specified by the order identifier in the ordered message 106', while utilizing the preprocessed results.

[0155] The amount of preprocessing that may be performed on an unordered message, and whether the results of that preprocessing need to be discarded or rolled back, may depend on the fields in the message, such as the message type field 110-1, the stock ticker symbol field 110-2, the side field 110-3, or the price field 110-4 according to the embodiment in Figure 1E. It may also depend on whether other unordered messages referencing the same value for common parameters in the message, such as the same stock ticker symbol, are currently unprocessed (i.e., the corresponding ordered message has not yet been received).

[0156] For example, if an unordered message with the message type "New Order" is received by core compute node 140-1, core compute node 140-1 reads the ticker symbol information related to the relevant portion of the order ledger into high-speed memory. If the new order matches an open order in the order ledger, compute node 140-1 begins generating a "Contracted" message, but refrains from committing to updating the order ledger and sending the "Contracted" message until it receives an ordered version of the message. However, if compute node 140-1 also receives other unordered "New Order" messages that refer to the same ticker symbol, side, price, etc., which also constitute potential matches for the same open order in the order ledger, core compute node 140-1 may perform its preprocessing in a different manner. In some embodiments, core compute node 140-1 may generate competing potential "Contracted" messages for each of two unordered "New Order" messages that could act as matches for an open order. Based on the ordered version of the messages, one of the potential "settlement" messages may be discarded, while the other is confirmed in the order ledger and sent to gateway 120. In other embodiments, if two or more potential unordered messages could potentially match the same open order, compute node 140-1 may not perform any preprocessing that would need to be discarded or rolled back (for example, not creating any potential "settlement" messages), or it may abort or suspend any such preprocessing with respect to these unordered messages.

[0157] As another example, an unprocessed, unordered "New Order" message that is a potential match for an outstanding order in the order ledger may conflict with an unprocessed, unordered "Replace Order" or "Cancel Order" message that attempts to replace or cancel the same outstanding order in the order ledger, respectively, acting as a potential match for the "New Order" message. In this case, depending on the relative order assigned by the sequencer between the "New Order" message and the "Replace / Cancel Order" message, the final result may be either a match between the outstanding order in the order ledger and the "New Order" message, or the outstanding order may be canceled or replaced by a new order of a different price or quantity. Compute node 140-1 cannot determine which of these two results should occur until an ordered version of the conflicting unordered messages is received by sequencer 150-1.

[0158] In this case, compute node 140-1 may perform preprocessing in different ways. In some embodiments, if there are multiple conflicting unprocessed unordered messages, compute node 140-1 may simply perform preprocessing that does not need to be rolled back or discarded, such as loading the relevant portion of the order book related to the tick marks referenced in both conflicting messages into high-speed memory. In other embodiments, compute node 140-1 may perform additional preprocessing, such as configuring one or more provisional potential responses, each corresponding to one of several conflicting scenarios. For example, compute node 140-1 may create potential "executed" messages and / or potential "replacement accepted" or "cancellation accepted" messages, and optionally perform provisional updates to the order book corresponding to one or more of several possible outcomes. In some embodiments, compute node 140-1 may perform this additional preprocessing for all such conflicting scenarios, while in other embodiments, compute node 140-1 may perform the additional preprocessing for only one or a subset of the conflicting scenarios. For example, compute node 140-1 may perform additional preprocessing on an unprocessed unordered message only if there are no other unprocessed, conflicting unordered messages. Alternatively or additionally, compute node 140-1 may prioritize performing additional preprocessing on conflicting unordered messages depending on the amount of time and / or complexity involved in rolling back or discarding the results of preprocessing. When compute node 140-1 receives an ordered version of an unprocessed unordered message, it may then determine the order in which the unprocessed unordered message should be processed (as assigned by sequencer 150-1) and complete processing of the message in that order. In some embodiments, this may involve rolling back or discarding one or more results of preprocessing.

[0159] In addition to the types of preprocessing already described above, in some embodiments, compute node 140-1 may additionally or alternatively perform preprocessing regarding the validity of a message to determine whether to accept or reject the message. For example, preprocessing may include performing real-time risk checks on a message, such as confirming that the price or quantity specified in the message does not exceed a maximum value (i.e., "maximum price check" or "maximum quantity check"), that the tick mark in the message is a known tick mark (i.e., "unknown tick mark check"), that trading is currently permitted for that tick mark (i.e., "tick mark suspension check"), or that the price is properly specified according to the correct number of decimal places (i.e., "subpenny check"). In some embodiments, the type of preprocessing may include "self-dealing prevention" validity checks to prevent certain potential matches from becoming self-dealing, i.e., that is, preventing a trading client from matching itself, if "self-dealing prevention" is enabled for a particular client or trading order. If a trading order fails one or more of these validity checks, the electronic trading system 100 may respond with an appropriate rejection message. Although these validations are described as being performed by compute node 140-1 in the above embodiments, it should be understood that at least some of these types of validations may be performed alternatively or additionally by gateway 120 or other nodes in the electronic trading system 100 in some embodiments.

[0160] In a further embodiment, it may be beneficial or necessary for gateway 120-1 to be notified of a unique system-wide sequence identifier associated with a message originating from a client. This information may enable gateway 120-1 to match the initial incoming message to a unique sequence number. This is used to ensure the proper prioritization of messages throughout the electronic trading system 100. Such a configuration in the gateway may require the electronic trading system 100 to achieve state determinism to provide fault tolerance, high availability, and disaster recovery with respect to activity at the gateway. One solution for configuring gateway 120-1 to hold information about a sequence identifier associated with an incoming message is to have gateway 120-1 wait for a reply from sequencer 150-1 along with the sequence identifier before forwarding the message to compute node 140-1. Such an approach may add latency to message processing. In a further example, in addition to forwarding the sequence-marked message initially received from gateway 120-1 to compute node 140-1, sequencer 150-1 may also send sequence-marked messages (e.g., marked message 106' as shown in Figure 1B-2) in parallel to gateway 120-1. As a result, gateway 120-1 can retain sequence identifier information while minimizing latency in the electronic trading system 100.

[0161] Figure 2 is a block diagram of an exemplary embodiment of a mesh node in a point-to-point mesh architecture of an electronic trading system, such as the electronic trading system 100 disclosed above. Figure 2 shows an exemplary embodiment of a mesh node 200 in the point-to-point mesh 172 architecture of the electronic trading system 100. The mesh node 200 may represent, for example, a gateway 120, a sequencer 150, or a core compute node 140. In this example, the functionality of the mesh node 200 is distributed across both hardware and software, but the mesh node 200 may be implemented in any appropriate combination of hardware and software, including purely hardware and purely software embodiments, and in some embodiments, one or all of the gateway 120, compute node 140, and / or sequencer 150 may be implemented with commercially available components.

[0162] In the embodiment shown in Figure 2, to achieve low latency, some functions are implemented in hardware in a fixed logic device 230, while other functions are implemented in software in a device driver 220 and a mesh software application 210. The fixed logic device 230 can be implemented in any appropriate manner, including an application-specific integrated circuit (ASIC), an embedded processor, or a field-programmable gate array (FPGA). The mesh software application 210 and the device driver 220 can be implemented as instructions that execute one or more programmable data processors, such as a central processing unit (CPU). Different versions or configurations of the mesh software application 210 may be installed on the mesh node 200 depending on its role. For example, different versions or configurations of the mesh software application 210 may be installed based on whether the mesh node 200 is operating as a gateway 120, a sequencer 150, or a core compute node 140.

[0163] Any suitable physical communication link layer may be employed (including Universal Serial Bus (USB), Peripheral Component Interconnect (PCI) Express (i.e., PCI-E), High-Definition Multimedia Interface (HDMI), 10 Gigabit Ethernet (GigE), 40 Gigabit Ethernet, 100 Gigabit Ethernet, or InfiniBand (IB), via fiber or copper cable), but in this example, the mesh node 200 has multiple low-latency 10 Gigabit Ethernet Small Form Factor Pluggable Plus (SFP+) connectors (interfaces) 270-1, 270-2, 270-3, ..., 270-n (collectively referred to as connector 270). Connector 270 may be connected directly to other nodes in a point-to-point mesh, for example, via dedicated connection 180, via shared connection 182, and / or to participant devices 130 via gateway 120. These connectors 270 are electronically coupled in this example to 10GigE media access (MAC) cores 260-1, 260-2, 260-3, ..., 260-n (collectively referred to as GigE core 260), respectively, which in this embodiment are implemented by a fixed logic device 230 to ensure minimal latency. In other embodiments, the 10GigE MAC cores 260 may be implemented by an external function of the fixed logic device 230, for example, in a PCI-E network interface card adapter.

[0164] In some embodiments, the fixed logic device 230 may include other components. In the example in Figure 2, the fixed logic device 230 also includes components of the fixed logic 240. In some embodiments, the fixed logic component 240 may perform different functions depending on the role of the mesh node 200, for example, whether it is a gateway 120, a sequencer 150, or a core compute node 140. The fixed logic device 230 also includes fixed logic memory 250, which can be memory accessed by the fixed logic 240 with minimal latency. The fixed logic device 230 also includes a PCI-E core 235, which can perform PCI Express functionality. In this example, PCI Express is used as a conduit mechanism for transferring data between hardware and software, more specifically between the fixed logic device 240 and the mesh software application 210, via the PCI Express bus 233 by the device driver 220. However, any appropriate data transfer mechanism, including direct memory access (DMA), shared memory buffers, or memory mapping, may be employed between the hardware and software.

[0165] In some embodiments, the mesh node 200 may also include other hardware components. For example, depending on its role in the electronic trading system 100, the mesh node 200 may, in some embodiments, include a high-resolution clock 195 (also illustrated and disclosed in relation to Figure 1D) used to implement high-resolution clock synchronization between nodes in the electronic trading system 100. Dynamic random access memory (DRAM) 280 may also be included in the mesh node 200 as additional memory in relation to the fixed logic memory 250. The DRAM 280 may be any suitable volatile or non-volatile memory, including one or more random access memory banks, hard disks and solid-state disks, and may be accessed via any suitable memory or storage interface. As disclosed above, the mesh node 200 may represent a gateway 120, a sequencer 150 or a core compute node 140, and may be configured in a point-to-point mesh architecture as disclosed above with respect to Figures 1A-D and further disclosed below with respect to Figure 3.

[0166] From the perspective of the exemplary embodiments described in Figures 1B-1, 1B-2, 1C, 1D, 1E and Figure 2, it should be clear that the point-to-point mesh architecture 172 of the electronic trading system 100 contributes to latency improvements in numerous ways. In embodiments where messages are exchanged between nodes within the electronic trading system 100 via direct dedicated connections 180 without crossing switches, the very act of avoiding switches enables numerous latency-related advantages. a) The time required for a switch to process a single message (the time interval from when a message enters the switch until it leaves the switch) is completely eliminated. This switch processing latency is typically about 1.0 microsecond and includes buffering time and time to route the message to the appropriate destination, as specified in the message header. b) Two transmission times (transmission time between the transmitting node and the switch and transmission time between the switch and the receiving node) are replaced by a single transmission time for direct message transmission between the transmitting node and the receiving node. c) Communicating via direct point-to-point connections without switches also eliminates the requirement to exchange messages within a mesh following a specific established protocol, such as TCP / IP, which may be required by switches but could also add processing time overhead to both the transmitting and receiving nodes. The time required for the transmitting node to perform protocol-specific processing, such as constructing the TCP / IP header and calculating the checksum, and the time required for the receiving node to further perform similar protocol-specific processing, such as interpreting the TCP / IP header and verifying the checksum, is approximately 0.5 microseconds for each of the transmitting and receiving nodes, totaling approximately 1.0 microsecond. This protocol-specific overhead can optionally be eliminated in some embodiments of the point-to-point mesh architecture 172, which exchanges messages according to one or more custom protocols rather than a specific established protocol required by switches, such as TCP / IP. d) Messages can be broadcast to directly connected nodes completely simultaneously and received completely simultaneously by directly connected receiver nodes, as will be described later. Different messages can also be received completely simultaneously from multiple directly connected sender nodes, as will be described later.

[0167] In embodiments where the direct, dedicated connections 180 between nodes are implemented via dedicated communication logic and dedicated interfaces, such as the GigE MAC cores 260 and connectors 270, as described above in relation to Figure 2, latency can be improved. In some such embodiments, each of the GigE MAC cores 260 handles message communication between its node and a single other node, and each of the connectors 270 is connected to a single other node in the point-to-point mesh architecture 172 via a dedicated connection 180. For example, considering three nodes in the point-to-point mesh of the embodiment in Figure 1D (gateway 120-1, core compute node 140-1, and core compute node 140-2), gateway 120-1 is connected to core compute node 140-1 via a dedicated connection 180-1-1, and gateway 120-1 is also connected to core compute node 140-2 via a separate dedicated connection 180-1-2. If each of these three nodes is implemented according to the embodiment in Figure 2, the GigE MAC core 260-1 and connector 270-1 for gateway 120-1 may be used independently for communication with core compute node 140-1 via dedicated connection 180-1-1, and the GigE MAC core 260-1 and connector 270-1 for core compute node 140-1 may be used independently for communication with gateway 120-1. Similarly, the GigE MAC core 260-2 and connector 270-2 for gateway 120-1 may be used independently for communication with core compute node 140-2 via dedicated connection 180-1-2, and the GigE MAC core 260-1 and connector 270-1 for core compute node 140-2 may be used independently for communication with gateway 120-1. Therefore, according to this embodiment, dedicated compute resources, such as one of the GigE MAC cores 260 and one of the connectors 270, are used for each node to communicate with each other node having a dedicated connection 180.

[0168] These dedicated compute resources per dedicated connection 180 enable a transmitting node to broadcast messages perfectly simultaneously to other mesh nodes directly connected to that transmitting node (e.g., via the dedicated connection 180), especially when the GigE core 260 is implemented in hardware such as a fixed logic device 230. Conversely, broadcast messages from a transmitting node can be received perfectly simultaneously by all receiving nodes. Each receiving node has its own dedicated connection 180 to the transmitting node, and for each node directly connected to a receiving node, reception is performed by dedicated connection communication logic and a dedicated interface. Since each receiving node receives broadcast messages perfectly simultaneously and the internal message latency between different nodes in the mesh should be identical, no preference is made between receiving nodes. Furthermore, each receiving node can receive different messages perfectly simultaneously from different nodes directly connected to it.

[0169] The ability to send and receive multiple messages completely simultaneously is in contrast to other environments where communication between servers takes place via a switch, which typically requires message serialization. When communication takes place via a switch, each server typically has a single connection (or multiple redundant connections, but still considered a single logical connection) between itself and the switch. If a server needs to broadcast a message to multiple other servers in the system, those messages are not actually sent to or received by the switch simultaneously, but instead need to be sent / received sequentially in series over a single logical connection between the server and the switch. Similarly, even if a switch can simultaneously receive multiple different messages destined for the same server but originating from different sending servers (because the switch may have a single connection to each of the originating servers), these multiple different messages still need to be serialized by the switch in order to send them one by one over a single logical connection between the switch and the destination server. The message serialization required in such environments not only increases the overall system latency but also results in unpredictable latency between servers within such a system. For example, if a message needs to be broadcast from one server to 15 other servers in the system, these 15 messages need to be serialized as they are sent to the switch, and the latency difference can be significant depending on whether the message addressed to a particular destination server is the first or 15th message in a series of broadcast messages.

[0170] In addition to the latency benefits that can be obtained from using dedicated connections 180 that do not require switch traversal, the point-to-point mesh architecture 172 enables further latency improvements. As described above in relation to Figure 1B-1, messages being sent between two nodes in the point-to-point mesh, such as from gateway 120-1 to core compute node 140-1, may be sent via a ranking path 117 to ensure that the messages are processed throughout the electronic trading system 100 in a deterministic order relative to other messages. As the messages traverse through the ranking path 117, they pass through sequencer 150-1, which marks the messages with a unique order identifier and then generates a sequenced message that sequencer 150-1 sends to the destination node (core compute node 140-1 in this example). Upon receiving an ordered message, the destination node (in this example, core compute node 140-1) can ensure that the message is processed in the correct deterministic order, as specified by the order identifier of the ordered message relative to other messages in the electronic trading system 100 (which are also ordered by the PLC).

[0171] As described above in relation to Figure 1B-1, this ranking path 117 may include multiple direct connections: direct connections between the sending node and the sequencer (e.g., direct connection 180-gw1-s1 between gateway 120-1 and sequencer 150-1) and further direct connections between the sequencer and the destination node (e.g., direct connection 180-c1-s1 between sequencer 150-1 and compute node 140-1). These multiple direct connections comprising the ranking path 117 may, in some embodiments, bypass switches and benefit from the latency advantages already described above. Nevertheless, in such embodiments, the latency of the ranking path 117 is affected by the message processing time in sequencer 150-1 (i.e., the time required for sequencer 150-1 to receive an unordered (i.e., unordered) message, mark it with an order identifier, and send the ordered message to the destination) and the transmission time of the message through the two direct connections. (That is, the unordered marked version is first sent from the sender node to the sequencer via a direct connection, and then the ordered marked version is sent from the sequencer to the destination node via a separate direct connection.) The tests show that the latency impact of this ranked path 117 compared to a single direct connection is an additional 0.5 to 1.0 microseconds for the ranked path 117, depending on factors such as whether a custom protocol is used rather than a specific established protocol such as TCP / IP.

[0172] Therefore, as an optimization to minimize the impact of potential latency across the ranking path 117, unordered marked messages may also be sent in parallel via activation link 180-1-1, which may be a single direct connection between the sender node and the destination node, such as direct connection 180-1-1 between gateway 120-1 and core compute node 140-1. Thus, in some embodiments, unordered marked messages may be sent completely simultaneously by the sending node (e.g., gateway 120-1) to the destination node (e.g., core compute node 140-1) via activation link 180-1-1 and to the sequencer 150-1 via direct connection (e.g., direct connection 180-gw1-s1) between the sending node and the sequencer 150-1 as part of the ranking path 117. In this embodiment, the sequencer 150-1 and the destination node (e.g., the core compute node 140-1) can simultaneously receive an unordered marked version of the message, thereby enabling the destination node to immediately activate processing of the unordered marked message upon receipt. While the ordered marked message is being generated by the sequencer 150-1 and sent to the destination node, the destination node may concurrently begin processing the unordered marked message, which may include reading data related to the ticker symbols referenced in the message into high-speed memory, constructing a preliminary response message, or initiating matching function activities for electronic trading. As described above, upon receiving the ordered marked message, the destination node may complete processing of the message according to the appropriate deterministic order (i.e., sequence) as specified by the order identifier.However, by the time the destination node receives the ordered message via the ranking path 117, it will have already had the opportunity to perform useful processing on the unordered version of the message (e.g., a processing time of 0.5 to 1.0 microseconds), allowing it to complete processing of the ordered message more quickly than if it had not yet received the unordered version of the message via the activation link 180-1-1, and to generate a response message with lower latency.

[0173] Therefore, as described above, point-to-point mesh architectures (e.g., 102 and 172) offer latency improvements in numerous ways. Through testing, at least some of these latency improvements have been empirically quantified. For example, the latency improvement resulting from the use of direct connections between nodes to bypass switches is at least 1.0 microsecond in total, and in some cases up to 3.0 microseconds in each direction (i.e., incoming versus outgoing). Furthermore, the latency improvement resulting from the opportunity for destination nodes to process unordered marked messages received via activation link 180-1-1, in parallel with the processing and transmission of messages via the ranking path 117, is an additional 0.5 to 1.0 microseconds in each direction. Thus, the overall possible latency improvement from the time an incoming message enters the electronic trading system 100 from participant device 130 through gateway 120 to the time a corresponding response message is sent from gateway 120 to participant device 130 is at least 2.5 to 7.0 microseconds. In latency-sensitive applications such as electronic trading, even a difference of a microsecond in latency can be critical. These latency improvements enable some embodiments of the electronic trading system 100 to reliably respond within a desired response time latency of 5.0 to 7.0 microseconds. This represents a significant improvement over conventional trading systems, the best of which currently have response time latencies in the range of 50 to 75 microseconds.

[0174] Figure 3 is a block diagram of another exemplary embodiment of the point-to-point mesh system 302. The point-to-point mesh system 302 includes a plurality of gateways 320, a plurality of core compute nodes 340, and a sequencer 350. Each gateway 320-1, 320-2, ..., 320-g of the plurality of gateways 320 is connected to each core compute node 340-1, 340-2, ..., 340-c of the plurality of core compute nodes 340 via its respective first direct connection, i.e., first direct connection 380-a. The sequencer 350 is connected to each gateway of the plurality of gateways 320 via its respective second direct connection, i.e., second direct connection 380-b, and to each core compute node of the plurality of core compute nodes via its respective third direct connection, i.e., third direct connection 380-c. Referring to Figures 1B-1, 1B-2, 1D and Figure 3, gateway 120-1 may be a given gateway among a plurality of gateways 320 that are interconnected in a communicative manner via a shared gateway network as disclosed above with respect to Figure 1D or via direct connections 384a, 384b, ..., 384g in Figure 3. The plurality of gateways 320 include gateways 320-1, 320-2, ..., 320-g in exemplary embodiments. Core compute node 140-1 may be a given core compute node among a plurality of core compute nodes 340, including core compute nodes 340-1, 340-2, ..., 340-c, which can be interconnected in a communicative manner via a shared core compute node network as disclosed above with respect to Figure 1D or via direct connections 386a, 386b, ..., 386c in Figure 3. It should be understood that the gateway number g may be the same as or different from the other number c of the core compute nodes in the point-to-point mesh system 302.According to an exemplary embodiment, the sequencer 150-1 may be a given sequencer among a plurality of sequencers that are interconnected in a manner that allows them to communicate with one another via a shared sequencer network, such as the sequencer-wide shared network 182-s disclosed above with respect to Figure 1D, or via direct connections, such as the fourth direct connection 482 disclosed below with respect to Figure 4.

[0175] Figure 4 is a block diagram of another exemplary embodiment of the point-to-point mesh system 402. Referring to Figures 1B-2 and 4, the sequencer 150-1 may be a given sequencer among a plurality of sequencers in the point-to-point mesh system 402, including sequencer 450-1 and at least one other sequencer 450-2.

[0176] In the point-to-point mesh system 402, each gateway 420-1, 420-2, ..., 420-g of the multiple gateways 420 is connected to each core compute node 440-1, 440-2, ..., 440-c of the multiple core compute nodes via their respective first direct connections, i.e., first direct connections 480a. Each gateway of the multiple gateways 420 is connected to each sequencer 450-1, ..., 450-2 of the multiple sequencers via their respective second direct connections, i.e., second direct connections 480-b-1 and 480-b-2. Each of the multiple core compute nodes 440-1, 440-2, ..., 440-c is connected to each of the multiple sequencers 450-1, ..., 450-2 via their respective third direct connections, namely third direct connections 480-c-1 and 480-c-2.

[0177] A given sequencer, i.e., a specific sequencer of sequencers 450-1...450-2, may be the currently active sequencer servicing the point-to-point mesh system, while each other sequencer in the plurality of sequencers may be a standby sequencer waiting to take over from the currently active sequencer. Each sequencer in the plurality of sequencers may be coupled to each other sequencer in the plurality of sequencers via their respective fourth direct connections, such as a fourth direct connection 482. Each gateway 420-1, 420-2, ..., 420-g of the plurality of gateways 420 may further be configured to send messages (not shown) destined for their respective compute nodes to a given sequencer among the plurality of sequencers 450-1, ..., 450-2. Each of the multiple core compute nodes 440-1, 440-2, ..., 440-c is further configured to send messages (not shown) destined for their respective gateways to a given sequencer among the multiple sequencers 450-1, ..., 450-2. At least one other sequencer is in standby mode.

[0178] The standby state (standby role) may represent a passive state in which at least one other sequencer is powered on and configured to be in "listening-only" mode. In "listening-only" mode, at least one sequencer is configured to receive messages but not send them, and is capable of taking over the active sequencer if the currently active sequencer fails. The currently active sequencer is in the active state (active role), capable of sending and receiving messages. Generally, a standby sequencer is a redundant (backup) sequencer that is powered on, capable of receiving messages, and capable of taking over the active sequencer. Such a switchover, i.e., switching from the standby state (role) to the active state (role), may be based on the reception of a switching command issued by the controller (not shown) due to a failure of the currently active sequencer or, in a non-limiting example, for other reasons such as a software update applied to the currently active sequencer.

[0179] A given sequencer (in the active state) may further be configured to send sequentially marked messages, such as sequentially marked messages 106' and sequentially marked responses 107', to each of the other sequencers 450-1, ..., 450-2 via each of the fourth direct connections 482, if the currently active sequencer fails or, in a non-limited example, is no longer designated as the active sequencer for other reasons, such as a software update, so that the standby sequencers can take over from the currently active sequencer. However, it should be understood that the active sequencer, i.e., a given sequencer in the active state, is not required to forward sequentially marked messages to the standby sequencer, i.e., a sequencer in the standby state, but instead may continuously broadcast / replicate a journal, also known as a status log (which will contain sequence information), to the standby sequencer as disclosed above with respect to Figure 1D.

[0180] The electronic trading system 100 may further include a system status log (not shown) as described above with reference to Figure 1D. An active sequencer may be configured to transmit the system status log from the active sequencer to at least one other sequencer of a plurality of sequencers via a shared sequencer network. For example, the shared sequencer network may include a fourth direct connection 482, and the system status log may be transmitted from sequencer 450-1 to sequencer 450-2 when sequencer 450-1 is a given sequencer in an active state.

[0181] Figure 5 is a flowchart 500 of an exemplary embodiment of a method for executing electronic trading. The method starts (502) and sends a message representing an electronic trading request with a limit price to buy or sell a financial instrument from the gateway to the core compute node via a first direct connection (504). The message is received by the core compute node to execute electronic trading functions in the electronic trading system. The method sends the message from the gateway to the sequencer of the electronic trading system via a second direct connection (506), and in response, the sequencer sends a sequenced message from the sequencer to the core compute node via a third direct connection (508). The first, second, and third direct connections each have their own non-shared bandwidth. The sequencer is inserted between the gateway and the core compute node via the second and third direct connections. The sequenced message sent by the sequencer is an ordered version of the message sent from the gateway via the second direct connection. The sequenced message is received by the core compute node. The method involves the core compute node receiving other messages from the gateway, receiving ordered versions of other messages from the sequencer, and determining the relative ranking of the ordered messages among the ordered versions of other messages received by the core compute node in the electronic trading system (510). The core compute node completes the electronic trading matching function for electronic trading requests according to the determined relative ranking. This completion matches limit orders with stop orders for financial instruments, enabling electronic trading of said financial instruments. The method then terminates in an exemplary embodiment (512).

[0182] The message may be a gateway message. The method may further comprise the step of receiving an incoming message from a participant device at the gateway. The step of sending a gateway message may include the step of the gateway sending a gateway message in response to the gateway receiving an incoming message from a participant device. The ordered message may be a first ordered message. The method may further comprise the step of sending a first ordered message from a sequencer to the gateway via a second direct connection. In response, the first ordered message may be received by the gateway. In response to the reception of the message, the method may further comprise the steps of sending a core compute node message from the core compute node to the gateway via a first direct connection, and sending the core compute node message to the sequencer via a third direct connection, in response to the second ordered message from the sequencer to the gateway via a second direct connection. The second ordered message is an ordered version of the core compute node message. The method may further comprise the steps of: determining the relative ranking of a second ordered message and ordered versions of other messages sent from the core compute node to the gateway at the gateway; sending an outgoing message to a participant device, wherein the outgoing message is sent according to the determined relative ranking; and sending the second ordered message from the sequencer to the core compute node via a third direct connection.

[0183] A gateway can be a given gateway among multiple gateways, and a core compute node can be a given core compute node among multiple core compute nodes. The method may further comprise the step of sending a message destined for a specific compute node from each gateway of the multiple gateways to all core compute nodes and sequencers of the multiple core compute nodes. The method may further comprise the step of sending a message destined for a specific gateway from each core compute node of the multiple core compute nodes to all gateways and sequencers of the multiple gateways, and the step of sending a sequenced message from the sequencer to the multiple gateways and multiple core compute nodes in accordance with the compute node-bound messages and gateway-bound messages received by the sequencer.

[0184] The sequencer can be a given sequencer among multiple sequencers. The method may further comprise the steps of sending messages destined for each compute node from each gateway of multiple gateways to the given sequencer, sending messages destined for each gateway from each core compute node of multiple core compute nodes to the given sequencer, and sending a sequence-marked message from the given sequencer to each of the other sequencers among multiple sequencers.

[0185] The method may further comprise the steps of: receiving the same message at multiple core compute nodes, the same message being a message sent by a given gateway to each of the compute nodes; generating response messages at the multiple core compute nodes in response to the receipt of the same message; and receiving the response messages at a given gateway from among the multiple core compute nodes. The method may further comprise the step of performing an action at a given gateway based on a given response message among the response messages generated in response to the receipt of the same message. The given response message may be the first to arrive at the given gateway relative to other response messages generated in response to the receipt of the same message. The method may further comprise the step of ignoring other response messages that arrive at the given gateway after the given response message. Thus, “the same message” means a single message broadcast from a single gateway to all compute nodes, thereby each compute node obtaining one copy of that single message. Then, each of the multiple compute nodes may respond to the same single message, and multiple functionally equivalent responses may be broadcast to all gateways, such as being received by a given gateway. The gateways, including the given gateway, can then sort these multiple functionally equivalent messages in "first-come, first-served" order.

[0186] The method may comprise the step of receiving multiple compute node-bound messages from at least two of a plurality of gateways at a given compute node. Each compute node-bound message in the plurality of compute node-bound messages may represent a different version of the same message, each version being input to each of the at least two gateways via its respective high-availability flow. The method may further comprise the step of performing an action at a given compute node based on a given compute node-bound message from the plurality of compute node-bound messages. A given compute node-bound message may be the first to arrive at the given compute node relative to other compute node-bound messages from the plurality of compute node-bound messages. The method may further comprise the step of ignoring other compute node-bound messages that arrive at the given compute node after the given compute node-bound message.

[0187] The method may further include a step of matching trading orders for financial instruments at the core compute node based on the electronic trading matching function being performed. The method may further include a step of maintaining the balance of financial instruments in the order ledger, where the balance is the monetary mismatch of financial instruments resulting from the electronic trading matching function being performed.

[0188] The method may further include the step of synchronizing the gateway, core compute node, and sequencer based on a clock.

[0189] The message may be a gateway message. The method may further comprise the steps of: providing a service to at least one participant device at the gateway; receiving an incoming message at the gateway; and transmitting a gateway message from the gateway to the sequencer and core compute nodes in response to the reception of the incoming message at the gateway. The incoming message may have been sent by at least one participant device. The method may further comprise the step of generating an ordered message by marking the message or its representation with a unique ordered identifier.

[0190] The method may further include the step of protecting a first direct connection, a second direct connection, and a third direct connection, or a set of some thereof, through at least one redundant direct connection of each.

[0191] A gateway may be a given gateway among a plurality of gateways, a core compute node may be a given core compute node among a plurality of core compute nodes, and a sequencer may be a given sequencer among a plurality of sequencers. The method may further comprise the steps of enabling a plurality of gateways to communicate via a shared gateway network, enabling a plurality of core compute nodes to communicate via a shared core compute node network, and enabling a plurality of sequencers to communicate via a shared sequencer network or via their respective fourth direct connections.

[0192] The method may further comprise the steps of sending a system state log from a given sequencer to at least one other sequencer among a group of sequencers via a shared sequencer network, or storing the system state log in a datastore by a given sequencer. The datastore may be accessible to the group of sequencers via the shared sequencer network.

[0193] The electronic trading system may be an active electronic trading system. The method may include the step of enabling at least one of a plurality of sequencers to communicate with a disaster recovery site. The disaster recovery site may include a standby electronic trading system. The standby electronic trading system is a replica of the active electronic trading system and may be configured to allow electronic trading to continue in the event of a failure of the active electronic trading system.

[0194] Figure 6 is a block diagram of an exemplary embodiment of a sequencer 650 for an electronic trading system, such as the electronic trading system 100 disclosed above. In the exemplary embodiment, the sequencer 650 includes a first communication module 652 configured to communicate directly with a gateway 620 via a first direct connection 680a in a point-to-point mesh system, such as the point-to-point mesh system 102 of the electronic trading system 100 disclosed above. The sequencer 650 further includes a second communication module 654 configured to communicate directly with a core compute node 640 via a second direct connection 680b in a point-to-point mesh system in the electronic trading system. The sequencer 650 further includes a sequencing logic 656 coupled with the first communication module 652 and the second communication module 654.

[0195] The ordering logic 656 is configured to generate an ordered message 606' by marking message 606 or its representation with a unique order identifier (not shown), and message 606 is received by the first communication module 652 or the second communication module 654 via the first direct connection 680a or the second direct connection 680b, respectively. The ordering logic 656 is further configured to transmit the ordered message 606' to the gateway 620 and the core compute node 640 via the first communication module 652 and the second communication module 654, respectively.

[0196] According to the exemplary embodiment, the ordering logic is, for example, the logic portion of the FPGA or ASIC in the fixed logic device 230 of Figure 2 disclosed above. Furthermore, both communication modules 652 and 654 may also be implemented in the FPGA, for example, in the 10 GigE MAC core 260 disclosed above with reference to Figure 2.

[0197] Figure 7 is a block diagram of another exemplary embodiment of an electronic trading system, such as the electronic trading system 100 disclosed above in Figures 1B-1 and 1B-2. Referring to Figures 1B-2 and 7, the gateway 120-1, core compute node 140-1, sequencer 150-1, first direct connection 180-1-1, second direct connection 180-gw1-s1, and third direct connection 180-c1-s1 constitute the first point-to-point mesh system 702a. The electronic trading system 100 may be a first electronic trading system 700a, communicatively coupled to a proxy node 772. The proxy node 772 may further be communicatively coupled to at least one participant device 730 and a second electronic trading system 700b. The second electronic trading system 700b may include a second point-to-point mesh system 702b, such as the point-to-point mesh system 102 disclosed above with respect to Figure 1B-2.

[0198] The proxy node 772 is configured to transmit message 703' to the first electronic trading system 700a and the second electronic trading system 700b upon receiving an incoming message 703 from at least one participant device 730. The first and second electronic trading systems may be configured to generate their respective responses to the message transmitted by the proxy node 772. The proxy node 772 may further be configured to transmit a response, i.e., an outgoing message 705, to at least one participant device 730 upon receiving a first arrival response 705' received from the first electronic trading system 700a or the second electronic trading system 700b. The first electronic trading system 700a and the second electronic trading system 700b may be synchronized, for example, based on a common clock that provides a date and time (777). However, the first electronic trading system 700a and the second electronic trading system 700b are not limited to being synchronized based on a common clock (777).

[0199] According to an exemplary embodiment, a single active / primary sequencer (not shown) of the entire system encompassing both electronic trading systems 700a and 700b may be employed to coordinate the sequence numbers between electronic trading systems 700a and 700b. For example, the single active / primary sequencer may coordinate the sequence numbers so that arrival responses 705' are assigned the same sequence number by both electronic trading systems 700a and 700b, thereby enabling the proxy node 772 to appropriately determine the relative ranking of messages. For example, if the single active / primary sequencer is in electronic trading system 700a, the active sequencer may be configured to communicate with electronic trading system 700b by communicating with a standby sequencer in electronic trading system 700b, and sequence numbers are assigned per message.

[0200] The above architectures, such as point-to-point mesh architectures, can be used for applications other than electronic trading systems. For example, they can be used for applications other than handling securities trading orders, such as monitoring data streams flowing through a network, capturing packets, decoding the raw data of packets, analyzing the contents of packets in real time, and providing responses.

[0201] Furthermore, the exemplary embodiments disclosed herein may be configured using computer program products, for example, the control may be programmed into software for carrying out the exemplary embodiments. Further exemplary embodiments may include non-temporary computer-readable media containing instructions that can be executed by a processor, which, once read and executed, can cause the processor to complete the methods described herein. It should be understood that the elements of the block diagrams and flow diagrams may be implemented in software or hardware by one or more arrangements of the circuit of Figure 2 disclosed herein, or equivalents thereof, firmware, custom-designed semiconductor logic, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), combinations thereof, or other similar embodiments to be determined in the future.

[0202] Furthermore, the elements of the block diagrams and flow diagrams described herein may be combined or divided in any way in software, hardware, or firmware. If implemented in software, the software may be written in any language that can correspond to the exemplary embodiments disclosed herein. The software may be stored in any form of computer-readable medium, such as one or more random access memories (RAM), read-only memories (ROM), or compact disk read-only memory (CD-ROM). In operation, a general-purpose or application-specific processor or processing core reads and executes the software in a manner well understood in the art. Furthermore, it should be understood that the block diagrams and flow diagrams may contain more or fewer elements, be arranged or oriented in different ways, or be represented in different ways. It should be understood that the number of block diagrams, flow diagrams, and / or network diagrams, as well as the number of block diagrams and flow diagrams, may be used to illustrate the execution of the embodiments disclosed herein.

[0203] Therefore, further embodiments may be implemented in various computer architectures, physical, virtual, cloud computers and / or any combination thereof, and thus the data processing systems described herein are for illustrative purposes only and are not intended to limit the embodiments.

[0204] While exemplary embodiments have been specifically illustrated and described, it will be understood by those skilled in the art that various modifications in form and detail can be made without departing from the scope of embodiments covered by the appended claims.

Claims

1. It is an electronic trading system, A gateway connected to a core compute node via an activation link and a ranking path, The gateway is configured to send messages to the core compute node via the activation link and the ranking path, wherein the activation link includes a single direct connection and the ranking path includes multiple direct connections. A sequencer electronically placed within the ranking path and configured to generate an ordered version of the message, The aforementioned version with sequence markings includes a sequence identifier, The core compute node receives the message and the sequenced version from the gateway and the sequencer, respectively. The aforementioned core compute node is (i) In response to receiving the message via the activation link, initiate a matching function activity for electronic transactions. (ii) An electronic trading system configured to prioritize the completion of the matching function activity toward servicing the electronic transaction, using the sequence identifier, in response to the receipt of the sequence-marked version via the ranking path.

2. The electronic transaction system according to claim 1, wherein the sequence identifier indicates the definitive position of a message among a plurality of messages communicated via the activation link and received by the sequencer via the ranking path.

3. The aforementioned message and the aforementioned marked-order version contain common metadata, The electronic transaction system according to claim 1, wherein the core compute node correlates the message to the ordered version based on common metadata in response to the receipt of the ordered version via the ranking path.

4. The electronic transaction system according to claim 1, wherein the message and the sequentially marked version include the same user data, and the user data is associated with an electronic transaction request.

5. The activation link is a first direct connection, The aforementioned ranking path includes a second direct connection and a third direct connection, The gateway is further configured to transmit the message to the sequencer via the second direct connection. The sequencer is configured to send the sequenced version of the message to the core compute node via the third direct connection. The message is a gateway message sent by the gateway in response to the reception of an incoming message received by the gateway from a participant device, the sequencer is further configured to send the sequenced version thereto to the gateway via the second direct connection, the sequenced version thereto is received by the gateway, the sequenced version is the first sequenced message, and the core compute node further, In response to receiving the gateway message, the system is configured to send a core compute node message to the gateway via the first direct connection. The core compute node message is configured to be sent to the sequencer via the third direct connection, the sequencer is further configured to be sent to the gateway via the second direct connection, the second sequentially marked message being an ordered version of the core compute node message, and the gateway is further configured It is configured to determine the relative ranking of the second ordered message and the ordered versions of other messages transmitted from the core compute node to the gateway, The electronic trading system according to claim 1, wherein the sequencer is configured to transmit outgoing messages to the participant devices, the outgoing messages being transmitted according to the determined relative ranking, and the sequencer is further configured to transmit the second ordered message thereto to the core compute node via the third direct connection.

6. The gateway is a given gateway among a plurality of gateways, and the core compute node is a given core compute node among a plurality of core compute nodes. The activation link is a first direct connection, and the ranking path includes a second direct connection and a third direct connection. Each of the plurality of gateways is connected to each of the plurality of core compute nodes via its respective first direct connection. The sequencer is connected to each of the plurality of gateways via its respective second direct connection, and to each of the plurality of core compute nodes via its respective third direct connection, and the plurality of gateways, the plurality of core compute nodes, the sequencer and its respective direct connection constitute at least a part of the point-to-point mesh system, and within the point-to-point mesh system, Each of the plurality of gateways is configured to send messages destined for each compute node transmitted from there to all compute nodes of the plurality of core compute nodes and the sequencer. Each of the plurality of core compute nodes is configured to transmit messages destined for each gateway sent from there to all of the gateways and sequencers of the plurality of gateways. The electronic trading system according to claim 1, wherein the sequencer is further configured to transmit the respective sequenced versions to the plurality of gateways and the plurality of core compute nodes in response to the receipt of a message addressed to each of the compute nodes or a message addressed to each of the gateways.

7. The sequencer is a given sequencer among a plurality of sequencers in the point-to-point mesh system, Each of the plurality of gateways is connected to each of the plurality of sequencers via the respective second direct connection. Each of the plurality of core compute nodes is connected to each of the plurality of sequencers via the respective third direct connection. The given sequencer is the currently active sequencer providing service to the point-to-point mesh system, and each of the other sequencers in the plurality of sequencers is a standby sequencer waiting to take over from the currently active sequencer. Each of the plurality of sequencers is coupled to each of the other sequencers of the plurality of sequencers via its respective fourth direct connection. Each of the plurality of gateways is further configured to send messages destined for their respective compute nodes to the given sequencer among the plurality of sequencers. Each of the plurality of core compute nodes is further configured to send messages destined for its respective gateway to the given sequencer among the plurality of sequencers. The electronic trading system according to claim 6, wherein the given sequencer is further configured to transmit the sequence-marked version to each of the other sequencers of the plurality of sequencers via each of the fourth direct connections, so that the standby sequencer can take over from the currently active sequencer if the currently active sequencer fails.

8. The messages sent by the given gateway to each of the compute nodes are the same messages received by the plurality of core compute nodes, the plurality of core compute nodes are configured to generate a response message in response to receiving the same message, the response message is received by the given gateway from among the plurality of core compute nodes, and the given gateway further, The system is configured to take action based on a given response message among the response messages generated in response to the reception of the same message, wherein the given response message is the first to arrive at the given gateway relative to other response messages generated in response to the reception of the same message, and the given gateway further, The electronic transaction system according to claim 6, configured to ignore any other response messages that arrive after the given response message.

9. Multiple messages destined for compute nodes, representing the same message, are received by a given compute node from among the multiple gateways, and the given compute node further... The system is configured to take action based on a given compute node-bound message among the multiple compute node-bound messages, wherein the given compute node-bound message is the first to arrive at the given compute node relative to other compute node-bound messages among the multiple compute node-bound messages representing the same message, and the given compute node further, The electronic transaction system according to claim 6, configured to ignore other messages addressed to compute nodes that arrive after the given message addressed to compute node.

10. The core compute node further comprises an order book accessible by the core compute node, and the core compute node further comprises It is configured to match trading orders for financial products using an electronic trading matching function. The electronic trading system according to claim 1, configured to maintain the outstanding balance of the financial instrument in the order ledger, wherein the mismatch amount of trading orders related to the financial instrument is obtained as a result of performing the electronic trading matching function, and the outstanding balance includes the mismatch amount of trading orders related to the financial instrument.

11. The electronic trading system according to claim 1, further comprising a clock, wherein the gateway, the core compute node and the sequencer are synchronized based on the clock.

12. The aforementioned gateway further, Configured to provide services to at least one participant device, The electronic trading system according to claim 1, wherein the gateway is configured to transmit an incoming message to the sequencer and the core compute node in response to the reception of the incoming message, the incoming message is sent by the at least one participant device, and the sequencer is further configured to generate a sequence-marked version by marking the message with a unique sequence identifier, or by creating a display of the received message, marking the display with the unique sequence identifier, and transmitting the marked display, the marked display being the sequence-marked version.

13. The activation link is a first direct connection, The aforementioned ranking path includes a second direct connection and a third direct connection, The electronic trading system according to claim 1, further comprising at least one redundant direct connection for each of the first direct connection, the second direct connection, and the third direct connection or any set of these.

14. The gateway is a given gateway among a plurality of gateways that are interconnected and can communicate with each other via a shared gateway network. The aforementioned core compute node is a given core compute node among a plurality of core compute nodes that are interconnected and able to communicate with each other via a shared core compute node network. The electronic trading system according to claim 1, wherein the sequencer is a given sequencer among a plurality of sequencers that are interconnected so as to be able to communicate with each other via a shared sequencer network or via their respective fourth direct connections.

15. The electronic trading system according to claim 14, further comprising a system status log, wherein the given sequencer is configured to transmit the system status log to at least one other sequencer among the plurality of sequencers via the shared sequencer network, or to store the system status log in a data store, the data store being accessible to the plurality of sequencers via the shared sequencer network.

16. The electronic trading system according to claim 14, wherein the electronic trading system is an active electronic trading system, at least one of the plurality of sequencers is communicably coupled to a disaster recovery site, the disaster recovery site includes a standby electronic trading system, the standby electronic trading system is a copy of the active electronic trading system and is configured to enable electronic trading to continue in the event of a failure of the active electronic trading system.

17. The activation link is a first direct connection, The aforementioned ranking path includes a second direct connection and a third direct connection, The gateway, the core compute node, the sequencer, and the first, second, and third direct connections constitute a first point-to-point mesh system. The electronic trading system is a first electronic trading system communicatively coupled to a proxy node, the proxy node further communicatively coupled to at least one participant device and a second electronic trading system, the second electronic trading system including a second point-to-point mesh system. The electronic trading system according to claim 1, wherein the proxy node is configured to transmit an incoming message from the at least one participant device to the first and second electronic trading systems upon receipt of the message, the first and second electronic trading systems are configured to generate their respective responses to the message transmitted by the proxy node, and the proxy node is further configured to transmit a response to the at least one participant device upon receipt of the first or second response which arrives among the respective responses which have been generated and received from the first or second electronic trading system.

18. A method for conducting electronic transactions, The steps include sending a message from the gateway to the core compute node via an activation link and a ranking path, with the sequencer electronically positioned within the ranking path, the activation link comprising a single direct connection, and the ranking path comprising multiple direct connections, The core compute node receives the message and a sequenced version of the message from the gateway and the sequencer, respectively, and the sequenced version includes a sequence identifier. In the aforementioned core compute node, (i) In response to receiving the message via the activation link, initiate a matching function activity for electronic transactions. (ii) A method comprising the step of, in response to receiving the ordered marked version via the ranking path, using the order identifier, prioritizing the completion of the matching function activity toward servicing the electronic transaction.

19. The method according to claim 18, wherein the sequence identifier indicates the definitive position of the message among a plurality of messages communicated via the activation link and received by the sequencer via the ranking path.

20. The aforementioned message and the aforementioned marked-order version contain common metadata, The method according to claim 18, wherein the core compute node correlates the message to the ordered version based on common metadata in response to the receipt of the ordered version via the ranking path.

21. The method of claim 18, wherein the message and the marked-sequence version include the same user data, and the user data is associated with an electronic transaction request.

22. The message is a gateway message, the method further comprises the step of receiving an incoming message from a participant device at the gateway, the step of transmitting the gateway message includes the step of the gateway transmitting the gateway message in response to the gateway receiving the incoming message from the participant device, the ordered version is a first ordered version, the activation link is a first direct connection, the ranking path includes a second direct connection and a third direct connection, and the method is A step of transmitting the first sequenced message from the sequencer to the gateway via the second direct connection, wherein the first sequenced message is received by the gateway. In response to receiving the aforementioned message, the core compute node message is sent from the core compute node to the gateway via the first direct connection. A step of sending the core compute node message to the sequencer via the third direct connection, and in response, sending a second sequenced message from the sequencer to the gateway via the second direct connection, wherein the second sequenced message is an ordered version of the core compute node message. The steps include determining the relative ranking of the second ordered message and the ordered versions of other messages transmitted from the core compute node to the gateway at the gateway, A step of sending an outgoing message to the participant device, wherein the outgoing message is sent according to the determined relative ranking, The steps include sending the second sequence-marked message from the sequencer to the core compute node via the third direct connection, The method according to claim 18, further comprising:

23. The gateway is a given gateway among a plurality of gateways, the core compute node is a given core compute node among a plurality of core compute nodes, and the method is The steps include sending each of the compute node-bound messages transmitted from each of the plurality of gateways to all of the core compute nodes and the sequencer of the plurality of core compute nodes, The steps include sending each gateway-bound message sent from each of the multiple core compute nodes to all of the gateways and the sequencer of the multiple gateways, The steps include: sending the respective sequenced versions from the sequencer to the multiple gateways and the multiple core compute nodes in response to the messages addressed to each compute node and each gateway received by the sequencer; The method according to claim 18, further comprising:

24. The sequencer is a given sequencer among a plurality of sequencers, and the method is The steps include sending messages destined for each compute node, transmitted from each gateway of the plurality of gateways, to the given sequencer, The steps include sending the respective gateway-bound messages sent from each of the plurality of core compute nodes to the given sequencer, The steps include transmitting the sequenced version from the given sequencer to each of the other sequencers among the plurality of sequencers, The method according to claim 23, further comprising:

25. A step of receiving the same message in the plurality of core compute nodes, wherein the same message is a message sent by the given gateway to each of the compute nodes, The steps include generating a response message in the multiple core compute nodes in response to receiving the same message, The steps include: receiving the response message from among the plurality of core compute nodes in the given gateway; A step of performing an action at a given gateway based on a given response message among the response messages generated in response to the receipt of the same message, wherein the given response message is the first to arrive at the given gateway relative to other response messages among the response messages generated in response to the receipt of the same message. The steps include ignoring other response messages that arrive at the given gateway after the given response message, The method according to claim 23, further comprising:

26. A given compute node receives multiple messages destined for multiple compute nodes from among the multiple gateways, wherein the multiple messages destined for multiple compute nodes represent the same message. A step of performing an action at a given compute node based on a given compute node-bound message among the plurality of compute node-bound messages, wherein the given compute node-bound message is the first to arrive at the given compute node relative to other compute node-bound messages among the plurality of compute node-bound messages that represent the same message; The steps include: ignoring other messages destined for other compute nodes that arrive at the given compute node after the given message destined for the given compute node; The method according to claim 23, further comprising:

27. The steps include matching trading orders for financial instruments in the core compute node based on the electronic trading matching function that is executed, A step of maintaining the balance of the financial instrument in the order ledger, wherein the balance is the amount of mismatch in trading orders related to the financial instrument brought about by the executed electronic trading matching function, The method according to claim 18, further comprising:

28. The method according to claim 18, further comprising the step of synchronizing the gateway, the core compute node, and the sequencer based on a clock.

29. The aforementioned message is a gateway message, and the method is The steps include providing a service to at least one participant device at the gateway, The steps include receiving an incoming message at the gateway, Steps include: transmitting the gateway message from the gateway to the sequencer and the core compute node in response to the reception of the incoming message at the gateway, wherein the incoming message was sent by at least one participant device; The steps include generating the sequence-marked version by marking the message or its display using a unique sequence identifier, The method according to claim 18, further comprising:

30. The activation link is a first direct connection, The aforementioned ranking path includes a second direct connection and a third direct connection, The method according to claim 18, further comprising the step of protecting the first direct connection, the second direct connection, and the third direct connection or any set of them through at least one redundant direct connection of each.

31. The gateway is a given gateway among a plurality of gateways, the core compute node is a given core compute node among a plurality of core compute nodes, the sequencer is a given sequencer among a plurality of sequencers, and the method is The steps include enabling the multiple gateways to communicate with each other via a shared gateway network, The steps include enabling the plurality of core compute nodes to communicate via a shared core compute node network, The steps include enabling the plurality of sequencers to communicate via a shared sequencer network or via their respective fourth direct connections, The method according to claim 18, further comprising:

32. The method according to claim 31, further comprising the steps of transmitting a system state log from a given sequencer to at least one other sequencer among the plurality of sequencers via the shared sequencer network, or storing the system state log in a data store by the given sequencer, wherein the data store is accessible to the plurality of sequencers via the shared sequencer network.

33. The electronic trading system is an active electronic trading system, and the method is The method according to claim 31, further comprising the step of enabling at least one of the plurality of sequencers to communicate with a disaster recovery site, wherein the disaster recovery site includes a standby electronic trading system, the standby electronic trading system being a replica of the active electronic trading system and configured to enable electronic trading to continue in the event of a failure of the active electronic trading system.

34. Equipped with gateways, sequencers, and core compute nodes arranged in a point-to-point mesh topology, The core compute node is configured to perform a matching function toward serving transaction requests received from participant devices and introduced into the point-to-point mesh topology via the gateway, the point-to-point mesh topology includes a first direct connection, a second direct connection and a third direct connection, The aforementioned sequencer, (i) configured to determine a definitive order (i.e., sequence) of messages communicated between the gateway and the core compute node via the first direct connection and received by the sequencer from the gateway or the core compute node via the second or third direct connection, respectively, the sequencer further (ii) an electronic trading system configured to communicate the position of the message in a definitive order by transmitting an ordered version of the message representing a transaction request or a response thereof to the gateway and the core compute node via the second and third direct connections, respectively.

Images