Design method of distributed high-frequency transaction system based on RAFT consensus algorithm

Abstract

The application provides a distributed high-frequency transaction system based on a RAFT consensus algorithm, transaction data in the order life cycle is regarded as a state through a total transaction state machine, initial data at the beginning of the day is obtained from a DB as an initial state, customer requests corresponding to the transaction order of the customer and downstream orders and pushing are regarded as trigger condition events of state conversion, the state is changed according to the trigger condition events, if it is global data, the data is synchronized and committed by a raft instance and then written into a corresponding sub-transaction state machine, if it is user data, the data is sharded according to a user account hash value, the data is committed by a corresponding raft instance and then written into a corresponding sub-transaction state machine, tasks and user accounts are submitted to a core computing thread pool to execute all corresponding state conversions and business processing, a master core of the raft instance is uniformly distributed on physical nodes through a load balancing strategy, and efficient utilization is realized.

Description

Technical Field

[0001] This invention relates to a design method for a distributed high-frequency trading system based on the RAFT consensus algorithm. Background Technology

[0002] The current mainstream trading system is as follows: After verifying the funds and securities of the order, the high-frequency trading system reports the order to the exchange and pushes and maintains the trading data according to the order. The trading data is stored in the database, other data middleware, or memory. If stored in the database or other data middleware, although the service state is transferred to make it a stateless service and achieve the purpose of clustered deployment, it cannot meet the needs of high-frequency trading business and cannot support the low latency throughput requirements of high-frequency trading. If stored in memory, when combined with the native RAFT consensus algorithm, since the native RAFT consensus algorithm only supports one master and multiple slaves, and the slave nodes are only used for disaster recovery and do not provide external services, there is a single point of performance bottleneck (it does not support horizontal scaling, and the maximum throughput is limited by the performance of a single node). Therefore, the data synchronization of multiple nodes cannot guarantee real-time consistency, the hardware resources of slave nodes are idle, the failover process is long, and the availability cannot be guaranteed, which brings difficulties to disaster recovery.

[0003] Currently, the mainstream disaster recovery methods for trading systems are mainly warm backup and cold backup. Warm backup involves writing data from the master node to disk via a bus or other mechanism and synchronizing it to the warm backup node. When the master node fails, the master node is manually switched to the warm backup node using techniques such as VIP migration. Cold backup, on the other hand, requires an additional manual startup and data synchronization process. Warm backup switching requires manual intervention and takes about 1 to 3 minutes, while cold backup startup takes more than 5 minutes. Summary of the Invention

[0004] The technical problem that the present invention needs to solve is:

[0005] 1. The difficulty of disaster recovery; 2. When designing a high-frequency trading system using the RAFT consensus algorithm, the native RAFT consensus algorithm only supports one master and multiple slaves, and the slave nodes are only used for disaster recovery, resulting in low utilization of hardware resources.

[0006] To address the aforementioned technical problems, the present invention provides a design method for a distributed high-frequency trading system based on the RAFT consensus algorithm, comprising the following steps: Each node in a traditional RAFT cluster is virtualized into multiple RAFT instances, and each node in the RAFT cluster maintains a transaction state machine. The overall transaction state machine treats transaction data within the order lifecycle as the state, retrieves the initial daily data from the database as the initial state, and treats customer requests corresponding to customer transaction orders and downstream order reports and pushes as triggering conditions for state transitions. It changes the state according to the triggering conditions and triggers the corresponding business processing logic. Through load balancing strategies, it maintains the uniform distribution of the main core of the Raft instance on physical nodes to achieve efficient utilization. The main transaction state machine includes multiple sub-transaction state machines corresponding to multiple Raft instances. If the data corresponding to the trigger condition event distributed by the main transaction state machine is global data, the main transaction state machine will broadcast the global data update event to all Raft instances. The Raft instances will synchronize the data, commit it, and write it to the corresponding sub-transaction state machine. Each sub-transaction state machine will modify its own replica data after receiving the event broadcast. If the data corresponding to the trigger condition event distributed by the main transaction state machine is user data, it will be sharded according to the user account hash value. The corresponding Raft instance will commit the data and write it to the corresponding sub-transaction state machine. The sub-transaction state machine will submit the task and user account to the core computing thread pool to execute all corresponding state transitions and business processing. The core computing thread pool calculates the user account hash value corresponding to the user account and binds it with the corresponding number of the thread in the core computing thread pool. The thread for task execution is determined based on the binding result. The core computing thread pool allocates a Disruptor queue to each thread. It manages the entry and exit of queue elements through memory barriers and CAS operations with head and tail pointers. It avoids false sharing of head and tail pointers by filling rows with cache. It binds the thread to the CPU core and retrieves task data from the queue through Busy Spin to execute all corresponding state transitions and business processing.

[0007] Preferably, in high-speed mode, the trigger condition event is first written into the overall transaction state machine to execute business logic, and then the trigger condition event is submitted to the Raft instance for asynchronous data synchronization. If the trigger condition event is lost, the Raft instance repairs the data through compensation.

[0008] Preferably, the Raft instance repairs the intraday transaction data after the initial daily data, calculates taxes and fees, performs ex-rights and ex-dividend operations, etc., to perform daily settlement, and generates the initial daily data for the next trading day and writes it to the DB. At the same time, the overall transaction state machine also backs up the data for the day and clears the data for the day in the overall transaction state machine.

[0009] Preferably, the load balancing strategy includes: detecting the current distribution of primary cores; if the number of Raft primary cores of the current physical node is two or more more than the number of the fewest physical nodes, then moving one of the primary cores of the current node to the node with the fewest number of primary cores.

[0010] This invention provides a distributed high-frequency trading system based on the RAFT consensus algorithm. The system treats transaction data throughout the order lifecycle as a state using a central transaction state machine. Initial state is obtained from the database at the beginning of the day. Customer requests corresponding to customer transaction orders, downstream order reports, and push notifications are considered triggering events for state transitions. Raft instances change states based on these triggering events and send them sequentially to the central transaction state machine for execution. If the data is global, the Raft instance synchronizes and commits the data before writing it to the corresponding sub-transaction state machine. If the data is user data, it is sharded based on the user account hash value, and the corresponding Raft instance commits the data before writing it to the corresponding sub-transaction state machine. Tasks and user accounts are submitted to the core computing thread pool to execute all corresponding state transitions and business processing. A load balancing strategy maintains the even distribution of the Raft instance's main core across physical nodes, achieving efficient utilization. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the state and transition rules provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the transaction state machine and threads provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the transaction state machine and threads in the high-speed mode provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a routing strategy provided in an embodiment of the present invention. Detailed Implementation

[0012] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0013] This invention provides a method for designing a distributed high-frequency trading system based on the RAFT consensus algorithm, comprising the following steps: Each node in a traditional RAFT cluster is virtualized into multiple RAFT instances, while each node in the RAFT cluster maintains a general transaction state machine.

[0014] The overall transaction state machine treats transaction data throughout the order lifecycle as the state. It retrieves initial daily data from the database as the initial state and considers customer requests corresponding to customer transaction orders, as well as downstream (exchange-side or other downstream) order reports and pushes, as triggering events for state transitions. Based on these triggering events, the state changes, triggering corresponding business processing logic. A load balancing strategy maintains the even distribution of the Raft instance's main core across physical nodes, achieving efficient utilization. This ensures consistent and accurate management of order states and transaction data in high-concurrency scenarios, maintaining transaction data such as risk control parameters, user initial daily positions, initial daily funds, and trading permissions.

[0015] Specifically, when the overall trading state machine receives a client request, receives a push notification from the exchange, or receives instructions from the management to modify risk control indicators or other data, these are considered as triggering events for state transitions, and synchronization is performed within the cluster based on these triggering events.

[0016] like Figure 1 As shown, considering the possibility of out-of-order push notifications, the specific status and transition rules are as follows: 1. Initialization: When a customer order enters this system, during the risk control and verification of funds and securities, the order status is set to initialization.

[0017] (1) If the risk control or capital verification fails, the order status is set to invalid.

[0018] (2) Once risk control and capital verification are passed, the relevant funds and positions are frozen, and order placement instructions are sent to downstream entities. The status is set to pending.

[0019] 2. Pending Report: Orders have been submitted to the downstream trading system, but no push notifications have been received from the downstream system.

[0020] (1) Upon receiving a push notification from the downstream transaction system, maintain the downstream order number and set the status to positive.

[0021] (2) Upon receiving the notification from the exchange, maintain the exchange order number and set the status to "reported".

[0022] (3) Upon receiving the transaction notification from the exchange, maintain the transaction and position data, and set the status to partial / full completion based on whether all transactions have been completed.

[0023] (4) Upon receiving the cancellation order notification, unfreeze the funds and position, and set the status to cancellation order.

[0024] 3. Positive report: We have received a push notification from the downstream trading system, which includes the downstream order number, but we have not yet received a push notification from the exchange.

[0025] (1) Upon receiving the notification from the exchange, maintain the exchange order number and set the status to "reported".

[0026] (2) Upon receiving the transaction notification from the exchange, maintain the transaction and position data, and set the status to partial / full completion based on whether all transactions have been completed.

[0027] (3) Upon receiving the cancellation order notification, unfreeze the funds and position, and set the status to cancellation order.

[0028] 4. Reported: The exchange has sent a notification confirming the report, and the exchange order number is included.

[0029] (1) Upon receiving a push notification from the downstream transaction system, maintain the downstream order number and keep the status unchanged.

[0030] (2) Upon receiving the transaction notification from the exchange, maintain the transaction and position data, and set the status to partial / full completion based on whether all transactions have been completed.

[0031] (3) Upon receiving the cancellation order notification, unfreeze the funds and position, and set the status to cancellation order.

[0032] (4) Upon receiving the customer's cancellation instruction, after verifying that both the downstream order number and the exchange order number exist, the cancellation instruction is sent to the downstream and the status is set to pending cancellation.

[0033] 5. Completed: The final state, all orders have been executed.

[0034] (1) Upon receiving a push notification from the downstream transaction system, maintain the downstream order number and keep the status unchanged.

[0035] (2) Upon receiving the notification from the exchange, maintain the exchange order number and keep the status unchanged.

[0036] 6. Partial completion: The order is partially completed.

[0037] (1) Upon receiving a push notification from the downstream transaction system, maintain the downstream order number and keep the status unchanged.

[0038] (2) Upon receiving the notification from the exchange, maintain the exchange order number and keep the status unchanged.

[0039] (3) Upon receiving a transaction notification from the exchange, maintain transaction and position data. If all transactions are completed, set the status to "complete".

[0040] (4) Upon receiving a customer's cancellation instruction, send a cancellation instruction to downstream customers and set the status to pending cancellation.

[0041] 7. Pending Cancellation: The order has been cancelled by sending a cancellation instruction to the downstream.

[0042] (1) Upon receiving a customer's order cancellation instruction, send an order cancellation instruction to the downstream, and the status remains unchanged.

[0043] (2) Upon receiving a transaction notification, maintain the transaction data. If all transactions are completed, set the status to "Completed". Otherwise, leave it unchanged.

[0044] (3) Upon receiving the cancellation order notification, unfreeze the funds and set the status to cancellation order.

[0045] (4) Upon receiving the order cancellation notification, unfreeze the funds and position, and set the status to partially cancelled / cancelled depending on whether there was a partial transaction.

[0046] 8. Partial cancellation: The transaction is partially completed and then partially cancelled.

[0047] (1) Upon receiving a transaction notification, maintain the transaction data and keep the status unchanged.

[0048] 9. Cancelled: The order has been cancelled and is in the final state.

[0049] 10. Cancelled Order: The order is cancelled.

[0050] (1) Upon receiving a push notification from the downstream transaction system, maintain the downstream order number and keep the status unchanged.

[0051] (2) Upon receiving the notification from the exchange, maintain the exchange order number and keep the status unchanged.

[0052] like Figure 2 As shown, the main transaction state machine includes multiple sub-transaction state machines corresponding to multiple Raft instances. If the data corresponding to the triggering condition event distributed by the main transaction state machine is global data, such as risk control parameters, each sub-state machine maintains a replica. The main state machine will broadcast events involving global data updates to all Raft instances. The Raft instances will synchronize the data, commit it, and write it to the corresponding sub-transaction state machine. Each sub-state machine will modify its own replica data after receiving the broadcast. If the data corresponding to the triggering condition event distributed by the main transaction state machine is user data, it will be sharded according to the user account hash value (the mapping relationship is: sub-state machine ID = hash(account)%total number of sub-state machines). The corresponding Raft instance will commit the data and write it to the corresponding sub-transaction state machine, enabling the transaction state machines to execute concurrently, optimizing data synchronization performance, and solving the problem of low machine utilization in the traditional Raft master-slave architecture.

[0053] 1. Overall State Machine: The central hub for event routing. All successfully committed events are sent to the Raft instance in an orderly manner for routing, and then distributed by the Raft instance to the child state machines for execution.

[0054] 2. Sub-state machine: Performs state transitions and business processing, and works with the overall state machine to maintain system consistency.

[0055] The default task queue of a thread pool is usually implemented using a blocking queue: other threads submit tasks to this queue, and threads in the thread pool retrieve tasks from the blocking queue and execute them. However, the blocking queue generally uses locks to implement mutual exclusion for threads, which leads to performance degradation because when putting or polling elements into the queue, threads need to acquire locks, and locks are always slow.

[0056] Therefore, based on the Disruptor queue and combined with core binding technology, a high-performance thread pool was implemented to replace the native thread pool. The core design is as follows: 1. Lock-free Design: After the sub-transaction state machine submits the task and user account to the core computing thread pool for all corresponding state transitions and business processing, the core computing thread pool calculates the user account hash value and binds it to the corresponding thread number in the core computing thread pool. Based on the binding result, the thread executing the task is determined. The core computing thread pool allocates a Disruptor queue to each thread, managing the entry and exit of queue elements using memory barriers and CAS operations with head and tail pointers. Cache-filled lines prevent false sharing of head and tail pointers, binding the thread to the CPU core. Task data is retrieved from the queue using a Busy Spin approach, executing all corresponding state transitions and business processing. Since the read pointer is only read by one thread, there is no contention. The write pointer, through memory barriers and CAS operations, ensures thread safety when writing to the queue. Furthermore, to avoid contention caused by false sharing of read and write pointers, cache-filled lines are used to eliminate this problem.

[0057] 2. Looping and Data Retrieval: The thread uses the Busy Spin method to retrieve data from the Disruptor queue, that is, it loops to check if there are any new tasks submitted to the Disruptor queue and retrieves the tasks for execution. This can avoid the CPU entering a blocking state, reduce system calls, and lower latency.

[0058] 3. Thread-Core Binding: This binds a thread to a specific CPU core for execution. Simultaneously, operating system configuration settings are adjusted, such as CPU isolation to prevent other threads from preempting the core to which the computation thread is bound, interrupt isolation to prevent interrupted tasks from executing on the computation core, disabling idle to prevent the CPU from entering an idle state, and disabling hyper-threading to improve single-core performance.

[0059] The above design successfully reduced the task submission latency to 100ns, at the cost that the CPU cores bound to the computing threads are always at 100% full load.

[0060] Based on the sharding technique described above, a transaction state machine is split into a main state machine and multiple sub-state machines. Each state machine needs to ensure thread safety during state transitions. This technique adheres to a lock-free approach, using thread binding to avoid performance issues caused by locks: the user account is unidirectionally bound to a core computing thread based on its hash value, and all state transitions and business processing are executed in this thread.

[0061] Furthermore, it can be noted that the tasks of the sub-state machine must be submitted by the total state machine, and the total state machine is single-threaded. Therefore, there is no write contention in the thread queue of the sub-state machine. Based on this, this technology optimizes the thread queue by using only memory barriers to ensure visibility instead of using CAS, thereby reducing the cross-thread communication latency between the total state machine and the sub-state machine.

[0062] like Figure 3 As shown, due to the high latency caused by network I / O in Raft log commits, a high-speed mode is provided for the master node: In high-speed mode, the trigger condition event is first written to the overall transaction state machine to execute the business logic, and then the trigger condition event is submitted to the Raft instance for asynchronous data synchronization. If the trigger condition event is lost, the Raft instance repairs the data through compensation methods. In high-speed mode, the latency (receiving the first byte of the customer order network packet -> sending the last byte of the downstream order) can be compressed to 20µs.

[0063] The Raft instance repairs the intraday transaction data after the initial daily data, calculates taxes and fees, performs ex-rights and ex-dividend operations, etc., to perform daily settlement, and generates the initial daily data for the next trading day and writes it to the database. At the same time, the overall transaction state machine also backs up the data for the day and clears the data for the day in the overall transaction state machine.

[0064] The native Raft is a master-slave cluster where only the master node provides services and performs related operations, while the slave nodes only synchronize the state. This results in some performance waste and makes it impossible to scale horizontally. Therefore, we started multiple Raft instances within a single process and used a hash modulo algorithm to map different accounts to specific Raft instances. The mapping relationship is: Raft instance Id = hash(account) % total number of Raft instances.

[0065] To avoid multiple Raft instances electing the same physical node as the master node during the election process, which would cause a fallback to the original one-master-multiple-slave logic, we added a load balancing strategy to the distribution of Raft master nodes: 1. Start a load balancing thread when the service starts. 2. This thread performs a scan every 30 seconds to obtain the distribution of the main cores of the current Raft instances on the physical nodes.

[0066] 3. If the number of primary cores of the Raft instance in the current physical node differs from the number of primary cores of the physical node with the fewest primary cores by more than or equal to 2, then move one primary core of the current node to the node with the fewest primary cores.

[0067] like Figure 4 As shown, the routing strategy is: 1. The transaction gateway updates the main core topology of the Raft instances on the physical nodes in real time in memory. 2. Upon receiving the request, the gateway will perform a hash modulo operation on the account to obtain the Raft instance ID. 3. The gateway queries the Raft cluster topology in memory based on the Raft instance ID to obtain the physical node where the main core resides. 4. Send the message to the physical node where the main core is located.

[0068] The requested account is hashed, and the Raft core number is obtained by taking the modulo of the Raft core. The node where the main core of the Raft core is located is obtained according to the Raft routing table, and the request is sent to that node.

[0069] Since both intraday market monitoring and post-market clearing require standard reading of transaction data, this system adds a learner node to the cluster. This learner node synchronizes logs from the leader and writes transaction data to the database.

[0070] Considering that learner nodes may also crash, this solution provides a high-performance and recoverable method for data persistence, ensuring that no matter how the learner service is restarted, data will not be duplicated or missing from the database.

[0071] 1. Create a new table in the database to record the log numbers currently written to the database. There is one record for each Raft instance. The learner node will cache this data in memory when it starts.

[0072] 2. When the learner state machine is maintaining the memory state, it will check whether the current log number has been written to memory based on step 1. If it has not been written to memory, it will synchronously generate a database data update SQL and submit it to the queue.

[0073] 3. The learner node starts a database persistence thread that consumes queue 2. After collecting a batch of data, the persistence thread will merge it into a large transaction and write it to the database in batches, and update the log number in table 1 in the transaction.

[0074] 1. The transaction state machine design treats the transaction data as the initial state of the state machine, and treats customer instructions and exchange pushes as events, defining the state and transition logic.

[0075] 2. The transaction state machine is combined with the Raft state machine to support the synchronization of transaction data among nodes in the Raft cluster. The logic is clear and easy to implement in engineering.

[0076] 3. Thread model: Performance is optimized through strategies such as circular queue, cache line filling, CAS, busy spin, and lock-free business design. Thread submission latency can be reduced to 100ns, and penetration latency can be reduced to 20us.

[0077] 4. The multi-Raft instance service model design improves the performance utilization of cluster nodes, enabling multiple physical nodes to provide services simultaneously, resulting in higher throughput and lower latency. 5. The Learner node is designed for database persistence, ensuring accurate data persistence to the database in one go, and high performance is guaranteed through transaction merging and other methods.

[0078] 1. Disaster Recovery: The entire process from leader downtime to successful election of a new leader and commencement of service is completed within seconds, requiring no human intervention. 2. Strong consistency: During data synchronization, storage, and master-slave switching, the Raft consensus algorithm guarantees strong consistency, and theoretically, there will be no data loss.

[0079] 3. High performance: Thread model optimization and lock-free design improve performance. Multiple Raft instances fully utilize the performance of multi-core and slave nodes, greatly improving throughput.

[0080] 4. The clear transaction state machine design combined with Raft reduces the complexity of engineering implementation, thus greatly reducing the cost of developing a high-performance, highly available, and strongly consistent transaction system.

[0081] 5. Transaction data is stored in the database through learner nodes, allowing other services to read it freely using standardized SQL without additional integration.

[0082] The same effect can be achieved by using the Paxos consensus algorithm instead of the Raft algorithm, but Paxos is more complex and not suitable for engineering.

[0083] A similar effect can be achieved using Kafka + ZoomKeeper, with Kafka replacing the log replication part of this solution and ZoomKeeper replacing the election part. However, it relies on more middleware, and Zab cannot provide strong consistency. Coupled with its complex algorithm design, using Zab instead of Raft is a poor choice.

Claims

1. A design method for a distributed high-frequency trading system based on the RAFT consensus algorithm, characterized in that, Each node in a traditional RAFT cluster is virtualized into multiple RAFT instances, and each node in the RAFT cluster maintains a general transaction state machine. The overall transaction state machine treats transaction data within the order lifecycle as the state, retrieves the initial daily data from the database as the initial state, and treats customer requests corresponding to customer transaction orders and downstream order reports and pushes as triggering conditions for state transitions. It changes the state according to the triggering conditions and triggers the corresponding business processing logic. Through load balancing strategies, it maintains the uniform distribution of the main core of the Raft instance on physical nodes to achieve efficient utilization. The main transaction state machine includes multiple sub-transaction state machines corresponding to multiple Raft instances. If the data corresponding to the trigger condition event distributed by the main transaction state machine is global data, the main transaction state machine will broadcast the global data update event to all Raft instances. The Raft instances will synchronize the data, commit it, and write it to the corresponding sub-transaction state machine. Each sub-transaction state machine will modify its own replica data after receiving the event broadcast. If the data corresponding to the trigger condition event distributed by the main transaction state machine is user data, it will be sharded according to the user account hash value. The corresponding Raft instance will commit the data and write it to the corresponding sub-transaction state machine. The sub-transaction state machine will submit the task and user account to the core computing thread pool to execute all corresponding state transitions and business processing. The core computing thread pool calculates the user account hash value corresponding to the user account and binds it with the corresponding number of the thread in the core computing thread pool. The thread for task execution is determined based on the binding result. The core computing thread pool allocates a Disruptor queue to each thread. It manages the entry and exit of queue elements through memory barriers and CAS operations with head and tail pointers. It avoids false sharing of head and tail pointers by filling rows with cache. It binds the thread to the CPU core and retrieves task data from the queue through Busy Spin to execute all corresponding state transitions and business processing.

2. The design method for a distributed high-frequency trading system based on the RAFT consensus algorithm as described in claim 1, characterized in that, In high-speed mode, the triggering condition event is first written into the overall transaction state machine to execute the business logic, and then the triggering condition event is submitted to the Raft instance for asynchronous data synchronization. If the triggering condition event is lost, the Raft instance will repair the data through compensation.

3. The design method for a distributed high-frequency trading system based on the RAFT consensus algorithm as described in claim 2, characterized in that, The Raft instance repairs the intraday transaction data after the initial daily data, calculates taxes and fees, performs ex-rights and ex-dividend operations, etc., to perform daily settlement, and generates the initial daily data for the next trading day and writes it to the DB. At the same time, the overall transaction state machine also backs up the data for the day and clears the data for the day in the overall transaction state machine.

4. The design method of a distributed high-frequency trading system based on the RAFT consensus algorithm as described in claim 1, characterized in that, The load balancing strategy includes: detecting the current distribution of master cores; if the number of Raft master cores of the current physical node is two or more more than the number of the fewest physical nodes, then moving one of the master cores of the current node to the node with the fewest master cores.

5. The design method for a distributed high-frequency trading system based on the RAFT consensus algorithm as described in claim 1, characterized in that, The binding refers to the one-way binding of the user account corresponding to the user account hash value to a thread in the core computing thread pool.

Images