Security transaction counter system crash recovery method based on event playback

Abstract

According to the security transaction counter system crash recovery method based on event playback, persistence is carried out before event processing, the memory state is rebuilt through the playback event during crash recovery to guarantee high consistency of data, full-memory operation is used in the data processing process, delay is reduced, the throughput of the system is improved, and the security transaction counter system crash recovery efficiency is improved. And for data such as persistent user requests and exchange returns, through an event playback mechanism, zero data loss of crash recovery is ensured, and the problem that a small part of data is lost during high-frequency transactions in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of system crash recovery, in particular to a securities trading counter system crash recovery method based on event playback. BACKGROUND

[0002] The securities trading counter system needs to process orders, funds and position data in real time, and has strict requirements for low latency (millisecond level response) and data strong consistency (zero fund / position error). The traditional scheme is to ensure consistency, and each business operation (such as order placement and order cancellation) needs to read and write the database synchronously, which leads to I / O bottleneck in high-frequency trading scenarios, and the system throughput is limited.

[0003] The existing industry mainstream scheme is a full-amount synchronous persistence scheme. When processing each user request or exchange return, the latest funds, positions, and rates are loaded from the database; then the business logic (such as deducting funds and positions) is executed; finally, the results are written to the database. The disadvantage of this scheme is that frequent database access will cause high latency in the processing process (the processing process of a single transaction link is often more than 10ms), which cannot meet the needs of high-frequency trading.

[0004] If a pure memory asynchronous scheme is used for optimization, the business intermediate state process is all memory calculation, and the database is asynchronously written at regular intervals or using a queue. Although the delay is very low and the speed is very fast, when the system crashes, a small amount of data will be lost, which violates the regulatory requirements of zero error in the securities industry and leads to violation of financial safety standards. SUMMARY

[0005] The technical problem to be solved by the technical scheme of the present application is: how to reduce latency and improve throughput while ensuring data strong consistency, and how to 100% rebuild the memory state during crash recovery to ensure zero data loss.

[0006] The technical scheme of the present application provides a securities trading counter system crash recovery method based on event playback, comprising the following steps: Step 1, before the crash of the securities trading counter system, the user's pre-verified order generates a client order number, which is collectively stored as persistent data in a persistent system; Step 2, the securities trading counter system executes business logic in memory according to user orders to obtain bid request data, and obtains a client order number to assemble a bid request and send it to an exchange and accept exchange returns. If the bid fails to send, it is stored as persistent data in the persistent system, and if the bid is successfully sent, the exchange returns the bid failure return or the bid success return according to the client order number and stores it as persistent data in the persistent system; Step 3, according to the exchange offer failure return or exchange offer success return, the relevant customer order information in the memory is searched, the business logic is executed in combination with the current memory account fund, the position state, and the execution result is stored as persistent data according to the customer order number to the persistent system; Step 4, when the securities trading counter system crashes, check whether the persistent data in the persistent system needs to be played back according to the time stamp, if it needs to be played back, check whether there is a user order associated with the offer sending failure record, if there is, directly mark the offer sending failure, and push the user return, if not, execute step 2 according to the user order, and push the user return; continue to query whether there is an execution result, if not, execute step 3 again, and push the user return; Step 5, the securities trading counter system crashes and recovers, and enters the normal business processing mode.

[0007] Preferably, the pre-verification includes verifying the account number, fund, and position in the user information.

[0008] Preferably, in step 4, the re-execution of step 2 according to the user order and the pushing of the user return are only required to be sent and pushed in the recovery process of the last persistent data, and if there is repeated data, the verification is realized according to the client order number / exchange order number / serial number to realize the de-duplication.

[0009] The present application provides a kind of securities trading counter system crash recovery method based on event playback, by being persisted before event processing, reconstructs memory state by playing back event when crash recovery, to guarantee the strong consistency of data, realizes using full memory operation in data processing process, reduces delay, improves the throughput of system, for the data such as persistent user request, exchange return, by event playback mechanism, ensure that crash recovery zero data loss, solve the problem of a small amount of data loss in prior art in high-frequency trading. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The push user order return process schematic diagram provided for the embodiment of the present application is provided; Figure 2 The push exchange return process schematic diagram provided for the embodiment of the present application is provided; Figure 3 The crash restart and recovery process schematic diagram provided for the embodiment of the present application is provided. DETAILED DESCRIPTION

[0011] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0012] This invention provides a crash recovery method for a securities trading platform system based on event replay, comprising the following steps: like Figure 1 As shown, 1. Securities trading counter system startup initialization: When the trading counter starts, it performs initialization operations, including loading user information, including core data such as account, funds, and holdings.

[0013] 2. User-managed processing flow: 2.1. Pre-transaction Verification: Upon receiving a user's order, the securities trading platform system first conducts a rigorous pre-transaction verification. Verification includes, but is not limited to: compliance of order details (such as stock code, price, and quantity), system readiness, whether it is within a valid trading session, and whether the account has sufficient funds. Orders that fail verification are immediately rejected, and a failure message is sent to the client.

[0014] 2.2. Order Persistence and Order Number Generation: Orders that have passed pre-verification are synchronously submitted to the persistence system for storage. The persistence system generates a unique client order number for each order for the day and returns this order number to the transaction counter's core processing module.

[0015] 2.3. In-memory business processing: The core module of the trading desk executes business logic processing related to the order in memory, including but not limited to: freezing / adjusting the corresponding funds or positions, updating the order status, etc.

[0016] 2.4. Order Submission: After the business logic is completed, the trading desk core module obtains the unique customer order number from the persistent system, assembles an order request containing the unique customer order number, and sends it to the exchange.

[0017] 2.5. Successful Order Submission Processing: If the order message is successfully sent to the exchange (subject to confirmation of successful transmission by the underlying network protocol), the securities trading counter system will push a "submitted" status report to the client.

[0018] 2.6. Handling Disk Sending Failures: 2.6.1 Result of failure to synchronize persistent delegate submission: 2.6.2 Rollback of business operations such as fund / position adjustments performed in memory in section 2.4; 2.6.3 Push a "dispatch failed" status report to the client.

[0019] As Figure 2 shown, the exchange return processing flow: 3.1. Return persistence: After receiving the exchange return (covering new orders accepted / rejected, partial execution, full execution, order cancellation accepted / rejected, etc.), the persistence system first synchronously persists the return message; 3.2. In-memory business processing: According to the key information (such as exchange order number, client order number) contained in the return, find the related client order information in memory, combine the current account funds, position state, and execute the corresponding business logic (such as: unfreeze / adjust funds / positions, update the transaction status, etc.).

[0020] 3.2. Return push: After the business processing is completed, push the corresponding order / transaction status change return to the client.

[0021] 4. Crash restart and recovery process, as Figure 3 shown: 4.1 Initialization: After the restart of the securities trading counter system, the same steps as the start initialization are executed (load disk initial user account, funds, positions, etc. information); 4.2 Replay events: Check if there is any persistent data that needs to be played back (persistent data is the data received by the counter system before the restart of the day, which includes user orders, failed bid sending, exchange returns, and related data before the restart of the day. If there is no, it means the restart recovery process is over, and the subsequent business is handled normally; 4.3 Sequential playback: Process the persistent data in sequence (persistent data is marked with a timestamp when stored, and playback is performed according to the timestamp at this time).

[0022] 4.3.1 User order record processing: Query if there is any persistent bid sending failure record associated with this order.

[0023] If yes, it means that the order has been confirmed as failed before the crash. The securities trading counter system does not need to perform the business adjustment logic such as funds and positions, and can directly mark the bid sending failure and push the user return; If not, it means that the order has been persisted before the crash, but the subsequent state is unknown (it may have crashed before bidding, or it may have been bid). The securities trading counter system re-executes the business processing of the order (same as 2.3), sends the bid message to the exchange and pushes the user return; 4.3.2 Bid sending failure record processing: It has been associated and processed in section 4.3.1, and does not need to be further processed; 4.3.3 Exchange return record processing: for the persisted exchange returns, the securities trading counter system re-executes the corresponding business process (same as 3.2) and pushes the user returns.

[0024] 4.4. Recovery completion: after all the persisted data is processed, the restart recovery process ends, and the securities trading counter system enters the normal business processing mode.

[0025] Key optimization: reduce repeated bids and user return pushing: Premise: message processing is linear, and the successful persistence of the next record means that all processing steps of the previous record have been completed (including business processing, bidding, and pushing).

[0026] Mechanism: In sections 4.3.1 and 4.3.3, sending bid messages to the exchange and pushing user returns only need to be sent and pushed during the recovery processing of the last persisted data. The sending and pushing steps can be skipped during the recovery processing of previous persisted data, reducing unnecessary repeated data sending processes.

[0027] Guarantee: The sending of bid messages to the exchange and the pushing of user returns have key de-duplication mechanisms based on unique customer order numbers, exchange order numbers, and serial numbers, etc. Repeatedly pushed bids or returns will be identified and ignored, and will not cause errors or repeated processing.

[0028] Data strong consistency effect analysis: The core goal of the crash recovery mechanism based on event playback proposed in this patent is to ensure that in the case of unexpected crashes of the securities trading counter system, the user account state and related records such as orders and transactions can also achieve strong consistency, without data loss. The securities trading counter system crash can occur at any stage of the business processing process. The following scenario analysis is conducted for key crash points to explain how the recovery mechanism ensures consistency: 5.1 Crash occurs before order persistence (before step 2.2).

[0029] State: order is persisted, in-memory business is not processed, bid is not sent.

[0030] Recovery processing: after restarting, check for no persisted record of this order, so it will not be processed. The order is as if it never happened.

[0031] Consistency guarantee: the user state is not affected, the client timeout does not receive confirmation, which meets the expectation. The client can report an error or resubmit the order.

[0032] 5.2 Crash occurs after order persistence (after step 2.2): 5.2.1 Crash happens before / while the in-memory order processing (in step 2.3); 5.2.2 Crash happens after the in-memory order processing is completed and before the bid is sent (in step 2.4); 5.2.3 Crash happens after the bid is sent successfully (after step 2.5); Status: The order is persisted, and the subsequent process is only half processed or successfully completed.

[0033] Recovery process: The recovery process will find the persisted order record as described in 4.3.1. Query the associated bid failure record, so re-execute the complete in-memory business processing, and then send the bid (actually the last record is sent). Since the initial state before and after the restart, and the business processing logic is completely consistent, the final recovery result is also consistent with the processing result before the crash.

[0034] Consistency guarantee: The order is successfully recovered. In the scenario of 5.2.3, even if the bid has been sent successfully before the crash, the exchange will perform deduplication when it is re-sent during recovery, and the final system state is consistent with the exchange.

[0035] 5.3 Crash happens after the bid is sent unsuccessfully, and the bid sending failure record is persisted (after step 2.6, before 2.6.1) Status: The order is persisted, the in-memory business processing is completed, and the bid sending fails but is not persisted.

[0036] Recovery process: As described in 4.3.1, the recovery process will find the persisted order record. Query the associated bid failure record, so re-execute the complete in-memory business processing, and then send the bid (actually the last record is sent).

[0037] Consistency guarantee: The order is successfully recovered. Since the bid is re-sent during recovery, the bid sending may succeed this time, which is different from the result before the restart when the bid sending fails. However, since messages are processed strictly in order, in this scenario, the record must be the last user order. The in-memory state before the restart has not been rolled back, and the return has not been sent to the customer. After the restart, the bid is re-sent and succeeds, which can be considered as a normal optimization retry mechanism for the user order, and there is no data inconsistency.

[0038] 5.4 Crash happens after the bid is sent unsuccessfully and the bid sending failure record is persisted (after step 2.6.1); Status: The order is persisted, the bid sending fails and is persisted, the in-memory data rollback state is unknown, and the customer return sending state is unknown.

[0039] Recovery: As described in 4.3.1, the recovery process will find the persistent delegation record. Query the associated bid failure record, the system does not need to make business adjustment logic such as capital position, can directly mark the bid sending failure, and push the user feedback (actually the last record is sent).

[0040] Consistency guarantee: The delegation state is marked as bid sending failure, and the memory business data remains unchanged. At this time, a user feedback may be sent repeatedly (because the user feedback state is unknown), but the feedback pushing system will do the deduplication processing, which has no actual effect on the user.

[0041] 5.5 Crash occurs before exchange feedback persistence (before step 3.1); State: Exchange feedback is not persisted, memory business is not processed, and client is not pushed.

[0042] Recovery processing: There is no persistent record of this feedback after restart, so there is no recovery processing. The feedback is "lost" (after the end of the recovery state, the exchange will retransmit during normal business processing).

[0043] Consistency guarantee: Depend on the retransmission mechanism of the exchange. When the system is running normally after recovery, it will reacquire the unreceived feedback through the exchange feedback retransmission interface, and then follow the normal processing process. The final state will catch up with the exchange. This is a common fault tolerance mechanism for broker systems, not a problem solved by this recovery mechanism alone. This mechanism ensures that the processing is reliable after the feedback is successfully persisted.

[0044] 5.6 Crash occurs after exchange feedback persistence (after step 3.1); 5.6.1 Before / while processing memory business (in step 3.2); 5.6.2 After memory business processing is complete, before / while / after feedback pushing (step 3.3); State: Exchange feedback has been persisted, memory business state is unknown, and user feedback pushing state is unknown.

[0045] Recovery processing: According to the description in 4.3.3, the system re-executes the corresponding business processing (same as 3.2) and pushes the user feedback.

[0046] Consistency guarantee: Business processing transactions are re-executed based on the initial state and persisted feedback information during recovery, ensuring the correctness of business processing data. Since the user feedback pushing state is unknown in the 5.6.2 scenario, there may be a situation where a user feedback is pushed multiple times, but the message pushing system will do the deduplication, ensuring that the user receives and only receives one feedback.

[0047] 6. Summary of strong consistency key design: 6.1 Event (Order / Trade) Prioritized Persistence: All external inputs (orders) or decisive events (exchange trade) that can change the system state are first synchronized and persisted, forming a reliable event source.

[0048] 6.2 Event Deterministic Replay: The core of crash recovery is to reprocess these events in the order of persistence (event occurrence order). The processing logic is deterministic.

[0049] 6.3 State Rebuild Instead of Recovery: Instead of checkpoint recovery, the memory state is rebuilt by replaying the persisted events in order after loading the initial state during initialization.

[0050] 6.4 Idempotency Design: 6.4.1 Business logic design (such as freeze / thaw, update state) is as idempotent as possible.

[0051] 6.4.2 Operations on external systems (quotes, pushes) are guaranteed to be executed only once through unique identification (order number / order number) and deduplication mechanisms.

[0052] 6.4.3 Utilize linear persistence characteristics and the optimization of "send / push only when the last record is processed", significantly reducing unnecessary and potentially side-effect-causing repeated operations.

[0053] Through the above mechanisms, the patent can achieve strong consistency of user funds, positions, order status, trade status, and other key data after a crash, which is consistent with the exchange records and business rules, without relying on full synchronization mechanisms and business processing in memory. This effectively solves the key data consistency problem in high availability of securities trading systems.

[0054] Key points of the embodiment of the invention: By persisting before event processing and rebuilding memory state through event replay during crash recovery, the strong consistency of data is guaranteed. During normal business processing, read operations and synchronous write operations to the database are completely avoided, significantly reducing system latency and improving system throughput. Protection point: persist before event processing and rebuild memory state through event replay during crash recovery to ensure strong consistency of data.

[0055] The embodiment of the invention can greatly improve the processing speed of orders and reduce the processing delay of individual processing steps from 10-50ms to 1-5ms while ensuring 100% data recovery after a crash.

[0056] Some alternative solutions are as follows: Event storage alternative: WAL log can be replaced by Kafka persistent queue or Redis transaction stream.

[0057] Playback logic alternative: Sequential playback can be replaced by parallel shard playback, with user ID sharding to guarantee state isolation.

[0058] State snapshot optimization: Generate memory state snapshots at regular intervals, and when recovering, play back from the snapshot + incremental time, reducing recovery time.

[0059] The above alternatives are all based on the core principle of "event persistence prior to business processing" unchanged, to ensure that data is not lost.

Claims

1. A crash recovery method for a securities trading platform system based on event replay, characterized in that, Includes the following steps: Step 1: Before the securities trading counter system crashes, the customer order number generated from the user orders verified beforehand is stored as persistent data in the persistent storage system. Step 2: The securities trading counter system executes business logic in memory according to the user order processing to obtain the order request data, obtains the customer order number, assembles it into an order request, sends it to the exchange, and receives a response from the exchange. If the order sending fails, it is stored as persistent data in the persistent storage system. If the order sending is successful, the order failure or success report from the exchange is stored as persistent data in the persistent storage system based on the customer order number. Step 3: Based on the exchange's order failure or success report, the relevant customer order is searched in memory. The system executes business logic based on the current account funds and holding status in memory, and stores the execution result as persistent data in the persistent system according to the customer's order number. Step 4: When the securities trading counter system crashes, it checks whether the persistent data in the persistent system needs to be replayed according to the timestamp. If so, it checks whether there is a record of failed order sending associated with the user's order. If there is, it marks the order sending as failed and pushes a report to the user. If not, it re-executes step 2 according to the user's order and pushes a report to the user. It continues to check whether there is an execution result. If not, it re-executes step 3 and pushes a report to the user. Step 5: The securities trading counter system crash recovery ends and enters the normal business processing mode.

2. The method for crash recovery of a securities trading counter system based on event replay as described in claim 1, characterized in that, The pre-verification includes verifying the user's account, funds, and holdings.

3. The method for crash recovery of a securities trading counter system based on event replay as described in claim 1, characterized in that, In step 4, the step of re-executing step 2 according to the user's entrustment and pushing the user's feedback only needs to be sent and pushed during the recovery process of the last persistent data. If there is duplicate data, it is deduplicated by verifying the customer's entrustment number / exchange order number / transaction number.

Images