Transaction risk control system and method based on memory and medium

Abstract

One aspect of the invention discloses a transaction risk control system based on a memory. On the other hand, the invention discloses a transaction risk control method based on the memory. Another aspect of the invention is to disclose a computer readable storage medium. Compared with the prior art, the method has the following technical advantages: 1) real-time performance: the response time of a rule engine is less than 5ms, and a trillion-level transaction volume per day is supported; 2) reliability: a triple check mechanism (CRC32 + data consistency + risk control management) realizes transaction security; 3) expansibility: the horizontal expansion supports more than 100 node clusters, and the processing capability is linearly improved; compared with the prior art, the method has the following commercial values: 1) the method is suitable for the scenes of securities, futures, digital currency exchanges and the like; 2) the hardware investment cost is reduced by 45% (compared with the traditional Oracle scheme); and 3) 98% of transaction loss caused by system faults is reduced.

Description

Technical Field

[0001] The present invention relates to a real-time risk control system and method for securities trading based on an in-memory database, which is particularly applicable to low-latency risk control decision-making, data consistency guarantee, and rapid disaster recovery in high-frequency trading scenarios, and belongs to the field of financial technology. Background Art

[0002] The following problems generally exist in traditional securities trading risk control systems:

[0003] First, performance bottleneck: Disk-based relational databases (such as Oracle, MySQL) result in trading latency > 50 ms and throughput < 5000 TPS.

[0004] Second, poor disaster recovery ability: RTO (Recovery Time Objective) > 30 minutes and RPO (Recovery Point Objective) > 10 minutes.

[0005] Third, lack of real-time performance: The calculation latency of the rule engine fluctuates greatly (P99 latency > 20 ms), unable to meet the millisecond-level high-frequency trading requirements.

[0006] Fourth, data integrity risk: Traditional log backup mechanisms lack multi-layer verification, and the data error rate > 0.01%.

[0007] Fifth, low memory utilization: The memory fragmentation rate > 15%, resulting in significant performance degradation after long-term system operation. Summary of the Invention

[0008] The objectives of the present invention are: 1) achieving sub-millisecond response for securities trading risk control (target P99 latency < 10 ms); 2) ensuring zero data loss (RPO = 0) and second-level switching (RTO < 2 s) during disaster recovery; 3) enhancing the concurrent processing ability of the in-memory database (target throughput > 100,000 TPS); 4) designing a multi-layer verification mechanism to ensure data integrity (error rate < 10 -9 )

[0009] To achieve the above objectives, one aspect of the present invention discloses a memory-based trading risk control system, which is characterized by including an in-memory database cluster and using a Kafka cluster to achieve cross-data center transaction log synchronization, wherein:

[0010] The in-memory database cluster includes:

[0011] The main process adopts a multi-threaded event loop model and manages the in-memory data storage through a T-tree + HASH index;

[0012] At least one sub-process achieves lock-free data synchronization through shared memory mapping, and the transaction log adopts a custom binary format with a CRC32 check code;

[0013] The transaction log is stored in a shared memory block;

[0014] Utilize the Kafka cluster to achieve cross-data center transaction log synchronization, realizing:

[0015] Asynchronously write the transaction log through the Kafka producer API;

[0016] The consumer group in the remote data center realizes dual-active storage of the log;

[0017] The SSD array writes to the database in Append-Only mode.

[0018] Preferably, the main process performs the following operations:

[0019] At startup, allocate a non-persistent shared memory and load the local database file into it;

[0020] Meanwhile, open a shared memory for transaction logs. For the subsequent data addition, deletion, query, and modification operations in the transaction service, corresponding transaction logs will be generated and written into the shared memory, notifying the child process to read and perform subsequent backup and distributed storage.

[0021] Preferably, the child process performs the following operations: Listen to the semaphore of the main process through POSIX. When a new transaction log is triggered, it needs to process the shared memory data;

[0022] Adopt the copy-on-write mechanism to ensure data consistency;

[0023] Generate transaction log entries containing global sequence numbers and nanosecond-level timestamps.

[0024] Preferably, the control header information of the shared memory includes: shared memory size, main process data writing position, child process reading position, version information; after the control header information is the transaction log with CRC32 checksum; a loopback marker is reserved at the end of the shared memory, which is used to indicate whether to reuse the shared memory from the beginning. When the memory is insufficient, the main process writes from the beginning, overwriting the historical transaction logs that have been read.

[0025] Preferably, the log synchronization process includes: generating log entries within 200 μs after the main node transaction is committed; realizing cross-data center transmission through the cluster network; adopting a unique transaction ID to ensure the sequential consistency of the logs;

[0026] The asynchronous transaction warehousing process includes: adopting batch transaction logs and atomic operation processing methods. For the transaction logs in the same batch, ensure atomic operation warehousing, and before warehousing, verify the sequence number and status of the transaction logs;

[0027] The disaster recovery process includes: triggering a switch within 1 second after detecting the failure of the primary node; loading and restoring the state of the in-memory database from the most recent snapshot of the local backup; replaying the Kafka log to compensate for uncommitted transactions.

[0028] Another aspect of the present invention discloses a memory-based transaction risk control method, which is characterized in that the above-mentioned transaction risk control system is adopted, including the following steps:

[0029] Receiving a transaction request and parsing session metadata;

[0030] Parallelly executing the compliance verification of a preset risk control rule set;

[0031] The transaction request passing the verification triggers an atomic operation of the in-memory database;

[0032] Asynchronously generating a transaction log with a timestamp and pushing it to the Kafka cluster.

[0033] Another aspect of the present invention discloses a computer-readable storage medium, which is characterized in that it stores program codes for implementing the above-mentioned transaction risk control method, including: a main process event loop scheduling module; a shared memory synchronization management module; a transaction log backup storage module.

[0034] Compared with the prior art solutions, the present invention has the following technical advantages:

[0035] 1) Real-time performance: The response time of the rule engine < 5ms, supporting a daily trading volume of trillions;

[0036] 2) Reliability: A triple verification mechanism (CRC32 + data consistency + risk control management) realizes transaction security;

[0037] 3) Scalability: Horizontal expansion supports a cluster of > 100 nodes, linearly improving the processing capacity;

[0038] And compared with the prior art solutions, the present invention has the following commercial values:

[0039] 1) Applicable to scenarios such as securities, futures, digital currency exchanges, etc.;

[0040] 2) Reducing the hardware input cost by 45% (compared with the traditional Oracle solution);

[0041] 3) Reducing the trading losses caused by system failures by 98%. Description of the Drawings

[0042] Figure 1 Schematically shows the overall architecture of the system of the present invention

[0043] Figure 2 It is a schematic diagram of the shared memory block for transaction logs;

[0044] Figure 3 Schematically shows the data processing and backup process;

[0045] Figure 4 Schematically shows the Kafka cluster to achieve cross-data center transaction log synchronization;

[0046] Figure 5 Is the flowchart of the real-time risk control method;

[0047] Figure 6 Schematically shows the overall process of the present invention. Specific embodiments

[0048] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0049] Combined with Figure 1 And Figure 6 , a memory-based transaction risk control system disclosed in an embodiment of the present invention adopts a memory database cluster and uses a Kafka cluster to achieve cross-data center transaction log synchronization.

[0050] The key technical solutions adopted by the memory database cluster aims.base.fastdb include:

[0051] Main process: multi-threaded database event loop processing, and the in-memory data structure adopts a T-tree + HASH index;

[0052] Sub-process: synchronize the shared memory of the main process through semaphore communication, adopt the Copy-on-Write mechanism to realize the local backup and restoration of transaction logs, and perform remote data synchronization backup and recovery through Kafka, and store it in EMS.DB (Oracle);

[0053] Transaction log: custom binary format (including CRC32 checksum).

[0054] Specifically, the memory database cluster includes:

[0055] The main process adopts a multi-threaded event loop model and manages the full-memory data storage through a T-tree + HASH index. The main process performs the following operations: At startup, it allocates a non-persistent shared memory and loads the local database file into it. At the same time, it opens a transaction log shared memory (persistent). For the subsequent data addition, deletion, query, and modification operations of the transaction service, corresponding transaction logs will be generated and written into the shared memory, notifying the child process to read and perform subsequent backup and distributed storage. This invention is significantly different from previous databases. Traditional databases rely on files, that is, each operation needs to submit and update the data to the file. In the system disclosed in this invention, only when starting for the first time, the content of the file is loaded into the shared memory. Subsequent operations on the database are all performed in the memory. The transaction log also directly uses the memory and semaphore method to achieve asynchronous backup.

[0056] At least one child process realizes lock-free data synchronization through the shared memory mapping method. The transaction log adopts a custom binary format with a CRC32 checksum. The child process performs the following operations: listens to the semaphore of the main process through POSIX. When a new transaction log is triggered, it needs to process the shared memory data. It uses the Copy-on-Write mechanism to ensure data consistency. It generates a transaction log entry containing a global sequence number and a nanosecond-level timestamp.

[0057] As Figure 2 shown, the transaction log is stored in the shared memory block, which solves the dependence on disk I / O. The control header information of the shared memory includes: shared memory size, main process data writing position, child process reading position, version information, etc. Followed by the transaction log with a CRC32 checksum. Each transaction log has a Header header information with a CRC checksum and Body information. Finally, a loopback marker will be reserved at the end of the shared memory. This marker will indicate whether to reuse the shared memory from the beginning. The entire shared memory block design is self-researched and adapted to the above system. Since the amount of transaction logs generated every day will increase with the increase in the usage of the service, a loopback mechanism is introduced. That is, when the memory is insufficient, the main process will write from the beginning and overwrite the historical transaction logs that have been read.

[0058] As Figure 4 shown, this invention uses a Kafka cluster to achieve cross-data center transaction log synchronization, and the RDMA network transmission delay < 200 μs.

[0059] The transaction processing flow includes the following steps:

[0060] 1) The main process performs a series of table-based addition, deletion, query, and modification operations on the database, generating corresponding transaction logs;

[0061] 2) The transaction log is stored in a local shared memory data block, and every time a new transaction log appears, a trigger signal notifies the subprocess to extract it;

[0062] 3) The subprocess restores the content of the shared memory data block locally according to the specified location and size;

[0063] 4) The restored data is transmitted to a specified Topic through Kafka for remote data cold backup;

[0064] 5) The remote service listens to the Topic of the Kafka cluster, pulls the data, and converts it into SQL statements to achieve storage in the database.

[0065] The asynchronous backup process includes:

[0066] - Local persistence: The subprocess generates a full - volume local database backup file according to the transaction log and time;

[0067] - Off - site backup: The Kafka producer group writes to the SSD array in the off - site computer room in real time;

[0068] - Off - site storage in the database: The Kafka consumer group reads the transaction log in real time, executes the stored procedure, and persists the transaction data into the DB;

[0069] - Data consistency: A unique transaction ID is adopted to achieve consistency during cross - computer - room log synchronization.

[0070] The distributed log service of the real - time risk control system for securities trading based on the in - memory database disclosed in the embodiments of the present invention specifically includes: the EMS trading system and the OMS instruction management system cluster; the Kafka multi - node cross - computer - room cluster deployment; the remote backup and consistency verification mechanism adopting the main process - subprocess architecture.

[0071] The distributed log service system realizes cross - computer - room transaction log synchronization based on Kafka, and achieves: asynchronously writing transaction logs through the Kafka producer API; realizing log dual - active storage by the consumer group in the off - site computer room; writing to the database in the Append - Only mode for the SSD array.

[0072] The log synchronization process includes: generating a log entry within 200 μs after the main node transaction is committed; realizing cross - computer - room transmission through the cluster network; adopting a unique transaction ID to ensure the sequential consistency of the logs.

[0073] The asynchronous transaction storage processing includes: adopting the batch transaction log + atomic operation processing method, for the transaction logs in the same batch, ensuring atomic operation for storage in the database, and verifying the sequence number and status of the transaction logs before storage.

[0074] The disaster recovery process includes: triggering a switch within 1 second after detecting the failure of the primary node; loading and restoring the in-memory database state from the most recent snapshot (local backup); replaying Kafka logs to compensate for uncommitted transactions.

[0075] Another aspect of the embodiments of the present invention is to provide a memory-based transaction risk control method, including the following steps:

[0076] Receiving a transaction request and parsing session metadata;

[0077] Parallelly executing compliance verification of a preset risk control rule set;

[0078] The transaction request passing the verification triggers atomic operations on the in-memory database;

[0079] Asynchronously generating a timestamped transaction log and pushing it to the Kafka cluster.

[0080] Among them, risk control management, permission management, and rule management are all real-time risk control methods based on the in-memory database. When a user conducts securities market transactions, the following basic condition validations need to be completed:

[0081] 1) Passing the account authentication of the company's Wind.d security authentication system. For user transactions, they first need to pass the Wind account authentication, which is a combination of password verification + SMS verification code. As long as using the company's products, account authorization is required. It should be noted that this authentication system is not the content of this patent.

[0082] 2) After logging in, the user needs to pass the verification of the risk control management system. First, it will check whether the account is in a risk control state, and second, it will check the account trading permissions (such as A-shares, Hong Kong stocks, US stocks, etc.). The system selectively provides different market data and indicator data for this account according to the corresponding permission content found in the database.

[0083] 3) Before logging in to the system, the user will also perform rule verification, including periodically detecting whether it is a commonly used device, updating the user's latest operation habits, updating the user's basic facial information, etc. Only when the rules meet the qualified user standards can active operations be performed.

[0084] 4) From the moment of placing a trade to sending it to the counter, the data already received from the counter, such as the designed trade-related data like entrustment and transaction, are all subject to real-time disk backup through the in-memory database.

[0085] Risk control management includes: querying historical transaction records and behavior records based on the LRU cache; using SIMD instructions to accelerate numerical calculations; and implementing real-time interception of blacklisted accounts.

[0086] Another aspect of the embodiments of the present invention is to provide a computer-readable storage medium, which is characterized in that it stores program codes for implementing the above method, including: a main process event loop scheduling module; a shared memory synchronization management module; a transaction log backup storage module.

[0087] The main process memory management adopts: an object allocator to implement a memory pool with 16-byte alignment; concurrent control is implemented based on the CAS atomic operation; malloc is used to replace the glibc memory allocator.

[0088] The child process implements: a transaction log block storage mechanism, and the loopback flag can automatically overwrite and clean up historical transaction logs.

[0089] The performance requirements of the storage medium, and the implemented system performance indicators include: the single-node processing throughput ≥ 25,000 TPS; the end-to-end request response latency ≤ 10 ms (P99); the disaster recovery RTO ≤ 2 seconds, and the RPO ≤ 5 milliseconds.

[0090] The risk control requirements for the storage medium, generate a risk score by combining user behavior characteristics; forcibly prohibit logging in for high-risk operations; use SIMD instructions to accelerate numerical calculations.

[0091] For the data comparison between the technical solution disclosed in the present invention and the prior art solution, please refer to the following table.

[0092] Indicator Traditional solution The present invention Improvement amplitude Throughput (TPS) 5,000 128,000 25.6 times P99 latency 50 ms 8.2 ms 84％↓ Disaster recovery RTO 30 minutes 1.3 seconds 99.9％↓ Memory fragmentation rate 18％ 2.3％ 87％↓

Claims

1. A memory-based transaction risk control system, characterized in that: This includes using an in-memory database cluster and using a Kafka cluster to synchronize transaction logs across computer rooms, including: The memory database cluster includes: The main process adopts a multi-threaded event loop model and manages full memory data storage through T-tree + HASH index; At least one child process implements lock-free data synchronization through shared memory mapping, and the transaction log uses a custom binary format with a CRC32 checksum. The transaction log is stored in a shared memory block; Use Kafka cluster to synchronize transaction logs across data centers to achieve: Asynchronously write transaction logs via the Kafka producer API; The remote data center consumer group realizes active-active log storage; The SSD array uses the Append-Only mode to write data to the database.

2. A memory-based transaction risk control system as claimed in claim 1, characterized in that: The main process performs the following operations: At startup, a non-persistent shared memory is allocated and the local database file is loaded into it; At the same time, a transaction log shared memory is opened. The subsequent data addition, deletion, query and modification operations performed by the transaction service will generate corresponding transaction logs and write them into the shared memory, notifying the child process to read them and perform subsequent backup and distributed storage.

3. A memory-based transaction risk control system as claimed in claim 1, characterized in that: The child process performs the following operations: monitors the semaphore of the main process through POSIX, and when a new transaction log is triggered, shared memory data needs to be processed; Use the copy-on-write mechanism to ensure data consistency; Generates transaction log entries that contain a global sequence number and nanosecond timestamp.

4. A memory-based transaction risk control system as claimed in claim 1, characterized in that: The control header information of the shared memory includes: the size of the shared memory, the main process data write position, the child process read position, and the version information; the control header information is followed by a transaction log with a CRC32 checksum; a loopback mark is reserved at the end of the shared memory, which is used to indicate whether to reuse the shared memory from the beginning. When the memory is insufficient, the main process writes from the beginning and overwrites the historical transaction log that has been read.

5. The memory-based transaction risk control system according to claim 1, characterized in that: The log synchronization process includes: generating log entries within 200μs after the master node transaction is committed; achieving cross-computer room transmission through the cluster network; using a unique transaction ID to ensure log sequence consistency; Asynchronous transaction storage processing includes: using batch transaction logs and atomic operation processing methods, ensuring atomic operation storage of transaction logs in the same batch, and verifying the sequence number and status of transaction logs before storage; The disaster recovery process includes: triggering switchover within 1 second after detecting a master node failure; restoring the memory database state from the latest snapshot of the local backup; and replaying the Kafka log to compensate for uncommitted transactions.

6. A memory-based transaction risk control method, characterized in that: The transaction risk control system according to claim 1 comprises the following steps: Receive transaction requests and parse session metadata; Compliance checks of preset risk control rule sets are performed in parallel; Triggering atomic operations on the in-memory database through verified transaction requests; Asynchronously generate timestamped transaction logs and push them to the Kafka cluster.

7. A computer-readable storage medium, characterized in that: The program code for implementing the party transaction risk control method described in claim 6 is stored, including: a main process event loop scheduling module; a shared memory synchronization management module; and a transaction log backup storage module.

Images