A distributed dispatching method suitable for securities transaction supervision business

Abstract

The application discloses a distributed dispatching method suitable for securities transaction supervision business, which adopts data time as the basis for dispatching triggering, calculates data time in different business data channels, designs channel data time expiration logic and a to-be-executed algorithm embodiment table, and finally adopts a distributed cluster operation mode; compared with the prior art, the dispatching system supports cluster deployment, can effectively avoid single point failure problems, designs a new data time calculation method, uniformly calculates data time from database data, introduces data expiration calculation logic, the data time calculation is more accurate, and the operation adopts a 24-hour non-stop operation mode, and supports business demands on non-trading days.

Description

Technical Field

[0001] This invention relates to the field of database algorithm technology, specifically a distributed scheduling method applicable to securities trading supervision. Background Technology

[0002] Detecting abnormal trading behavior using securities trading data and regulatory algorithms is a core component of the securities trading regulatory system. Firstly, the securities trading data originates from different channels within the securities trading system. The regulatory system needs to integrate the trading data from these channels, write the data into the database in an orderly manner, and calculate a reliable data time based on the trading data. Secondly, since the algorithms have specific execution modes and cycles, the regulatory system needs to generate regulatory algorithm execution instances with execution times according to the algorithm definition. Finally, the execution of the algorithm execution instance is triggered based on this data time. To ensure the timeliness of the algorithm execution instance execution and the accuracy of the algorithm execution instance calculations, a suitable scheduling system is needed to schedule the execution of the algorithm execution instances.

[0003] A scheduling system suitable for transaction supervision operations must have the following functions:

[0004] (1). Data time calculation method: The execution of algorithm running instances depends on data time. Therefore, an accurate data time calculation method can accurately reflect the current data time and ensure that algorithm running instances are executed in a timely manner and the data is accurate.

[0005] (2) Complete execution record of algorithm running instances: In order to monitor the execution efficiency of algorithm running instances in real time and for subsequent data analysis, the execution status of algorithm running instances needs to be recorded completely;

[0006] (3) Support for re-execution of algorithm execution instances: When an algorithm execution instance fails or the business requires re-execution of the algorithm execution instance, the scheduling system must support the re-execution of the algorithm execution instance;

[0007] (4) High availability of the system: In order to avoid the impact of system failures on the stability of the scheduling system, the scheduling system must meet the requirements of high availability.

[0008] The existing transaction supervision and scheduling system is a single-process scheduling system. This system is an independent process that starts at 6:00 AM and stops at 7:30 PM every trading day. The system's operation flow is as follows:

[0009] a. After the scheduling system starts, it reads the algorithm configuration data, generates the algorithm execution instances to be executed that day, including information such as algorithm name, algorithm execution instance name, planned execution time of the algorithm execution instance, and time range of execution data for the algorithm execution instance, and stores it in the memory queue;

[0010] b. After the scheduling system starts, it accesses securities trading data from different channels via TCP to obtain data from different channels in real time (time T). i The global data time T is calculated as follows:

[0011] T = min(T) i )

[0012] c. When the global data time exceeds the planned execution time of the algorithm instance, the algorithm instance is triggered to execute, and the algorithm instance completes the calculation by querying transaction data in the database;

[0013] d. After the algorithm instance is executed, the results are written to the database.

[0014] The existing transaction supervision business scheduling system only supports operation on trading days and cannot support business operations on non-trading days. The system is deployed at a single point. When the single point fails, the scheduling will be abnormal, causing the running instances to fail to execute. Furthermore, when the system fails, all memory algorithm running instances will be reloaded, causing all algorithm running instances to be re-executed. Summary of the Invention

[0015] The purpose of this invention is to overcome the shortcomings of the prior art and provide a distributed scheduling method applicable to securities trading supervision.

[0016] To achieve the above objectives, a distributed scheduling method suitable for securities trading supervision is designed. The scheduling method uses data time as the basis for scheduling triggering, calculates the data time in different business data channels, and also designs channel data time expiration logic and a table of examples of algorithms to be executed. Finally, a distributed cluster operation mode is adopted, as detailed below.

[0017] T1. When the system starts, each transaction data channel is initialized with a set of data. This set of data includes the maximum inbound time, the data time corresponding to the maximum inbound time, the system time corresponding to the maximum inbound time, and the channel status. The maximum inbound time represents the channel's maximum inbound time, with an initial value of 0. The data time corresponding to the maximum inbound time also has an initial value of 0. The system time corresponding to the maximum inbound time has an initial value of the current system time. In the channel status, 0 indicates invalid, and 1 indicates valid, with an initial value of 1.

[0018] T2. When the system starts up, the global data initialization time T is 0;

[0019] T3. Query the maximum inbound time and corresponding maximum data time for each channel from the transaction data written to the database;

[0020] T4. Compare the maximum inbound time with the maximum inbound time of each channel. If they are not equal, update the maximum inbound time to the maximum inbound time of each channel, update the data time corresponding to the maximum inbound time to the corresponding maximum data time, and update the system time corresponding to the maximum inbound time found to the current system time.

[0021] T5. Calculate the time difference between the current system time and the system time corresponding to the maximum inbound time found. If the time difference exceeds 1 minute, update the channel status to 0.

[0022] T6. Select channels whose status is not 0, and calculate the temporary data time from these channels;

[0023] T7. If the temporary data time value is greater than the global data time T value, then update the global data time T value to the temporary data time.

[0024] The present invention also includes the following preferred technical solutions:

[0025] Furthermore, the distributed scheduling method applicable to securities trading supervision is as follows:

[0026] S1. When the scheduling node starts, a heartbeat maintenance thread is created. This thread obtains the host's IP and process PID. When the heartbeat maintenance thread runs, it writes the host IP, process PID, and current timestamp into the database heartbeat information table in a loop.

[0027] S2. When a scheduling node starts, a heartbeat monitoring thread is created. This thread continuously queries the heartbeat information table for information on all scheduling node instances and determines whether the heartbeat data of each scheduling node has timed out.

[0028] S3. When the scheduling node starts, it initiates a trading day update polling task. This task queries the database for the current trading day information and updates it in memory.

[0029] S4. When the scheduling node starts, it starts the algorithm running instance update polling task. This task queries the table of pending algorithm instances to see if the algorithm running instance corresponding to the algorithm for the current trading day has been generated. If the running instance of a certain algorithm has not been generated, the algorithm running instance is generated according to the algorithm configuration data and written to the table of pending algorithm instances.

[0030] S5. When the system starts, a scheduling thread is created for scheduling algorithm execution instances. This thread first creates a thread pool for asynchronous execution of algorithm execution instances, and then executes the scheduling thread in a loop.

[0031] Furthermore, the loop steps of the subsequent execution scheduling thread loop are as follows:

[0032] E1. Check the remaining available resources in the thread pool for the algorithm execution instance. If there are no remaining available resources, skip this loop.

[0033] E2. Query currently executable algorithm instances.

[0034] Furthermore, the query for currently executable algorithm instances includes:

[0035] D1. Query the current business time;

[0036] D2. Query executable algorithm instances. Based on the global data time T, query the table of executable algorithm instances to be executed for executable running instances;

[0037] D3. Iterate through the retrieved algorithm instances and perform the following operations for each instance:

[0038] D4. After the algorithm instance is executed, update the status of the algorithm instance in the table of pending algorithm instances to "success".

[0039] Compared with the prior art, the advantages of this invention are:

[0040] 1. The scheduling system of the present invention supports cluster deployment, which can effectively avoid the problem of single point of failure.

[0041] 2. The scheduling system of the present invention adopts a distributed processing approach, supports horizontal scaling, and can improve scheduling performance.

[0042] 3. This invention designs a new data time calculation method, which uniformly calculates data time from database data and introduces data expiration calculation logic, making data time calculation more accurate.

[0043] 4. This invention stores all algorithm instances in the database, enabling real-time monitoring of algorithm instance execution performance. Furthermore, by changing the state of the algorithm instance, the algorithm instance can be re-executed.

[0044] 5. The method of the present invention operates 24 hours a day without interruption, supporting business needs on non-trading days. Attached Figure Description

[0045] Figure 1 This is an overview diagram of the scheduling system of the present invention.

[0046] Figure 2 This is a flowchart of the scheduling system of the present invention. Detailed Implementation

[0047] See Figure 1 The diagram below is an overview of the scheduling system of the present invention. The entire system mainly includes the following modules:

[0048] 1. Database Model Design: To ensure the normal operation of the scheduling system cluster, a data model was designed that includes a heartbeat table, a shard table, an application operation status table, an algorithm instance table to be executed (alarm_schedule_rules_h), and an algorithm instance generation log table to be executed (alarm_schedule_rules_log).

[0049] 2. Heartbeat Maintenance Submodule: This module updates the database data and updates the status of the scheduling node to the heartbeat table in real time.

[0050] 3. Heartbeat Monitoring Submodule: This module updates the database data and monitors the status of each scheduling node in the heartbeat table in real time. If the node status is abnormal, an alarm will be issued.

[0051] 4. Shard Data Maintenance Submodule: When the status of the scheduling node changes, the data in the shard_items table will be updated synchronously, and the status information of the row in the app_status table where the field app_name = sharding will also be updated.

[0052] 5. Trading Day Data Update Submodule: This module is responsible for maintaining the information of the current trading day (T day) in the scheduling node.

[0053] 6. Algorithm Run Instance Maintenance Module: This module interacts with the alarm_schedule_rules_h table in the database and is responsible for inserting and updating algorithm run instance information. This module exposes the algorithm run instance generation interface, allowing for custom implementations.

[0054] 7. Data Time Calculation Module: This module calculates the current data time in real time and exposes an interface that allows for custom data time calculation methods.

[0055] 8. Algorithm Execution Instance Scheduling Module: This module queries executable algorithm execution instances in real time based on the current data time and schedules their execution asynchronously.

[0056] In cluster mode, the scheduling system cluster contains multiple scheduling process nodes. The scheduling execution flowchart of each scheduling node is as follows: Figure 2 As shown, the specific steps are as follows:

[0057] 1. When a scheduling node starts, a heartbeat maintenance thread is created. This thread obtains the host's IP address and the process PID. While running, the heartbeat maintenance thread continuously writes the host IP address, process PID, and current timestamp to the `heartbeat` table in the database. When process information is inserted into the `heartbeat` table, it indicates a new scheduling node has been added to the scheduling system cluster. At this point, the status of the row in the `app_status` table where `app_name = sharding` needs to be updated to `WAIT`. The heartbeat maintenance thread also includes a heartbeat retry mechanism. If heartbeat data is not updated in time for unknown reasons and is determined to be an abnormal heartbeat by the heartbeat monitoring thread, the heartbeat maintenance thread will rewrite the process information into the `heartbeat` table. If the number of consecutive abnormal heartbeats exceeds 10, the scheduling node information will no longer be restored to the `heartbeat` table, and an alert will be issued.

[0058] 2. When the scheduling node starts, a heartbeat monitoring thread is created. This thread queries the heartbeat table for information on all scheduling container instances in a loop and determines whether the heartbeat data of each container has timed out. If the heartbeat of a container times out, the information of that container is deleted from the heartbeat table, the status of the shard in the app_status table is updated to WAIT, and the status of the algorithm instance executed by that container in the alarm_schedule_rules_h table is changed to FAILED.

[0059] 3. When the scheduling node starts, it initiates a trading day update polling task. This task queries the database for the current trading day information and updates it in memory.

[0060] 4. When the scheduling node starts, it initiates an algorithm runtime instance update polling task. This task queries the `alarm_schedule_rules_h` table to check if an algorithm runtime instance has been generated for the current trading day. If no runtime instance has been generated for a certain algorithm, it generates an algorithm runtime instance based on the algorithm configuration data and writes it to the `alarm_schedule_rules_h` table. The algorithm runtime instance generation provides an externally exposed interface, allowing for custom implementations to adapt to different algorithm configuration methods.

[0061] 5. When the system starts, a scheduling thread is created for scheduling algorithm execution instances. This thread first creates a thread pool for asynchronous execution of algorithm execution instances, and then the scheduling thread performs the following operations in a loop:

[0062] 5.1 Check the remaining available resources in the thread pool of the algorithm execution instance. If there are no remaining available resources, skip this loop.

[0063] 5.2 Query currently executable algorithm instances:

[0064] 1) Query the current business time

[0065] (1) When the system starts, it initializes a set of data (M) for each transaction data channel i. i T i N i S i ) i M i This represents the maximum inbound time for channel i, with an initial value of 0; T i This represents the data time corresponding to the maximum inbound time, with an initial value of 0; N i This indicates that M was found. i The corresponding system time, initially set to the current system time; S i This indicates the state of channel i (0: invalid, 1: valid), with an initial value of 1.

[0066] (2) When the system starts, the global data is initialized at time T=0.

[0067] (3) Query the maximum inbound time M for each channel from the transaction data written to the database. i-new and the corresponding maximum data time T i-new .

[0068] (4) Compare M i With M i-new Check if they are equal; if not, update M. i For M i-new Update T i For T i-new Update N i This is the current system time.

[0069] (5) Calculate the current system time and N i The time difference, if it exceeds 1 minute, will be S i Updated to 0.

[0070] (6) Select S i For channels ≠ 0, calculate the temporary data time K = min(T) from these channels. i ).

[0071] (7) If the value of K is greater than the value of T, then update the global data time T to the temporary data time K.

[0072] 2) Query executable algorithm instances: Query the alarm_schedule_rules_h table for executable algorithm running instances based on the global data time T.

[0073] 3) Iterate through the retrieved algorithm instances and perform the following operations for each instance:

[0074] (1) Query the status data corresponding to the row in the app_status table where app_name = sharding. If the status is WAIT, update the data in the shard_items table according to the information in the heartbeat table.

[0075] (2) Based on the algorithm running instance information and the data in the shard_items table, confirm whether the algorithm running instance is being executed on the current node. If it is not being executed on the current node, skip the subsequent steps.

[0076] (3) Update the status of the algorithm running instance in the alarm_schedule_rules_h table to RUNNING;

[0077] (4) Submit the algorithm running instance to the thread pool for asynchronous execution.

[0078] 4) After the algorithm instance is executed, update the status of the algorithm instance in the alarm_schedule_rules_h table to SUCCESS.

[0079] Furthermore, the implementation process of the present invention will be illustrated with examples.

[0080] Suppose you have a database that supports transactions and a server that can run Java applications.

[0081] 1. Create a schedule database instance in the database, and create the following tables in the schedule database:

[0082] (1)heartbeat table

[0083]

[0084] (2) app_status table

[0085]

[0086]

[0087] (3) shard_items table

[0088]

[0089] (4) alarm_schedule_rules_h table

[0090]

[0091]

[0092] (5)alarm_schedule_rules_log table

[0093]

[0094] 2. Update the database connection information of the scheduling system to the database created in step 1, and package the scheduling system into a runnable jar file.

[0095] 3. Start two processes on the application server using the JAR package from step 2. At this point, the scheduling system will insert two entries into the heartbeat table, for example...

[0096] host_ip pid modify_time 192.*** 1 ...... 192.*** 2 ......

[0097] 4. Create a new regulatory business algorithm A1. This algorithm requires calculating the data of the most recent 5 minutes every minute within the time range of [0930, 1500].

[0098] 5. The scheduling system reads the algorithm from step 4 and writes the algorithm execution instance information into the `alarm_schedule_rules_h` table according to the algorithm requirements, for example:

[0099]

[0100] 6. At this point, write the business data into the database, for example:

[0101] Channel number Data Time Write time Channel 1 09:35:04 09:35:10 Channel 2 09:35:01 09:35:05 Channel 3 09:34:00 09:34:02

[0102] Assuming the system time is 09:35:11, according to the data time calculation logic, the data in "Channel 3" is invalid, and the time of "Channel 1" and "Channel 2" is taken as the smaller time. Therefore, the data time at this time is 09:35:01.

[0103] 7. When the data time is 09:35:01, algorithm instance A1-1 can be scheduled for execution;

[0104] 8. At this point, the status field of the row in the app_status table where app_name = sharding has a value of WAIT. Therefore, sharding needs to be performed before executing A1-1. After sharding, the sharded table data is as follows.

[0105] host_ip pid modify_time item 192.*** 1 ...... 0 192.*** 2 ...... 1

[0106] 9. Calculate the hash value of A1, modulo 2 of the total number of fragments. If the result is 0, then A1-1 will be executed in the process with pid=1.

[0107] 10. After the execution of algorithm instance A1-1 is completed, the running status corresponding to the alarm_schedule_rules_h table will be updated to SUCCESS.

[0108] The above description is merely a specific embodiment of the invention, but the scope of protection of the invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the technical scope disclosed in the invention, based on the technical solutions and novel concepts of the invention, should be covered within the scope of protection of the invention.

Claims

1. A distributed scheduling method suitable for securities transaction supervision business, characterized in that: The specific method is as follows: S1. When the scheduling node starts, a heartbeat maintenance thread is created, which obtains the IP and process PID of the host. When the heartbeat maintenance thread runs, the host IP, process PID, and current timestamp are written into the database heartbeat information table in a loop; S2. When the scheduling node starts, a heartbeat monitoring thread is created, which queries all scheduling node instance information from the heartbeat information table in a loop, and judges whether the heartbeat data of each scheduling node is overdue; S3. When the scheduling node starts, a trading day update polling task is started, which queries the current trading day information from the database and updates it to the memory; S4. When the scheduling node starts, an algorithm running instance update polling task is started, which queries whether the algorithm running instance corresponding to the algorithm of the current trading day in the to-be-executed algorithm instance table is generated. If the running instance of an algorithm is not generated, generate the algorithm running instance according to the algorithm configuration data and write it to the to-be-executed algorithm instance table; S5. When the system starts, a scheduling thread is created for algorithm running instance scheduling. The thread first creates a thread pool for asynchronous execution of algorithm running instances, and then executes the subsequent scheduling thread loop; The loop steps of the subsequent execution scheduling thread loop are as follows: E1. Query the remaining available resources in the algorithm running instance execution thread pool. If there is no remaining available resource, skip this loop; E2. Query the currently executable algorithm instance; The query of the currently executable algorithm instance includes D1. Query the current business time; D2. Query the executable algorithm instance according to the global data time T in the to-be-executed algorithm instance table; D3. Traverse the queried algorithm instance and perform specific operations on each algorithm instance; D4. After the execution of the algorithm instance, update the state of the algorithm instance in the to-be-executed algorithm instance table to success; The scheduling method uses data time as the basis for scheduling triggering, calculates the data time in different business data channels, designs the channel data time expiration logic and the to-be-executed algorithm instance table, and finally adopts a distributed cluster running mode. The D1 is as follows: T1. When the system starts, initialize each trading data channel to a set of data, which includes the maximum import time, the data time corresponding to the maximum import time, the system time corresponding to the maximum import time, and the state of the channel. The maximum import time represents the maximum import time of the channel, the initial value is 0; the data time corresponding to the maximum import time is also 0; the system time corresponding to the maximum import time is the current system time; the state of the channel is 0 for invalid and 1 for valid, the initial value is 1; T2. Initialize the global data time T to 0 when the system starts; T3. Query the maximum import time and the corresponding maximum data time of each channel from the trading data written to the database; T4. Compare whether the maximum storage time is equal to the maximum storage time of each channel, if not, update the maximum storage time to the maximum storage time of each channel, and update the data time T corresponding to the maximum storage time i For the corresponding maximum data time, update the system time corresponding to the maximum storage time to the current system time; T5. Calculate the time difference between the current system time and the system time corresponding to the maximum import time. If the time difference exceeds 1 minute, update the state of the channel to 0; T6. Selecting the channels whose states are not 0, calculating the temporary data time K = min(T i ) from these channels. T7. If the temporary data time value is greater than the global data time T value, then update the global data time T value to be the temporary data time.

Images