Data distributed liquidation batch processing system and method

Abstract

The invention discloses a data distribution type liquidation batch processing system and method, and relates to the technical field of data processing, through system structural module design, a traditional integrated liquidation system is divided into three subsystems according to function modules, namely a batch processing subsystem, a scheduling subsystem and a centralized liquidation subsystem; wherein the scheduling subsystem is used for connecting the processing processes of the batch processing subsystem and the centralized clearing subsystem in series, so that the originally coupled processing logic is divided into independent clearing steps, and logic decoupling is realized. Meanwhile, a plurality of parallel batch databases are arranged in the batch processing subsystem, the large-data-volume files are split and distributed to the batch databases, processing of the batch databases is independent and does not affect one another, parallel processing can be achieved, and the processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data distributed clearing batch processing system and method. Background Art

[0002] The clearing system is a basic financial system used by financial institutions to check, sort, record and settle large-volume transaction funds. Due to the advantages of large-scale mainframes such as high integration, good compatibility, safety and reliability, large-scale mainframes are widely used in financial securities and other industries. The clearing system of banking institutions is implemented on such integrated large-scale mainframes. The clearing system database on the mainframe is deployed as a single database, which means that all data must be processed in one database. The original transaction data, intermediate data, report data, and fund data processed in the clearing process are all stored in the same database.

[0003] However, as the business develops, the daily transaction volume has reached tens of millions, which has exceeded the carrying capacity of a single database. Problems such as excessive database resource usage, longer SQL statement execution time, and slower liquidation process speed have gradually emerged, and even serious problems such as memory overflow and hard disk capacity exhaustion may occur. In addition, since the integrated single database deployment solution does not take into account subsequent expansion, if you want to solve the above problems by expanding the database, you will also face large-scale modifications to the deployment architecture and applications, and the maintenance cost is high. Summary of the invention

[0004] The main purpose of this application is to provide a data distributed clearing batch processing system and method, aiming to solve the technical problems of the existing clearing system that the clearing process is slow and the maintenance cost is high due to the increase in transaction volume.

[0005] To achieve the above objectives, the present application proposes a data distributed clearing batch processing system, the data distributed clearing batch processing system comprising:

[0006] Batch processing subsystem, dispatching subsystem and centralized clearing subsystem;

[0007] The scheduling subsystem is used to obtain a clearing step configuration table, and monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0008] The scheduling subsystem is further configured to send a batch processing instruction to the batch processing subsystem when monitoring that the batch processing subsystem obtains the transaction file;

[0009] The batch processing subsystem is used to split the transaction file upon receiving the batch processing instruction, and perform batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data;

[0010] The scheduling subsystem is further configured to send a centralized processing instruction to the centralized clearing subsystem when the clearing intermediate data is detected;

[0011] The centralized clearing subsystem is used to collect the intermediate clearing data and perform centralized clearing based on the collected batch clearing results when receiving the centralized processing instruction, so as to obtain the target clearing result.

[0012] In one implementation, the batch processing instruction includes: a file splitting instruction and a batch clearing instruction;

[0013] The batch processing subsystem is used to obtain a file splitting rule when receiving the file splitting instruction;

[0014] The batch processing subsystem is further used to obtain the routing factor in the transaction file based on the file splitting rule;

[0015] The batch processing subsystem is further used to split the transaction file based on the routing factor to obtain batch transaction data;

[0016] The scheduling subsystem is further configured to send a batch clearing instruction to the batch processing subsystem when the batch transaction data is monitored;

[0017] The batch processing subsystem is also used to distribute the batch transaction data to each batch database for batch data clearing when receiving the batch clearing instruction, so as to obtain clearing intermediate data.

[0018] In one implementation, the scheduling subsystem is further used to obtain a clearing step configuration table;

[0019] The scheduling subsystem is further used to determine the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0020] The scheduling subsystem is further used to monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

[0021] In one implementation, the scheduling subsystem is further used to obtain the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0022] The scheduling subsystem is further used to determine the clearing step configuration table to obtain the clearing execution procedures and execution preconditions of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem;

[0023] The scheduling subsystem is further used to assign an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program and the execution precondition.

[0024] In one implementation, the scheduling subsystem is further used to determine the current execution status of the clearing step when monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem;

[0025] The scheduling subsystem is further used to update the clearing step operation table based on the current clearing execution step and the clearing step configuration table;

[0026] The scheduling subsystem is also used to generate a clearing step execution instruction based on the clearing step execution status in the clearing step operation table, and send the clearing step execution instruction to the batch processing subsystem or the centralized clearing subsystem; the clearing step execution instruction includes: a batch processing instruction or a centralized processing instruction.

[0027] In one implementation, the scheduling subsystem is further used to obtain a liquidation execution cycle;

[0028] The scheduling subsystem is further used to initialize the execution state of the clearing step when the clearing execution cycle is converted;

[0029] The scheduling subsystem is further configured to generate an abnormality alarm when the initialization process is abnormal.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a data distributed clearing batch processing method, which is used in the data distributed clearing batch processing system as described above, and the data distributed clearing batch processing method includes:

[0031] The scheduling subsystem obtains a clearing step configuration table, and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0032] When the scheduling subsystem detects that the batch processing subsystem has acquired the transaction file, the scheduling subsystem sends a batch processing instruction to the batch processing subsystem;

[0033] When receiving the batch processing instruction, the batch processing subsystem splits the transaction file, and performs batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data;

[0034] When the scheduling subsystem detects the intermediate clearing data, the scheduling subsystem sends a centralized processing instruction to the centralized clearing subsystem;

[0035] When receiving the centralized processing instruction, the centralized clearing subsystem collects the intermediate clearing data and performs centralized clearing based on the collected batch clearing results to obtain the target clearing result.

[0036] In one implementation, the batch processing instruction includes: a file splitting instruction and a batch clearing instruction; when the batch processing subsystem receives the batch processing instruction, it splits the transaction file and performs batch clearing based on the batch transaction data obtained by the splitting, and the step of obtaining the clearing intermediate data includes:

[0037] The batch processing subsystem obtains a file splitting rule when receiving the file splitting instruction;

[0038] The batch processing subsystem obtains the routing factor in the transaction file based on the file splitting rule;

[0039] The batch processing subsystem splits the transaction file based on the routing factor to obtain batch transaction data;

[0040] When the scheduling subsystem detects the batch transaction data, the scheduling subsystem sends a batch settlement instruction to the batch processing subsystem;

[0041] When receiving the batch clearing instruction, the batch processing subsystem distributes the batch transaction data to each batch database for batch data clearing to obtain clearing intermediate data.

[0042] In one implementation, the scheduling subsystem obtains a clearing step configuration table, and based on the clearing step configuration table, performs clearing execution step monitoring on the batch processing subsystem and the centralized clearing subsystem, including:

[0043] The scheduling subsystem obtains a clearing step configuration table;

[0044] The scheduling subsystem determines the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0045] The scheduling subsystem monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

[0046] In one implementation, the step of determining, by the scheduling subsystem, the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table includes:

[0047] The scheduling subsystem obtains the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0048] The scheduling subsystem determines the liquidation step configuration table to obtain the liquidation execution procedures and execution preconditions of each of the liquidation execution steps in the batch processing subsystem and the centralized liquidation subsystem;

[0049] The scheduling subsystem assigns an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program, and the execution precondition.

[0050] One or more technical solutions proposed in this application have at least the following technical effects:

[0051] The data distributed clearing batch processing system of the present application includes: a batch processing subsystem, a scheduling subsystem and a centralized clearing subsystem; the scheduling subsystem obtains a clearing step configuration table, and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; when the batch processing subsystem is monitored to obtain the transaction file, the batch processing instruction is sent to the batch processing subsystem; when the batch processing subsystem receives the batch processing instruction, the transaction file is split, and the batch transaction data obtained by the splitting are cleared in batches to obtain the clearing intermediate data; when the scheduling subsystem monitors the clearing intermediate data, the scheduling subsystem sends a centralized processing instruction to the centralized clearing subsystem; when the centralized clearing subsystem receives the centralized processing instruction, the centralized clearing subsystem collects the clearing intermediate data, and performs centralized clearing based on the collected batch clearing results to obtain the target clearing result. Because it is designed by modularization of the system structure, the original integrated clearing system is split into three subsystems according to the functional modules, namely the scheduling subsystem, the batch processing subsystem and the centralized clearing subsystem. The scheduling subsystem connects the processing processes of the batch processing subsystem and the centralized clearing subsystem in series, so that the originally coupled processing logic is split into independent clearing steps, realizing logical decoupling. At the same time, the batch processing subsystem of the present application is provided with multiple parallel batch databases. By evenly splitting and distributing large data volume files to each batch database, the processing of each batch database is independent and does not affect each other, which can realize parallel processing and improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 This is a functional structure diagram of the first embodiment of the distributed data clearing batch processing system of this application;

[0055] Figure 2 This is a schematic diagram of the overall process in one implementation of the distributed clearing batch processing system for data in this application;

[0056] Figure 3 A schematic diagram of the file splitting and downloading process in one implementation of the distributed clearing batch processing system for data of this application;

[0057] Figure 4 A schematic diagram of the scheduling process in one implementation of the distributed clearing batch processing system for data in this application;

[0058] Figure 5 A flow chart of the first embodiment of the distributed clearing batch processing method for data in this application.

[0059] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] This application provides a solution to the shortcomings of the integrated clearing system, and invents an easily scalable large-volume distributed clearing batch processing solution to support clearing and settlement services with large transaction volumes. It mainly solves the problems of logical coupling of clearing processing, low processing capacity of a single database, long clearing process, and difficulty in resource expansion in the integrated clearing system.

[0063] Specifically, the present application proposes a data distributed clearing batch processing system. Figure 1 , Figure 1 This is a functional structure diagram of the first embodiment of the data distributed clearing batch processing system of this application.

[0064] In the embodiments of the present application, Figure 1 As shown, the data distributed clearing batch processing system includes: a batch processing subsystem, a scheduling subsystem and a centralized clearing subsystem;

[0065] The scheduling subsystem is used to obtain a clearing step configuration table, and monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0066] The scheduling subsystem is further configured to send a batch processing instruction to the batch processing subsystem when monitoring that the batch processing subsystem obtains the transaction file;

[0067] The batch processing subsystem is used to split the transaction file upon receiving the batch processing instruction, and perform batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data;

[0068] The scheduling subsystem is further configured to send a centralized processing instruction to the centralized clearing subsystem when the clearing intermediate data is detected;

[0069] The centralized clearing subsystem is used to collect the intermediate clearing data and perform centralized clearing based on the collected batch clearing results when receiving the centralized processing instruction, so as to obtain the target clearing result.

[0070] It should be noted that the batch processing subsystem in the embodiment of the present application is a system including multiple databases, and its main function is to split a large amount of transaction data and evenly distribute the split batch transaction data to each batch database. By performing reconciliation and accumulation in parallel in each batch database, batch clearing of transaction data is achieved, and clearing intermediate data is obtained.

[0071] It should be explained that the centralized clearing subsystem in the embodiment of the present application refers to a system that uses a single database. Its main function is to collect the clearing intermediate data generated by each batch database in the batch processing subsystem, and to summarize, net, check and account for the clearing intermediate data, and finally output the target clearing result.

[0072] It can be understood that through the batch processing subsystem and the centralized clearing subsystem, the transaction data can be cleared in batches and then centrally cleared, and large data files can be evenly split and distributed to each batch database. Batch clearing is performed through each batch database, and data processing is independent of each other and does not affect each other. At the same time, parallel processing can be performed, which improves data processing efficiency.

[0073] It should be noted that in the embodiment of the present application, the scheduling subsystem is the scheduling control center of the data distributed batch processing system of the present application. As a "bridge" connecting the batch processing subsystem and the centralized clearing subsystem, it only includes a scheduling database. The main function of the scheduling subsystem of the present application is to control the execution order of the entire clearing step, trigger the execution of the clearing steps of the batch processing subsystem and the centralized clearing subsystem, and store data files related to scheduling control in the scheduling database.

[0074] It can be understood that the scheduling subsystem of the embodiment of the present application can monitor the workflow steps in the batch processing subsystem and the centralized clearing subsystem through connection with the batch processing subsystem and the centralized clearing subsystem, and further realize the execution control of the clearing steps in the batch processing subsystem and the centralized clearing subsystem.

[0075] In one implementation of the embodiment of the present application, all the clearing execution steps of the embodiment of the present application can be divided into four clearing modules according to their functions: a file splitting and downloading module, an intermediate data generation module, an intermediate data collection module, and an output clearing result module. Through the operation of the above four clearing modules, a large amount of transaction data can be concentrated in multiple batch databases for parallel processing to obtain clearing intermediate data with a small amount of data. The clearing intermediate data with a small amount of data is handed over to the centralized clearing subsystem for processing to generate the final target clearing result. The scheduling system focuses on controlling the clearing steps to be executed in sequence, which solves the problems of the clearing processing logic coupling of the integrated clearing system, the low processing capacity of a single database, the long clearing process, and the difficulty of resource expansion.

[0076] Among them, the intermediate data generation module is used to start a series of clearing execution steps such as reconciliation and accumulation of transaction data after the batch processing subsystem downloads the transaction file and obtains the transaction data, and generates clearing intermediate data such as the number of transactions with unbalanced reconciliation, the principal summary amount, and the handling fee summary amount, and saves these clearing intermediate data in the batch database. The intermediate data collection module is used to query the various batch databases of the batch processing subsystem when the batch processing subsystem generates clearing intermediate data, and collect and summarize the clearing intermediate data in these batch databases to finally obtain complete clearing data. The clearing result output module is used to obtain complete clearing data when the centralized clearing subsystem collects and summarizes the clearing intermediate data of each batch database of the batch processing subsystem, and uses this complete clearing data to generate the final verification report and accounting report, and finally completes the clearing and accounting of the entire fund.

[0077] In one implementation of the embodiment of the present application, the overall workflow diagram of the distributed data clearing batch processing system of the present application can be as follows: Figure 2 shown.

[0078] Reference Figure 2In the batch processing subsystem of the present application, several processors may be provided (four are shown in the figure, namely, the first processor 1, the second processor 2, the third processor 3, and the fourth processor 4), and each processor may be provided with a batch database (shown as DTA01, DTA02, DTA03, and DTA04). In the scheduling subsystem, a scheduling machine (shown as scheduling machine 1) may be provided, and a scheduling database (shown as DTC00) may be provided in the scheduling machine. In the centralized clearing subsystem, a clearing machine (shown as clearing machine 1) may be provided, and a centralized clearing database (shown as DTB00) may be provided in the clearing machine.

[0079] In one implementation of the embodiment of the present application, the batch processing instruction includes: a file splitting instruction and a batch clearing instruction;

[0080] The batch processing subsystem is used to obtain a file splitting rule upon receiving the file splitting instruction;

[0081] The batch processing subsystem is further used to obtain the routing factor in the transaction file based on the file splitting rule;

[0082] The batch processing subsystem is further used to split the transaction file based on the routing factor to obtain batch transaction data;

[0083] The scheduling subsystem is further configured to send a batch clearing instruction to the batch processing subsystem when the batch transaction data is monitored;

[0084] The batch processing subsystem is also used to distribute the batch transaction data to each batch database for batch data clearing when receiving the batch clearing instruction, so as to obtain clearing intermediate data.

[0085] It should be noted that for the file splitting and downloading module, the batch processing subsystem can download the transaction files sent by the transaction system, and parse and put the transaction records in the transaction files into a table for subsequent settlement. Since the transaction files are usually large, they can be evenly split into several parts according to certain rules, and these parts can be evenly distributed to each processor (batch database) in the batch processing subsystem for processing.

[0086] In one implementation of the embodiment of the present application, the splitting rule of the transaction file is to split it according to the serial number. Specifically, since the first three digits of the serial number are randomly generated three-digit numbers, and these three digits are evenly distributed between 0-9, when receiving the transaction file and the file splitting instruction sent by the scheduling system, the batch processing subsystem can use the first three digits of the serial number as a "routing factor" and split the transaction file evenly according to the value range of the routing factor, and the number of split batches is consistent with the number of batch databases of the batch processing subsystem.

[0087] Furthermore, in order to enable each batch database of the batch processing subsystem to obtain the same file splitting rule, the embodiment of the present application can define a file splitting rule table in the centralized clearing database of the centralized clearing subsystem. Through the splitting rules in the file splitting rule table, the name of each batch database in the batch processing subsystem and the routing factor range and other fields can be defined to indicate which batch database in the batch processing subsystem the transactions with different routing factors are downloaded to. Therefore, before the batch processing subsystem downloads the transaction file, it can first call the centralized clearing subsystem to query the file splitting rule table, so as to filter and download the transaction records in the transaction file according to the splitting rules in the file splitting rule table.

[0088] It should be noted that when the batch processing subsystem obtains the file splitting rules, the transaction file can be downloaded. When downloading each transaction record in the transaction file, the batch processing subsystem can intercept the first three digits of the serial number as the routing factor, and use the routing factor to match the file splitting rule table. If the routing factor meets the routing factor range of the current batch database, the transaction record will be parsed and placed in the table; if it does not meet the range, it will be directly filtered out. The above method can evenly split the large transaction file to each batch database for processing. The specific process can be as follows: Figure 3 The routing factor range corresponding to the batch database DTA01 may be 000-249, the routing factor range corresponding to the batch database DTA02 may be 250-499, the routing factor range corresponding to the batch database DTA03 may be 500-749, and the routing factor range corresponding to the batch database DTA04 may be 750-999.

[0089] The data distributed clearing batch processing system in the embodiment of the present application supports the configuration of data routing rules. Each piece of data falls into a designated batch database according to the configured routing rules, and the large data volume files are evenly split and distributed to each batch database for processing according to the routing rules, reducing the load pressure of a single batch database and improving the throughput of the overall system. In addition, through the solution of the present application, it is possible to implement the configuration of the execution batch database that supports clearing batch processing, and quickly achieve system expansion by configuring a new database to participate in clearing batch processing without adjusting the deployment architecture and application programs, making the clearing system flexible, efficient, and easy to expand.

[0090] The data distributed clearing batch processing system of the embodiment of the present application includes: a batch processing subsystem, a scheduling subsystem and a centralized clearing subsystem; the scheduling subsystem obtains a clearing step configuration table, and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; when the batch processing subsystem is monitored to obtain a transaction file, a batch processing instruction is sent to the batch processing subsystem; when the batch processing subsystem receives the batch processing instruction, the transaction file is split, and batch clearing is performed based on the batch transaction data obtained by splitting, and the clearing intermediate data is obtained; when the scheduling subsystem monitors the clearing intermediate data, the scheduling subsystem sends a centralized processing instruction to the centralized clearing subsystem; when the centralized clearing subsystem receives the centralized processing instruction, it collects the clearing intermediate data and performs centralized clearing based on the collected batch clearing results to obtain the target clearing result. Because it is designed by modularization of the system structure, the original integrated clearing system is split into three subsystems according to the functional modules, namely the scheduling subsystem, the batch processing subsystem and the centralized clearing subsystem. The scheduling subsystem connects the processing processes of the batch processing subsystem and the centralized clearing subsystem in series, so that the originally coupled processing logic is split into independent clearing steps, realizing logical decoupling. At the same time, the batch processing subsystem of the present application is provided with multiple parallel batch databases. By evenly splitting and distributing large data volume files to each batch database, the processing of each batch database is independent and does not affect each other, which can realize parallel processing and improve processing efficiency.

[0091] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 ,In the embodiment of the present application, the scheduling subsystem is also used to obtain a clearing step configuration table;

[0092] The scheduling subsystem is further used to determine the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0093] The scheduling subsystem is further used to monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

[0094] It can be understood that the scheduling machine in the scheduling system can realize the scheduling of batch processing subsystem and centralized clearing subsystem, such as controlling file splitting and downloading, generating intermediate data, collecting intermediate data, and outputting clearing results.

[0095] In the embodiment of the present application, since the liquidation execution steps need to be executed in a certain order, the scheduling subsystem in the present application can save the execution order and dependency of the liquidation execution steps. Specifically, a liquidation step configuration table can be defined in the scheduling database inside the scheduling subsystem. In the configuration table, fields such as step directory, step number, step description, execution program, and pre-wait condition can be set. The meanings of these fields are as follows:

[0096] Step directory: Identifies which clearing module the clearing step belongs to. There can be four enumeration values: A, B, C, and D, which correspond to the four clearing modules: file splitting and downloading, generating intermediate data, collecting intermediate data, and outputting clearing results. The clearing execution steps corresponding to these four clearing modules are executed in sequence. The clearing modules with a later English ranking will definitely wait until the clearing modules with a higher English ranking are completed before they are executed. Therefore, the file splitting and downloading module corresponding to A is executed first, and the output clearing result module corresponding to D is executed last.

[0097] The step number is the unique key value of the liquidation execution step, which is used to identify a unique liquidation execution step. The first digit of the step number is taken from the step directory, followed by a unique sequence number;

[0098] Step description: functional description of the clearing execution step;

[0099] Execution procedure: The clearing execution step corresponds to the name of the procedure to be executed in the batch processing subsystem or the centralized clearing subsystem;

[0100] Precondition: The step number of other clearing execution steps that the clearing execution step needs to wait for before execution. Although the execution order of the overall clearing execution steps can be controlled through the step directory, if the clearing execution steps in the same step directory need to be executed in a certain order, this can be controlled by configuring the precondition.

[0101] Specifically, the scheduling subsystem is further used to obtain the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0102] The scheduling subsystem is further used to determine the clearing step configuration table to obtain the clearing execution procedures and execution preconditions of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem;

[0103] The scheduling subsystem is further used to assign an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program and the execution precondition.

[0104] It should be noted that after defining the clearing step configuration table, you also need to define a clearing step running table in the running state. The clearing step running table is generated from the clearing step configuration table. In addition to all the fields of the original configuration table, it also adds two additional running state fields: date and status.

[0105] Among them, Date: refers to the settlement date, which indicates the date of the transaction to be settled; Status: represents the execution status of the settlement step, and there are four enumeration values: I, S, A, and X, which represent initial state, executing, success, and failure respectively.

[0106] Specifically, the scheduling subsystem is further used to determine the current execution status of the clearing step when monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem;

[0107] The scheduling subsystem is further used to update the clearing step operation table based on the current clearing execution step and the clearing step configuration table;

[0108] The scheduling subsystem is also used to generate a clearing step execution instruction based on the clearing step execution status in the clearing step operation table, and send the clearing step execution instruction to the batch processing subsystem or the centralized clearing subsystem; the clearing step execution instruction includes: a batch processing instruction or a centralized processing instruction.

[0109] It should be noted that the scheduling subsystem of the embodiment of the present application calls four modules (A-file splitting and downloading module, B-generating intermediate data module, C-collecting intermediate data module and D-outputting clearing result module) and controls the operation of the clearing execution steps of each module through the clearing step configuration table, so that the entire system can run in an orderly and stable manner.

[0110] In one implementation method of an embodiment of the present application, the specific functions of the file splitting and downloading module may include: A1-acquiring transaction file download, A2-issuing transaction file download, A3-error transaction file download, etc.; the specific functions of the intermediate data generation module may include: B1-total principal generation, B2-total handling fee generation, B3-unbalanced number of transactions generation, etc.; the specific functions of the intermediate data collection module may include: C1-total principal collection, C2-total handling fee collection, C3-unbalanced number of transactions collection, etc.; the specific functions of the settlement result output module may include: D1-fund report output, D2-fund verification, D3-fund accounting, etc.

[0111] The scheduling subsystem in the embodiment of the present application obtains a clearing step configuration table; determines the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order. Since the clearing execution steps of the clearing process are controlled through the clearing step configuration table, the problems of the clearing processing logic coupling, the low processing capacity of a single database, and the long clearing time of the integrated clearing system are solved.

[0112] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment and / or the second embodiment can be referred to the above description, and will not be described in detail later. On this basis, in the embodiment of the present application, the scheduling subsystem is also used to obtain the clearing execution cycle;

[0113] The scheduling subsystem is further used to initialize the execution state of the clearing step when the clearing execution cycle is converted;

[0114] The scheduling subsystem is further configured to generate an abnormality alarm when the initialization process is abnormal.

[0115] It should be noted that, in the embodiment of the present application, a clearing execution cycle may be provided to realize periodic clearing of data. In one implementation of the embodiment of the present application, data clearing may be performed on a daily basis. Specifically, at midnight every day, after all transactions of the previous day have been completed, the scheduling subsystem may initialize the clearing cycle to all clearing execution steps of the previous day through a scheduled job (which may be the first scheduled job), that is, the states of the clearing execution steps are all changed to the initial state, thereby preparing to start clearing all transactions of the previous day. At the same time, another scheduled job (which can be called the second scheduled job) scans the clearing step operation table once a minute. If a clearing execution step that meets the following conditions is found: (1) the status is the initial state; (2) the step belongs to the first step directory among all the clearing execution steps in the initial state; (3) the clearing execution steps corresponding to the prerequisites of the step have been completed, then it can be decided to go to the batch processing subsystem or the centralized clearing subsystem to start execution according to the configuration of the execution program, and the status of the clearing execution step is updated to being executed; if a clearing execution step with a status of being executed is found, then it can be decided to go to the batch processing subsystem or the centralized clearing subsystem to query the corresponding execution status according to the configuration of the execution program, and the status of the clearing execution step is updated to the clearing step operation table.

[0116] In one implementation of the embodiment of the present application, since the scheduling subsystem is the control center of the entire liquidation batch processing, and the liquidation step operation table records the operation status of each liquidation step, the scheduling subsystem of the present application can also build a monitoring alarm mechanism to monitor the abnormal situation of the liquidation batch processing. The scheduling system periodically scans the step operation table through a scheduled job (which can be a third scheduled job) and executes the following check logic in sequence: (1) If the liquidation step operation table has not initialized all the liquidation steps of the previous day, an alarm is triggered; (2) If there are liquidation steps in a failed state in the step operation table, an alarm is triggered; (3) If all steps in the step operation table have not been fully executed at a certain time (such as 9 o'clock in the morning), an alarm is triggered. The above method can ensure that the relevant personnel can be quickly notified when the liquidation step is executed abnormally, so as to handle the abnormality in time and ensure the stability and reliability of the system.

[0117] In one implementation of an embodiment of the present application, if there is a need to increase transaction processing capacity and improve clearing processing capabilities, since the processing of large quantities of transaction data in the solution of the present application is concentrated in the batch processing subsystem, it is only necessary to expand the number of batch databases in the batch processing subsystem, and there is no need to expand the database in the centralized clearing subsystem, thereby improving the applicability of the solution of the present application.

[0118] It should be noted that when the data distributed clearing batch processing system of the present application expands the batch database of the batch processing subsystem, thanks to the easily extensible architecture design of the distributed clearing system, expanding the database will not affect the architecture design of the entire clearing system, and there is no need to modify the program code. It is only necessary to modify the file splitting rule table stored in the centralized clearing database of the centralized clearing subsystem, add a new batch database configuration to the file splitting rule table, and adjust the routing factor range of each batch database to ensure that the routing factor range covers the 3-digit range of 000-999.

[0119] Furthermore, the execution program can be used to automatically adjust the file splitting rule table by obtaining the total number of batch databases of the current batch processing subsystem and automatically and evenly allocating the routing factor range according to the total number of batch databases, thereby simplifying the manual configuration operation.

[0120] In the embodiment of the present application, the scheduling subsystem obtains the liquidation execution cycle; when the liquidation execution cycle changes, the execution state of the liquidation step is initialized; when the initialization process is abnormal, an abnormal alarm is generated. Because the scheduling subsystem performs periodic initialization and generates an alarm when the initialization fails, the abnormality can be handled in time to ensure the stability and reliability of the system.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data distributed clearing batch processing system of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] This application also provides a data distributed clearing batch processing method, please refer to Figure 5 The data distributed clearing batch processing method is applied to the data distributed clearing batch processing system as described above, and the method comprises:

[0123] Step S10, the scheduling subsystem obtains a clearing step configuration table, and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0124] Step S20, when the scheduling subsystem detects that the batch processing subsystem has acquired the transaction file, the scheduling subsystem sends a batch processing instruction to the batch processing subsystem;

[0125] Step S30, when receiving the batch processing instruction, the batch processing subsystem splits the transaction file, and performs batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data;

[0126] Step S40, when the scheduling subsystem detects the clearing intermediate data, the scheduling subsystem sends a centralized processing instruction to the centralized clearing subsystem;

[0127] Step S50, when receiving the centralized processing instruction, the centralized clearing subsystem collects the intermediate clearing data and performs centralized clearing based on the collected batch clearing results to obtain the target clearing result.

[0128] The data distributed clearing batch processing method provided by the present application adopts the data distributed clearing batch processing system in the above embodiment, which can solve the technical problem that the clearing process of the existing clearing system is slow due to the increase in transaction volume. Compared with the prior art, the beneficial effects of the data distributed clearing batch processing method provided by the present application are the same as the beneficial effects of the data distributed clearing batch processing system provided by the above embodiment, and the other technical features in the data distributed clearing batch processing method are the same as the features disclosed in the above embodiment system, which will not be repeated here.

[0129] In an implementation of the embodiment of the present application, the batch processing instruction includes: a file splitting instruction and a batch clearing instruction; when the batch processing subsystem receives the batch processing instruction, it splits the transaction file and performs batch clearing based on the batch transaction data obtained by splitting, and the step of obtaining the clearing intermediate data includes:

[0130] The batch processing subsystem obtains a file splitting rule when receiving the file splitting instruction;

[0131] The batch processing subsystem obtains the routing factor in the transaction file based on the file splitting rule;

[0132] The batch processing subsystem splits the transaction file based on the routing factor to obtain batch transaction data;

[0133] When the scheduling subsystem detects the batch transaction data, the scheduling subsystem sends a batch settlement instruction to the batch processing subsystem;

[0134] When receiving the batch clearing instruction, the batch processing subsystem distributes the batch transaction data to each batch database for batch data clearing to obtain clearing intermediate data.

[0135] In one implementation of the embodiment of the present application, the scheduling subsystem obtains a clearing step configuration table, and the step of monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table includes:

[0136] The scheduling subsystem obtains a clearing step configuration table;

[0137] The scheduling subsystem determines the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0138] The scheduling subsystem monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

[0139] In one implementation of the embodiment of the present application, the step of determining, by the scheduling subsystem, the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table includes:

[0140] The scheduling subsystem obtains the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table;

[0141] The scheduling subsystem determines the liquidation step configuration table to obtain the liquidation execution procedures and execution preconditions of each of the liquidation execution steps in the batch processing subsystem and the centralized liquidation subsystem;

[0142] The scheduling subsystem assigns an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program, and the execution precondition.

[0143] In one implementation of the embodiment of the present application, the method further includes:

[0144] The scheduling subsystem determines the current execution status of the clearing step when monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem;

[0145] The scheduling subsystem updates the clearing step operation table based on the current clearing execution step and the clearing step configuration table;

[0146] The scheduling subsystem generates a clearing step execution instruction based on the clearing step execution status in the clearing step operation table, and sends the clearing step execution instruction to the batch processing subsystem or the centralized clearing subsystem; the clearing step execution instruction includes: a batch processing instruction or a centralized processing instruction.

[0147] In one implementation of the embodiment of the present application, the method further includes:

[0148] The scheduling subsystem is also used to obtain the clearing execution cycle;

[0149] The scheduling subsystem is further used to initialize the execution state of the clearing step when the clearing execution cycle is converted;

[0150] The scheduling subsystem is further configured to generate an abnormality alarm when the initialization process is abnormal.

[0151] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data distributed clearing batch processing system, characterized in that: The system includes: a batch processing subsystem, a scheduling subsystem and a centralized clearing subsystem; The scheduling subsystem is used to obtain a clearing step configuration table, and monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; The scheduling subsystem is further configured to send a batch processing instruction to the batch processing subsystem when monitoring that the batch processing subsystem obtains the transaction file; The batch processing subsystem is used to split the transaction file upon receiving the batch processing instruction, and perform batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data; The scheduling subsystem is further configured to send a centralized processing instruction to the centralized clearing subsystem when the clearing intermediate data is detected; The centralized clearing subsystem is used to collect the intermediate clearing data and perform centralized clearing based on the collected batch clearing results when receiving the centralized processing instruction, so as to obtain the target clearing result.

2. The data distributed clearing batch processing system according to claim 1, characterized in that: The batch processing instructions include: file splitting instructions and batch clearing instructions; The batch processing subsystem is used to obtain a file splitting rule when receiving the file splitting instruction; The batch processing subsystem is further used to obtain the routing factor in the transaction file based on the file splitting rule; The batch processing subsystem is further used to split the transaction file based on the routing factor to obtain batch transaction data; The scheduling subsystem is further configured to send a batch clearing instruction to the batch processing subsystem when the batch transaction data is monitored; The batch processing subsystem is also used to distribute the batch transaction data to each batch database for batch data clearing when receiving the batch clearing instruction, so as to obtain clearing intermediate data.

3. The data distributed clearing batch processing system according to claim 1, characterized in that: The scheduling subsystem is also used to obtain a clearing step configuration table; The scheduling subsystem is further used to determine the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; The scheduling subsystem is further used to monitor the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

4. The data distributed clearing batch processing system according to claim 3, characterized in that: The scheduling subsystem is further used to obtain the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; The scheduling subsystem is further used to determine the clearing step configuration table to obtain the clearing execution procedures and execution preconditions of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem; The scheduling subsystem is further used to assign an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program and the execution precondition.

5. The data distributed clearing batch processing system according to claim 4, characterized in that: The scheduling subsystem is further used to determine the current execution status of the clearing step when monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem; The scheduling subsystem is further used to update the clearing step operation table based on the current clearing execution step and the clearing step configuration table; The scheduling subsystem is further used to generate a clearing step execution instruction based on the clearing step execution status in the clearing step operation table, and send the clearing step execution instruction to the batch processing subsystem or the centralized clearing subsystem; The clearing step execution instructions include: batch processing instructions or centralized processing instructions.

6. The data distributed clearing batch processing system according to claim 5, characterized in that: The scheduling subsystem is also used to obtain the clearing execution cycle; The scheduling subsystem is further used to initialize the execution state of the clearing step when the clearing execution cycle is converted; The scheduling subsystem is further configured to generate an abnormality alarm when the initialization process is abnormal.

7. A data distributed clearing batch processing method, characterized in that: The method is used in the data distributed clearing batch processing system according to any one of claims 1 to 6, and the method comprises: The scheduling subsystem obtains a clearing step configuration table, and monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; When the scheduling subsystem detects that the batch processing subsystem has acquired the transaction file, the scheduling subsystem sends a batch processing instruction to the batch processing subsystem; When receiving the batch processing instruction, the batch processing subsystem splits the transaction file, and performs batch clearing based on the batch transaction data obtained by the splitting to obtain clearing intermediate data; When the scheduling subsystem detects the intermediate clearing data, the scheduling subsystem sends a centralized processing instruction to the centralized clearing subsystem; When receiving the centralized processing instruction, the centralized clearing subsystem collects the intermediate clearing data and performs centralized clearing based on the collected batch clearing results to obtain the target clearing result.

8. The data distributed clearing batch processing method according to claim 7, characterized in that: The batch processing instruction includes: a file splitting instruction and a batch clearing instruction; when the batch processing subsystem receives the batch processing instruction, it splits the transaction file and performs batch clearing based on the batch transaction data obtained by splitting, and the step of obtaining the clearing intermediate data includes: The batch processing subsystem obtains a file splitting rule when receiving the file splitting instruction; The batch processing subsystem obtains the routing factor in the transaction file based on the file splitting rule; The batch processing subsystem splits the transaction file based on the routing factor to obtain batch transaction data; When the scheduling subsystem detects the batch transaction data, the scheduling subsystem sends a batch settlement instruction to the batch processing subsystem; When receiving the batch clearing instruction, the batch processing subsystem distributes the batch transaction data to each batch database for batch data clearing to obtain clearing intermediate data.

9. The data distributed clearing batch processing method according to claim 7, characterized in that: The step of the scheduling subsystem acquiring a clearing step configuration table, and monitoring the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table, includes: The scheduling subsystem obtains a clearing step configuration table; The scheduling subsystem determines the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; The scheduling subsystem monitors the clearing execution steps of the batch processing subsystem and the centralized clearing subsystem based on the execution order.

10. The data distributed clearing batch processing method according to claim 9, characterized in that: The step of determining, by the scheduling subsystem, the execution order of the clearing execution steps in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table comprises: The scheduling subsystem obtains the clearing step key value of each clearing execution step in the batch processing subsystem and the centralized clearing subsystem based on the clearing step configuration table; The scheduling subsystem determines the liquidation step configuration table to obtain the liquidation execution procedures and execution preconditions of each of the liquidation execution steps in the batch processing subsystem and the centralized liquidation subsystem; The scheduling subsystem assigns an execution order to each of the clearing execution steps based on the clearing step key value, the clearing execution program, and the execution precondition.

Images